Difference between pages "Global categories taxonomy/Wines" and "Product Opener - How to update a taxonomy"

From Open Food Facts wiki
(Difference between pages)
Jump to navigation Jump to search
 
 
Line 1: Line 1:
[[Category:OpenWineFacts]]
+
[[Project:ProductOpener]] > [[Product Opener documentation]]
<pre>
+
[[Category:ProductOpener]]
<en:Alcoholic beverages
+
== Introduction ==
en:Wines
 
es:Vinos
 
fr:Vins, vin
 
  
<en:Beverages
+
Global taxonomies are used in Product Opener to "tag" products. For instance Open Food Facts uses taxonomies for categories, countries, labels, additives etc.
en:Non-Alcoholic beverages
 
fr:Boissons sans alcool, boisson sans alcool
 
  
<en:Wines
+
The taxonomies are multilingual (each tag value can have translations in different languages) and hierarchical (tags have parents, and applying a tag to a product also applies the parent tags).
<en:Non-Alcoholic beverages
 
en:Non-Alcoholic wines, Alcohol-free wines
 
fr:Vins sans alcool, vins déalcoolisés
 
  
<en:Wines
+
== Taxonomy files ==
en:Bordeaux, Bordeaux wines
 
fr:Bordeaux
 
  
<en:Wines
+
Taxonomies are kept in the taxonomies directory:
en:French wines, Wines from France
 
fr:Vins français, vin français
 
  
<en:French wines
+
<pre>
fr:Vins de pays, vin de pays
+
stephane@ks3095298:~/product-opener/taxonomies$ ls
 +
additives.result.sto  categories.result.txt  countries.txt
 +
additives.result.txt  categories.txt        labels.result.sto
 +
additives.txt          countries.result.sto  labels.result.txt
 +
categories.result.sto  countries.result.txt  labels.txt
 +
</pre>
  
<en:Wines
+
* categories.txt is the source definition of the categories taxonomy
en:Red wines
+
* categories.result.txt is the resulting normalized definition after compilation in the same format as the source categories.txt
fr:vins rouges, vin rouge
+
** entries are normalized (e.g. the en:value is preferred to specify parents)
 +
** entries can be sorted in a different order
 +
* categories.result.sto in the resulting binary representation of the taxonomy, in Perl's native format
  
<en:Wines
+
== Taxonomy source format ==
en:White wines
 
fr:vins blancs, vin blanc
 
  
<en:Wines
+
See [[Global taxonomies]]
en:Wines from Italy, Italian wines
 
fr:Vins italiens, vin italien, vins d'Italie, vin d'Italie
 
  
<en:Wines
+
== Taxonomy source files ==
en:Wines from Portugal, Portuguese wines
 
fr:Vins portugais, Vins du Portugal, Vin portugais, Vin du Portugal
 
  
<en:Wines from Portugal
+
Open Food Facts taxonomy source files are on the wiki so that contributors can improve them (adding tags, adding translations, reorganizing tags): http://en.wiki.openfoodfacts.org/Global_taxonomies
pt:Denominação de Origem Controlada
 
  
<pt:Denominação de Origem Controlada
+
== Compilation ==
pt:Vinho verde (DOC)
 
  
<pt:Denominação de Origem Controlada
+
Before taxonomies can be used in Product Opener, they need to be compiled in a binary format. The compilation has two goals:
pt:Porto (DOC)
 
  
<pt:Denominação de Origem Controlada
+
# Generate as many synonyms as possible for each tag
pt:Douro (DOC)
+
#* This is so that end users can enter any form of a tag and still have it matched to the canonical tag
 +
#* e.g. "Yaourts fraise" and "Yaourt aux fraises"
 +
# Compute the hierarchy
 +
#* for each tag, knowing which tags are parents and childrens
 +
 +
The compilation code is in the build_tags_taxonomy() function of Tags.pm. This function is called by script/build_tags_taxonomy.pl
  
<pt:Denominação de Origem Controlada
+
=== Compilation steps ===
pt:Távora Varosa (DOC)
 
  
<pt:Denominação de Origem Controlada
+
The compilation steps are:
pt:Dão (DOC)
+
* read translations and synonyms
 +
* compute synonyms for each tag (using synonyms for some of the words)
 +
* add more synonyms (remove stopwords, generate simple singular/plural forms)
 +
* compute the hierarchy
  
<pt:Denominação de Origem Controlada
+
Note: if the taxonomy is big and/or contain many synoym
pt:Bairrada (DOC)
 
  
<pt:Denominação de Origem Controlada
+
=== Compilation script ===
pt:Beira Interior (DOC)
 
  
<pt:Denominação de Origem Controlada
+
==== Testing ====
pt:Encostas de Aire (DOC)
+
<pre>
 +
stephane@ks3095298:~/product-opener/scripts$ ./build_tags_taxonomy.pl categories
 +
</pre>
 +
This will compile the categories but generate only categories.results.txt from categories.txt
  
<pt:Denominação de Origem Controlada
+
You should review the resulting txt to make sure unwanted synonyms or relations were not created.
pt:Alentejo (DOC)
 
  
<pt:Denominação de Origem Controlada
+
===== empty names : =====
pt:Ribatejo (DOC)
 
  
<pt:Denominação de Origem Controlada
+
An empty name " : " indicates there is an error in the taxonomy source file, usually a non-existing entry was entered as the parent.
pt:Lourinhã (DOC)
 
  
<pt:Denominação de Origem Controlada
+
<pre>
pt:Alenquer (DOC)
+
< :
 +
< en:Wheat flours
 +
en:Wheat malt flours
 +
es:Harinas de trigo malteadas, Harinas de malta de trigo
 +
fr:Farines de malt de blé
 +
</pre>
  
<pt:Denominação de Origem Controlada
+
Check the source file to see which entry is wrong:
pt:Torres Vedras (DOC)
 
  
<pt:Denominação de Origem Controlada
+
<pre>
pt:Arruda (DOC)
+
<en:Wheat flours
 +
<fr:Malt flours
 +
en:Wheat malt flours
 +
es:Harinas de trigo malteadas, Harinas de malta de trigo
 +
fr:Farines de malt de blé
 +
</pre>
  
<pt:Denominação de Origem Controlada
+
Here "fr:Malt flours" is non existing (and should be "en:Malt flours").
pt:Bucelas (DOC)
 
  
<pt:Denominação de Origem Controlada
+
Note: the build_tags_taxonomy.pl script has been updated to report those errors.
pt:Colares (DOC)
 
  
<pt:Denominação de Origem Controlada
+
==== Publishing ====
pt:Carcavelos (DOC)
 
  
<pt:Denominação de Origem Controlada
+
<pre>
pt:Palmela (DOC)
+
stephane@ks3095298:~/product-opener/scripts$ ./build_tags_taxonomy.pl categories publish
 +
</pre>
 +
This will also store categories.result.sto
  
<pt:Denominação de Origem Controlada
+
== Test ==
pt:Muscat de Setúbal (DOC)
 
  
<pt:Denominação de Origem Controlada
+
Taxonomies are very powerful tools, but with great powers comes great risks. The synonyms, stopwords etc. can create unwanted synonyms.
pt:Lagos (DOC)
 
  
<pt:Denominação de Origem Controlada
+
=== Testing tools ===
pt:Portimão (DOC)
 
  
<pt:Denominação de Origem Controlada
+
We need testing tools. Please create some. :-)
pt:Lagoa (DOC)
 
  
<pt:Denominação de Origem Controlada
+
== Restart Apache ==
pt:Tavira (DOC)
 
  
<pt:Denominação de Origem Controlada
+
<pre>
pt:Madère (DOC)
+
</pre>
  
 +
After restarting Apache, editing a product will recompute all tags using the newest published taxonomies.
  
<en:Wines
 
en:Wines from Spain, Spanish wines
 
fr:Vins espagnols, Vin espagnol, Vin d'Espagne, Vins d'Espagne
 
  
<en:Wines
+
== Updating products ==
fr:Vins américains
 
  
<fr:Vins américains
+
The scripts directory contains scripts that will recompute tags for all products in the database.
en:Wines from California, Californian Wines, California Wines
 
fr:Vins californiens, vins de Californie, vin californien, vin de Californie
 
wikidata:en:Q1134713
 
 
 
<en:Wines
 
fr:Vins effervescents
 
en:Sparkling wines
 
wikidata:en:Q321263
 
 
 
<fr:Vins effervescents
 
fr:Blanquette de Limoux
 
wikidata:en:Q881685
 
 
 
<fr:Vins effervescents
 
fr:Champagne, champagnes, vins de Champagne
 
en:Champagne, champagnes
 
 
 
<fr:Vins effervescents
 
fr:Clairette de Die, Clairettes de Die
 
wikidata:en:Q1094848
 
 
 
<fr:Vins effervescents
 
fr:vins mousseux, vin mousseux, mousseux
 
 
 
<fr:Vins effervescents
 
fr:vins perlant, vin perlant, perlé
 
 
 
<fr:Vins effervescents
 
fr:vins pétillant, vin pétillant, pétillant
 
 
 
<en:Wines
 
fr:Vins mutés, vin muté
 
en:Fortified wines
 
wikidata:en:Q722338
 
 
 
<en:Wines from Portugal
 
<fr:Vins mutés de type vin doux naturel
 
fr:Porto, vins de Porto, vin de Porto
 
 
 
<en:Red wines
 
<fr:Porto
 
en:red ports
 
fr:porto rouges, porto rouge
 
 
 
<en:Red ports
 
fr:porto réductifs, porto réductif
 
 
 
<fr:Porto réductifs
 
fr:Porto Crusted, Crusted
 
 
 
<fr:Porto réductifs
 
fr:Porto Late Bottled Vintage, LBV
 
 
 
<fr:Porto réductifs
 
fr:Porto Ruby
 
 
 
<fr:Porto réductifs
 
fr:Porto Single Quinta Vintage
 
 
 
<fr:Porto réductifs
 
fr:Porto Vintage, porto millésimé
 
 
 
<en:Red ports
 
fr:portos oxydatifs, porto oxydatif
 
 
 
<fr:Portos oxydatifs
 
fr:Porto Tawny, tawny
 
 
 
<en:White wines
 
<fr:Porto
 
en:White Ports
 
fr:porto blancs, porto blanc
 
 
 
en:Sweet wines, natural sweet wines
 
es:Vinos dulces
 
fr:Vins doux, Vin doux, Vins doux naturels, Vin doux naturel
 
 
 
<en:Sweet wines
 
<fr:Vins mutés
 
fr:vins mutés de type vin doux naturel, VDN
 
 
 
<fr:Vins mutés de type vin doux naturel
 
fr:Banyuls
 
 
 
<fr:Vins mutés de type vin doux naturel
 
fr:Maury
 
 
 
<fr:Vins mutés de type vin doux naturel
 
fr:Muscats, Muscat
 
 
 
<fr:Muscats
 
fr:Muscat de Beaumes-de-Venise, Muscats de Beaumes-de-Venise
 
 
 
<fr:Muscats
 
fr:Muscat de Frontignan, Muscats de Frontignan
 
 
 
<fr:Muscats
 
fr:Muscat de Lunel, Muscats de Lunel
 
 
 
<fr:Muscats
 
fr:Muscat de Mireval, Muscats de Mireval
 
 
 
<fr:Muscats
 
fr:Muscat de Rivesaltes, Muscats de Rivesaltes
 
 
 
<fr:Muscats
 
fr:Muscat de Saint-Jean-de-Minervois, Muscats de Saint-Jean-de-Minervois
 
 
 
<fr:Muscats
 
fr:Muscat du Cap-Corse, Muscats du Cap-Corse
 
 
 
<fr:Vins mutés de type vin doux naturel
 
fr:Rasteau
 
 
 
<fr:Vins mutés de type vin doux naturel
 
fr:Rivesaltes
 
 
 
<fr:Vins mutés
 
fr:mistelle, vins mutés de types mistelle, vins de liqueur, vin de liqueur
 
 
 
<fr:Mistelle
 
fr:Pineau des Charentes, Floc de Gascogne, Macvin du Jura
 
 
 
<fr:Vins mutés
 
fr:vins vinés, vin viné
 
 
 
<en:Wines
 
fr:Vins rosés, vin rosé
 
  
 +
<pre>
 +
stephane@ks3095298:~/product-opener/scripts$ ./update_all_products_categories.pl
 
</pre>
 
</pre>

Revision as of 15:34, 2 May 2015

Project:ProductOpener > Product Opener documentation

Introduction

Global taxonomies are used in Product Opener to "tag" products. For instance Open Food Facts uses taxonomies for categories, countries, labels, additives etc.

The taxonomies are multilingual (each tag value can have translations in different languages) and hierarchical (tags have parents, and applying a tag to a product also applies the parent tags).

Taxonomy files

Taxonomies are kept in the taxonomies directory:

stephane@ks3095298:~/product-opener/taxonomies$ ls
additives.result.sto   categories.result.txt  countries.txt
additives.result.txt   categories.txt         labels.result.sto
additives.txt          countries.result.sto   labels.result.txt
categories.result.sto  countries.result.txt   labels.txt
  • categories.txt is the source definition of the categories taxonomy
  • categories.result.txt is the resulting normalized definition after compilation in the same format as the source categories.txt
    • entries are normalized (e.g. the en:value is preferred to specify parents)
    • entries can be sorted in a different order
  • categories.result.sto in the resulting binary representation of the taxonomy, in Perl's native format

Taxonomy source format

See Global taxonomies

Taxonomy source files

Open Food Facts taxonomy source files are on the wiki so that contributors can improve them (adding tags, adding translations, reorganizing tags): http://en.wiki.openfoodfacts.org/Global_taxonomies

Compilation

Before taxonomies can be used in Product Opener, they need to be compiled in a binary format. The compilation has two goals:

  1. Generate as many synonyms as possible for each tag
    • This is so that end users can enter any form of a tag and still have it matched to the canonical tag
    • e.g. "Yaourts fraise" and "Yaourt aux fraises"
  2. Compute the hierarchy
    • for each tag, knowing which tags are parents and childrens

The compilation code is in the build_tags_taxonomy() function of Tags.pm. This function is called by script/build_tags_taxonomy.pl

Compilation steps

The compilation steps are:

  • read translations and synonyms
  • compute synonyms for each tag (using synonyms for some of the words)
  • add more synonyms (remove stopwords, generate simple singular/plural forms)
  • compute the hierarchy

Note: if the taxonomy is big and/or contain many synoym

Compilation script

Testing

stephane@ks3095298:~/product-opener/scripts$ ./build_tags_taxonomy.pl categories

This will compile the categories but generate only categories.results.txt from categories.txt

You should review the resulting txt to make sure unwanted synonyms or relations were not created.

empty names :

An empty name " : " indicates there is an error in the taxonomy source file, usually a non-existing entry was entered as the parent.

< :
< en:Wheat flours
en:Wheat malt flours
es:Harinas de trigo malteadas, Harinas de malta de trigo
fr:Farines de malt de blé

Check the source file to see which entry is wrong:

<en:Wheat flours
<fr:Malt flours
en:Wheat malt flours
es:Harinas de trigo malteadas, Harinas de malta de trigo
fr:Farines de malt de blé

Here "fr:Malt flours" is non existing (and should be "en:Malt flours").

Note: the build_tags_taxonomy.pl script has been updated to report those errors.

Publishing

stephane@ks3095298:~/product-opener/scripts$ ./build_tags_taxonomy.pl categories publish

This will also store categories.result.sto

Test

Taxonomies are very powerful tools, but with great powers comes great risks. The synonyms, stopwords etc. can create unwanted synonyms.

Testing tools

We need testing tools. Please create some. :-)

Restart Apache


After restarting Apache, editing a product will recompute all tags using the newest published taxonomies.


Updating products

The scripts directory contains scripts that will recompute tags for all products in the database.

stephane@ks3095298:~/product-opener/scripts$ ./update_all_products_categories.pl