Document how dictionaries are created

[juanjoDiaz]

So people can work on them, improve what we have, create custom dictionaries, etc...

[adbar]

Yes, it would be better, I'll work on this.

[adbar]

I'll document it later in a readme in training/ but here are a few conditions:

• The data is present in two columns: lemma TAB word form
• Redundant and noisy cases are filtered out but it's best if the data is clean
• To do so, you need a source comprising enough words without sacrificing quality, I believe the kaikki.org project is best for additional languages at this point (I can provide a script later on)
• The new language (two- or three-letter code) has to be added to the dictionary data (using the dictionary_pickler script), it should then be available in SUPPORTED_LANGUAGES
• Ideally the data should be reviewed and if possible tested on an authoritative source like UD

[juanjoDiaz]

I could create scripts similar to what I did for evaluation on #116, but for training.
However, I still don't understand fully where the source data is coming from or how are you creating the dictionaries.

Could you elaborate a bit more on that?

[adbar]

@juanjoDiaz Yes, I'll work on the repository in October.

[juanjoDiaz]

HI @adbar ,

Are you still maintaining this library?
Any plan to still document this and release v1.0?

[adbar]

Hi, I'm working on something else at the moment but I still plan to work on it.

[juanjoDiaz]

Hi @adbar ,

I just reviewed the doc that you created and I have some questions:

You mention multiple dataset:

• http://unimorph.ethz.ch/languages
• https://github.com/lenakmeth/Wikinflection-Corpus
• https://github.com/TALP-UPC/FreeLing/tree/master/data/
• https://github.com/explosion/spacy-lookups-data/tree/master/spacy_lookups_data/data
• https://github.com/tatuylonen/wiktextract
• http://pauillac.inria.fr/~sagot/index.html#udlexicons
• https://unimorph.github.io/
• https://kaikki.org/

However, I'm very unclear about what of those datasets you actually use to create the current dictionary.
You documentation only covers Kaiki
If I want to rebuild the dictionaries from scratch, how could I do it ensuring the same result?
(This would be needed if we want to re-add to the dictionaries the words removed because they match a rule)

Most of them seem to just get the data from wiktionary, aren't they?

[adbar]

This project started as a simple experiment and I didn't implement version control on the data so what I did is not reproducible I'm afraid. The lists mentioned in the Readme were first used roughly in that order and in that importance.

Small mistakes needed to be fixed in the lists so there are a lot of small steps involved which I tried to replicate in the dictionary pickling module, e.g. if the first column is obviously too different from the second one.
Side note: Sometimes there are forms generated automatically which don't exist in the target language (in TALP-UPC lists for example if I remember correctly), I tried to remove them but it's not a big deal since there are simply never found as they don't exist.

I believe the current state is not a problem as the aggregated lists currently present in the data can be extracted and further refined.

Over time it became clear that Kaikki was the best option to add new languages as the data is cleaner and encompasses more languages. It is not necessarily more comprehensive per language but it keeps getting better (both Wiktionary data and extraction). I expect Kaikki's part to continue becoming larger, finding and cleaning lists for additional languages is too much of a hassle.
Note: UD lists are good in principle but then you're left with nothing to evaluate the lemmatizer on.