Better dictionaries

[zupo]

The Story

Handbook documents: User story & Work process

As a dev,
I want to use a better dictionary,
so that there are less false-positives.

Problem

Currently, we use aspell to check for typos in localization files. Aspell's dictionaries for non-English languages are quite bad, for Romanian language, we need to ignore ~2600 words.

Proposal

Find if there are better alternatives. I believe LibreOffice publishes their dictionaries, can we use those?

Pitfalls

• aspell is quick, we may considerably slow down build times by using a different spell check.

Best practices (DoD)

• Documentation is revised:
  e.g. help articles or technical docs.
• Product users are informed. e.g. blog post describing a new feature or bugfix.
• Test coverage is 100%.

Expectations (AC)

• We no longer have to ignore +2600 words for Romanian localization files.
• User story demo uploaded to sprint release.

[dz0ny]

Enchant is an engine that supports 8 different backends aspell/libreoffice/mozilla... So it can be used as a direct replacement for aspell... and yes we can combine both backends, use one etc..

[ikolar]

Romanian really seems to be tricky.

Haven't tried pyenchant yet but it seems to be unmaintained at moment. People are suggesting using hunspell (libreoffice dict) instead, however, that one is actually even worse than aspell.

This is the number of unique spelling mistakes in all Romanian localization files, after filtering out the anglish pws, with 1) aspell 2) hunspell.

(localizations) ike@stvm:/devel/niteo/localizations$ cat Countries/Romania/.html | aspell --encoding=utf-8 --personal=$PWD/.travis/dictionaries/en.pws list | sort | uniq | aspell --lang=ro --encoding=utf-8 list | wc -l
343
(localizations) ike@stvm:/devel/niteo/localizations$ cat Countries/Romania/.html | aspell --encoding=utf-8 --personal=$PWD/.travis/dictionaries/en.pws list | sort | uniq | hunspell -d ro_RO -l | wc -l
347

There doesn't seem to be a Romanian dict for ispell, at least not in the debian repos.

I'll see if I can do anything better with pyenchant.

Any other ideas?

[ikolar]

pyenchant (the default version from pip) sadly isn't better. 357 false positives, compared to 343 of aspell and 347 of hunspell.

cat Countries/Romania/* | aspell --lang=en --encoding=utf-8 list | sort | uniq > ro.all

python pyenchant.py |wc -l
357

The script is:

from enchant.checker import SpellChecker
import codecs

with codecs.open("ro.all", "r", "utf-8") as f:
t = f.read()

chkr = SpellChecker("ro_RO")
chkr.set_text(t)
for err in chkr:
print(err.word)

There is one more backend I can try:

Uspell (primarily Yiddish, Hebrew, and Eastern European languages - hosted on AbiWord github)

[ikolar]

Sadly, AbiWord doesn't have a Romanian dictionary. Tried googling for another one, but couldn't find anything obvious.

Do we know anyone in Romania that could possibly point us to a better dictionary?

Also, do you still want me to switch the code from aspell to pyenchant? Pyenchant just calls enchant, which in turn calls either aspell, myspell, ispell or uspell (or hspell, voikko, and zemberek for Hebrew, Finnish or Turkish). So assuming that aspell's dictionaries are the best, it would just add overhead.

[zupo]

Anything else we can do here? I'm fine with "no, let's close it".

[mkcdq]

I reached the same conclusion @ikolar did: The best solution is already implemented, in my opinion.

In the end it all boils down to three spell checkers in use today:

• ispell (BSD License)
• aspell (LGPL License)
• hunspell (GPL/LGPL/MPL License)

All the other solutions found pyenchant, nuspell use the above under the hood.

All the above spell checkers seem to use the same Romanian dictionary rospell (maybe different versions of it) and no better dictionary seems to exist at this time.

LibreOffice, Mozilla, Chrome and so, use hunspell because of the license, even though aspell seems to have superior correction capabilities and is faster.

This project already uses a custom dictionary, along the official one, so not much can be done in this direction.

Unless we want to write our own solution, or start using some ai/ml/nltk or so, at this time, I see no real alternative. My only suggestion is to keep an eye on nuspell. Maybe some alternatives emerge.

Redoing @ikolar tests and adding nuspell to the list:

time (cat Countries/Romania/*.html | aspell --encoding=utf-8 --personal=$PWD/.travis/dictionaries/en.pws list | sort | uniq | aspell -d ro list | wc -l)
343

real  0m0,056s
user  0m0,036s
sys   0m0,020s

time (cat Countries/Romania/*.html | aspell --encoding=utf-8 --personal=$PWD/.travis/dictionaries/en.pws list | sort | uniq | hunspell -d ro_RO -l | wc -l)
347

real  0m0,173s
user  0m0,191s
sys   0m0,021s

time (cat Countries/Romania/*.html | aspell --encoding=utf-8 --personal=$PWD/.travis/dictionaries/en.pws list | sort | uniq | nuspell -d ro_RO -l | wc -l)
INFO: I/O  locale name=en_US.UTF-8, lang=en, country=US, enc=utf-8
INFO: Pointed dictionary /usr/share/hunspell/ro_RO.{dic,aff}
347

real  0m0,146s
user  0m0,145s
sys   0m0,032s

Yields the same results of course. Nuspell, using hunspell, gives the same result.

Links:

• Aspell and Hunspell: A Tale of Two Spell Checkers
• Nuspell: the new spell checker
• Romanian.udic
• Hunspell Uses
• MySpell
• How to Write a Spelling Corrector
• Natural Language Toolkit
• polyglot
• SymSpell
• Martha Snow: Spell chequer

[zupo]

As per @ikolar and @mkcdq, there is not a lot of room for improvement here, so I'm closing.