letter-frequency

This is a python script which tries to identify the language of a text by the frequency of its letter.

See also this Blog post.

This is not in any way an accurate general purpose language identifier, just some hacking around with the table at Wikipedia: Letter frequencies and only applies to very small subset of languages listed there (list at the end of this readme).

It runs through the text, builds the frequency list for all occurring letters and then calculates mean squared error (for all letters) for each language.

Run

Run with python letter_frequency.py, here the five top languages for each text snippet. The script will also generate a graph if matplotlib is installed (optional).

Scores for nl_columbus_short.txt

In de prachtige zeestad Genua, de trotsche bijgenaamd, werd omstreeks het jaar 1435 een knaapje geboren, dat nu in alle landen als Christophorus Columbus bekend is. (...)

• Dutch: MSE 0.0016 | 639.8 points
• German: MSE 0.006 | 166.3 points
• Danish: MSE 0.0076 | 131.6 points
• French: MSE 0.0127 | 78.6 points
• Swedish: MSE 0.0137 | 72.9 points

Scores for it_boccaccio_short.txt

Solone, il cui petto un umano tempio di divina sapienzia fu reputato, e le cui sacratissime leggi sono ancora alli presenti uomini chiara testimonianza dell'antica giustizia, era, (...)

• Italian: MSE 0.0007 | 1503.6 points
• Spanish: MSE 0.0042 | 237.0 points
• French: MSE 0.0057 | 175.9 points
• Esperanto: MSE 0.0068 | 147.4 points
• Portuguese: MSE 0.0072 | 139.4 points

Scores for fi_kivi_short.txt

Jukolan talo, eteläisessä Hämeessä, seisoo erään mäen pohjaisella rinteellä, liki Toukolan kylää. Sen läheisin ympäristö on kivinen tanner, mutta alempana alkaa pellot, joissa, (...)

• Finnish: MSE 0.002 | 491.5 points
• Esperanto: MSE 0.008 | 124.4 points
• Czech: MSE 0.0144 | 69.6 points
• Swedish: MSE 0.0154 | 65.0 points
• Icelandic: MSE 0.0172 | 58.0 points

The three snippets provided are from Project Gutenberg, each one contains only the first 200 lines of the original book.

Languages

Here's a list of languages in the list, which are checked against an example text:

• French
• German
• Spanish
• Portuguese
• Esperanto
• Italian
• Turkish
• Swedish
• Polish
• Dutch
• Danish
• Icelandic
• Finnish
• Czech