This is a simple example of how n-grams and cosine similarity can be used to 
statistically detect language. The system is designed to work with Twitter,
and has rudimentary support for twitter metadata and formatting. However, the
technique could be easily adopted to other domains.

Since each tweet has an associated user profile language, the application of
guessing what language a tweet is in is rather academic. The more practical use
is for detecting when a tweet is not written in the designated profile language,
which actually happens quite often. The idea would be to gather tweets from a
single profile language, with the assumption being that the majority are, in
fact, in that language. The similarity metric can be used to identify outliers.

There are two CLI python scripts: train.py and detect.py. train.py is run over
a set of json-formatted tweet files, each file corresponding to one profile
language, and produces a language model. detect.py uses the model to predict the
language of new tweets. The language detection is done purely on the text of the
tweet, without any help from the tweet metadata. The following represents sample
usage of the scripts.

    $ ./train.py en.json de.json ja.json 
    processing en.json...
    processing de.json...
    processing ja.json...
    model written to langid.model
    $ ./detect.py langid.model "Go #Giants! Beat the #Tigers"
    en

For more background, see the blog posts and demos at:

    http://blog.bbdouglas.com