README

Modify your bashrc
# Add custom Python modules to the Python path.
PYTHONPATH=$PYTHONPATH:/path/to/your/lydia/
export PYTHONPATH

you can change ~/sandbox/ to wherever you are keeping the code


To get a dump of arabic wikipedia
http://download.wikimedia.org/arwiki/

the one that I downloaded
    |wget -c http://download.wikimedia.org/arwiki/20110121/arwiki-20110121-pages-articles.xml.bz2

To fragment the wikipedia dump to different xml files, each contain one article
    |$wikipedia/fragmenter.py wikiDump.xml outputDirectory
The outputDirectory will contain all the files. The files are named after the ids of the wikipedia articles.

In each xml file, the text is contained between the <text></text> tags. Moreover, the text available with the wiki markup language
doStrip contains most of the formatting need to remove the wiki markup language.

To extract the text and remove the markup language.
$wikipedia/formatter.py inputFile
This will generate an output file called inputFile.f

$aner/NER.py folderName
It will run over all the files that have .f extension, it will delete all the temp files except .bw.post.NER file which has five columns. The stats will will be stored in folderName/stats

$aner/NER filename
It will run on the file passed and make .bw.ner file, it will not delete any temp files.