Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing PR 1326 and providing some tests for unicode wiki corpora #1333

Merged
merged 6 commits into from
May 27, 2017
Merged

Fixing PR 1326 and providing some tests for unicode wiki corpora #1333

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b560052
Fixing PR 1326 and providing some tests for unicode wiki corpora
alekol May 17, 2017
6e69ed9
Fixing old test that was using non-unicode string literal
alekol May 17, 2017
eb6a3cb
Reverting changes to word2vec
alekol May 18, 2017
65b4fa6
Renaming test method to force travis build
alekol May 18, 2017
8208713
Dropping 2.6 checks
alekol May 18, 2017
bb9dd96
Trying to limit the # of processes used to stabilize the test on py 2…
alekol May 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gensim/corpora/wikicorpus.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ def tokenize(content):
"""
# TODO maybe ignore tokens with non-latin characters? (no chinese, arabic, russian etc.)
return [
token.encode('utf8') for token in utils.tokenize(content, lower=True, errors='ignore')
utils.to_unicode(token) for token in utils.tokenize(content, lower=True, errors='ignore')
if 2 <= len(token) <= 15 and not token.startswith('_')
]

Expand Down
Binary file added gensim/test/test_data/bgwiki-latest-pages-articles-shortened.xml.bz2
View file
Open in desktop
Binary file not shown.
16 changes: 14 additions & 2 deletions gensim/test/test_wikicorpus.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
module_path = os.path.dirname(__file__) # needed because sample data files are located in the same folder
datapath = lambda fname: os.path.join(module_path, 'test_data', fname)
FILENAME = 'enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2'
FILENAME_U = 'bgwiki-latest-pages-articles-shortened.xml.bz2'

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -50,9 +51,20 @@ def test_first_element(self):
wc = WikiCorpus(datapath(FILENAME))

l = wc.get_texts()
self.assertTrue(b"anarchism" in next(l))
self.assertTrue(b"autism" in next(l))
self.assertTrue(u'anarchism' in next(l))
self.assertTrue(u'autism' in next(l))

def test_first_element_unicode(self):
"""
First unicode article in this sample is
1) папа
"""
if sys.version_info < (2, 7, 0):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we dropped support for 2.6 , we can also remove this line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll clean this up too

return
wc = WikiCorpus(datapath(FILENAME_U))

l = wc.get_texts()
self.assertTrue(u'папа' in next(l))

if __name__ == '__main__':
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
Expand Down