Faster analogies

[sebastien-j]

Speed up analogy-making, generally by an order of magnitude or more.

Some minor issues:

• If you want to use 3CosMul or some other objective, you need to call the function differently.
• section['correct'] and section['incorrect'] only store the number of good/bad answers instead of the questions.

[gojomo]

Nice! I got a 13x speedup in my try.

But, in Python3, the line num_batches = num_questions/batchsize + bool(num_questions%batchsize) gives a float which breaks the following xrange(num_batches). Changing that line's division operator to // to force integer-division seemed enough to fix, and should work in Python2 as well.

(You should be able to use from __future__ import division if you want to reproduce the issue in Python2.)

[piskvorky]

Great!

Are we good to merge? The logic seems pretty complex -- can you add some basic sanity checks, in unit tests?

[sebastien-j]

I trained a very small model and evaluated the accuracy with the current version of Gensim.
I made a test verifying that we still obtain the same results.

Are there any licensing issues if I use the word2vec analogy questions?

[sebastien-j]

The test fails on python 3.x. Is there a simple way to load a model pickled with python 2.x?

[cscorley]

@sebastien-j seems to be an encoding problem. You can specify the encoding with pickle.load: https://docs.python.org/3/library/pickle.html#pickle.load

[gojomo]

I like the idea of a small loadable model for sanity testing, including across versions, but a pickled version has problems, both for this sort of 2-to-3 encoding issue, and because it's somewhat opaque to easy review yet might trigger arbitrary code on unpickling.

Two ideas: (1) use the C word2vec format; or (2) train up a model on one of the already included corpuses. (As long as the corpus gets at least one analogy question right, and the same before/after changes, it seems an OK sanity check.)

It looks like the test/test_data/lee_background.cor is enough to get 2 questions right:

sentences = LineSentence('tests/test_data/lee_background.cor')
model = Word2Vec(sentences, min_count=1)
model.accuracy('tests/test_data/questions-words.txt')  # gets 2/1258 right for me

[piskvorky]

@sebastien-j some analogy speed up has been implemented in #458.

It doesn't including batching though, and it doesn't include your additions around various similarity functions.

Can you revive this PR (rebase on develop, add tests?)

[tmylk]

@sebastien-j Should we plan this for the February release?

[bloody76]

Wouldn't it be clearer if you put reverse=True: sorted(iteritems(self.vocab), key=lambda item: item[1].count, reverse=True) ?

It's more verbose but the intention is clear at least

[menshikh-iv]

Ping @sebastien-j

[menshikh-iv]

Close because it is abandoned