Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement shrink_windows argument for Word2Vec. #3169

Merged
merged 14 commits into from
Jun 29, 2021
Merged

Implement shrink_windows argument for Word2Vec. #3169

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
6afb3ff
Implemented `reduced_windows` argument for Word2Vec.
pandrey-fr Jun 7, 2021
9bdb5a1
Improve the way `reduced_windows` is passed around and used.
pandrey-fr Jun 10, 2021
77698cd
Renamed `reduced_windows` to `shrink_windows`.
pandrey-fr Jun 10, 2021
11a464f
Removed `shrink_windows` argument from `Word2Vec.train`.
pandrey-fr Jun 11, 2021
7b6e73a
Merge branch 'develop' into fixed-winsize
M-Demay Jun 14, 2021
a657518
Aesthetic fix.
pandrey-fr Jun 15, 2021
ff0d30e
Fixed old word2vec models' reloading.
pandrey-fr Jun 15, 2021
f98eb43
Fixed undue docstring.
pandrey-fr Jun 15, 2021
3b27f1c
Added `shrink_windows` argument to Doc2Vec.
pandrey-fr Jun 15, 2021
27a49d7
Added `shrink_windows` argument to FastText.
pandrey-fr Jun 15, 2021
6fde7e8
Fixed and optimized `shrink_windows` backend use.
pandrey-fr Jun 17, 2021
486b3f5
Added tests for `shrink_windows=False` in Word2Vec-based models.
pandrey-fr Jun 17, 2021
f0af84e
Added docstring mentions of `shrink_window` being experimental.
pandrey-fr Jun 17, 2021
49cf50e
Rolled back some purely aesthetic changes.
pandrey-fr Jun 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions gensim/models/doc2vec_corpusfile.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,11 @@ cdef void prepare_c_structures_for_batch(
document_len[0] = i

if train_words and reduced_windows != NULL:
for i in range(document_len[0]):
if shrink_windows:
if shrink_windows:
for i in range(document_len[0]):
reduced_windows[i] = random_int32(next_random) % window
else:
else:
for i in range(document_len[0]):
reduced_windows[i] = 0

if doc_tag < docvecs_count:
Expand Down
8 changes: 5 additions & 3 deletions gensim/models/doc2vec_inner.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,12 +364,13 @@ def train_document_dbow(model, doc_words, doctag_indexes, alpha, work=None,
c.document_len = i

if c.train_words:
# single randint() call avoids a big thread-synchronization slowdown
if model.shrink_windows:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit repetitive.

Wouldn't it be better to define a model.initialize_reduced_windows method that does this, and call it when needed?

def initialize_reduced_windows(self, ...):
  if model.shrink_windows:
    for i, item in ...:
        ...
  else:
    ...

That way, if we find a logic error in the initialization code, we don't have to remember to fix it in a half a dozen other places (e.g. https://github.com/RaRe-Technologies/gensim/pull/3169/files#r649243075).

We should probably also do a bounds check on c.reduced_windows to ensure that it's at least document_len elements long.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you on the potential for refactored code that would ease future maintenance and updates. That being said, I actually built on the existing, repetitive code, so in my humble opinion this could be handled as a distinct PR (this one would add a feature, while a second one would enhance the validated, working code).

I also think that your suggestion of a model.initialize_reduced_windows is not entirely suitable here:
If I am not mistaken, in this code, c is not model: it is a special C structure (of a different class depending on the trained model class: word2vec, doc2vec or fasttext), and the reduced_windows PyArray is initialized to have the proper size, so there should not be a need to do a bounds check.
I do not know how we could implement a common method (or function) that would also fill the array with either deterministic or random values; but I do agree with you that it could be worth it.

To be honest I lack experience with cython, so that I do not feel entirely at ease with diving into this refactoring. I am willing to give it a try at some point if you want, but would prefer it to be a distinct effort (and PR) from the current one - partly because Mathis and I are pushing the shrink_windows feature due to our needing it for a current research project at work, while contributing to refactoring the code base would be something I would (willingly) do on my time off.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alas, there's a lot of pre-existing cut & paste duplication between all these related algorithms & modes that could be refactored. I think that'd be OK to note-and-defer-for-later, as either an issue or a FIXME comment in the code.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, we can deal with this separately.

# single randint() call avoids a big thread-synchronization slowdown
for i, item in enumerate(model.random.randint(0, c.window, c.document_len)):
c.reduced_windows[i] = item
else:
c.reduced_windows[:] = int(0)
piskvorky marked this conversation as resolved.
Show resolved Hide resolved
for i in range(c.document_len):
c.reduced_windows[i] = 0

for i in range(c.doctag_len):
c.doctag_indexes[i] = doctag_indexes[i]
Expand Down Expand Up @@ -504,7 +505,8 @@ def train_document_dm(model, doc_words, doctag_indexes, alpha, work=None, neu1=N
for i, item in enumerate(model.random.randint(0, c.window, c.document_len)):
c.reduced_windows[i] = item
else:
c.reduced_windows[:] = int(0)
for i in range(c.document_len):
c.reduced_windows[i] = 0

for i in range(c.doctag_len):
c.doctag_indexes[i] = doctag_indexes[i]
Expand Down
7 changes: 4 additions & 3 deletions gensim/models/fasttext_corpusfile.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,11 @@ cdef void prepare_c_structures_for_batch(
break

# precompute "reduced window" offsets in a single randint() call
for i in range(effective_words[0]):
if shrink_windows:
if shrink_windows:
for i in range(effective_words[0]):
reduced_windows[i] = random_int32(next_random) % window
else:
else:
for i in range(effective_words[0]):
reduced_windows[i] = 0


Expand Down
3 changes: 2 additions & 1 deletion gensim/models/fasttext_inner.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -605,7 +605,8 @@ def train_batch_any(model, sentences, alpha, _work, _neu1):
for i, randint in enumerate(model.random.randint(0, c.window, num_words)):
c.reduced_windows[i] = randint
else:
c.reduced_windows[:] = int(0)
for i in range(num_words):
c.reduced_windows[i] = 0

# release GIL & train on all sentences in the batch
with nogil:
Expand Down
7 changes: 4 additions & 3 deletions gensim/models/word2vec_corpusfile.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,10 +226,11 @@ cdef void prepare_c_structures_for_batch(
break # TODO: log warning, tally overflow?

# precompute "reduced window" offsets in a single randint() call
for i in range(effective_words[0]):
if shrink_windows:
if shrink_windows:
for i in range(effective_words[0]):
reduced_windows[i] = random_int32(next_random) % window
M-Demay marked this conversation as resolved.
Show resolved Hide resolved
else:
else:
for i in range(effective_words[0]):
reduced_windows[i] = 0


Expand Down
6 changes: 4 additions & 2 deletions gensim/models/word2vec_inner.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -570,7 +570,8 @@ def train_batch_sg(model, sentences, alpha, _work, compute_loss):
for i, item in enumerate(model.random.randint(0, c.window, effective_words)):
c.reduced_windows[i] = item
else:
c.reduced_windows[:] = int(0)
for i in range(effective_words):
c.reduced_windows[i] = 0

# release GIL & train on all sentences
with nogil:
Expand Down Expand Up @@ -669,7 +670,8 @@ def train_batch_cbow(model, sentences, alpha, _work, _neu1, compute_loss):
for i, item in enumerate(model.random.randint(0, c.window, effective_words)):
c.reduced_windows[i] = item
else:
c.reduced_windows[:] = int(0)
for i in range(effective_words):
c.reduced_windows[i] = 0

# release GIL & train on all sentences
with nogil:
Expand Down