Easy import of GloVe vectors using Gensim #625

manasRK · 2016-03-09T14:12:33Z

word2vec embeddings start with a line with the number of lines (tokens?) and the number of dimensions of the file. This allows gensim to allocate memory accordingly for querying the model. Larger dimensions mean larger memory is held captive. Accordingly, this line has to be inserted into the GloVe embeddings file.

tmylk · 2016-03-13T09:36:35Z

@manasRK Thanks for the PR.

It would be better to have this converter as a standalone script with usage like this:
python -m gensim.scripts.convert_glove_to_gensim glove_file.txt gensim_file.txt

If you could add a test for a small file <100Kb using check_output that would be great

piskvorky · 2016-03-13T12:54:34Z

glove-word2vec-gensim.py

+	Function use to prepend lines using bash utilities in Linux. 
+	(source: http://stackoverflow.com/a/10850588/610569)
+	"""
+	with open(infile, 'r') as old:


Use binary mode: rb, wb. Otherwise won't work on Windows.

piskvorky · 2016-03-13T12:57:09Z

This PR has code style issue (indentation, spaces around binary operators etc -- see PEP8).

I'm not sure I understand its purpose -- does it insert one line at the beginning of a file? Isn't it easier to just use cat? Or what is the advantage of using this script?

piskvorky · 2016-03-13T12:59:46Z

Also, it seems to only work for a few hard-coded filenames (reads the dimensionality from filename?).

Isn't that too constraining / narrow purpose?

manasRK · 2016-03-13T14:43:04Z

Agreed @piskvorky . The major motivation to create this code was to allow easy port of GloVe vectors into Gensim. I was already using Gensim massively in my explorations with word2vec. I didn't want to write another code for GloVe separately. I found out later that there is a minor difference in the formats and hence wanted to make it Gensim-compatible.

Not a big fan of 'hard-coding' either, but this is something that was suggested in a Github pull request earlier in my repositories. My earlier approach was to calculate thenum_lines and dims on the fly to create a new model file, something like this;

#Model File
fname="glove.6B.50d.txt" #GloVe models

#extract information to pre-pend to the final Gensim compatible file
word2vec_convert_file="word2vec_line.txt"
num_lines = sum(1 for line in open(fname))
dims= re.findall('\d+',fname.split('.')[2])

print '%d lines with %d dimensions' %(num_lines,dims)

with open(word2vec_convert_file,'wb') as f:
    f.write(str(num_lines)+ " " +str(dims) + '\n')
f.close()

model_file='glove_model.txt' #for Gensim's consumption

filenames = [word2vec_convert_file,fname]

with open(model_file, 'wb') as outfile:
    for fname in filenames:
        with open(fname) as infile:
            for line in infile:
                outfile.write(line)
outfile.close()

Might be a bit in-efficient, but would love your suggestions to make it useful for guys who want to use Gensim for both GloVe and word2vec, without writing more code.

@tmylk I will add a test file soon as well. Thanks !

piskvorky · 2016-03-13T15:56:57Z

Sure. I think we can add such CLI script to convert glove => word2vec, it looks useful.

The script should accept the two filenames (glove, gensim) as CLI arguments, then use smart_open to do the actual writing/reading:

python -m gensim.scripts.glove2word2vec glove_vectors.txt word2vec_vectors.txt

In other words, rather than an ad-hoc script with print statements for a few hardcoded files, we definitely want this to be a generic script for any glove input.

manasRK · 2016-03-13T16:40:45Z

Great, I will edit my script to do the same. Make it a CLI file, which can take in arguments as you suggested and create a file to consume in Gensim.

piskvorky · 2016-03-14T04:14:38Z

Thanks @manasRK .

Have a look at other scripts in the /gensim/scripts directory for inspiration and code style. Best of luck!

CLI USAGE: python glove2word2vec.py <GloVe vector file> <Output model file> Convert GloVe vectors into Gensim compatible format to instantiate from an existing file on disk in the word2vec C format; model = gensim.models.Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False) # C text format word2vec embeddings start with a line with the number of lines (tokens?) and the number of dimensions of the file. This allows gensim to allocate memory accordingly for querying the model. Larger dimensions mean larger memory is held captive. Accordingly, this line has to be inserted into the GloVe embeddings file.

manasRK · 2016-03-14T08:25:10Z

I have edited the code for use in commandline in current form as

python glove2word2vec.py <GloVe vector file> <Output model file>

Further suggestions welcome. :)

piskvorky · 2016-03-14T09:02:02Z

glove2word2vec.py

+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2016 Manas Ranjan Kar <[email protected]>
+# Licensed under the MIT License https://opensource.org/licenses/MIT


Gensim uses LGPL (see headers of other files).

@piskvorky

Updated changes with @piskvorky 's suggestions.

piskvorky · 2016-03-14T09:12:08Z

Good progress :) I added a few more comments around Pythonic code style.

More suggestions integrated. Hopefully, didn't miss anything.

@piskvorky

Updated with more suggestions from @piskvorky

manasRK · 2016-03-15T07:48:52Z

@piskvorky I have updated the file with your recommended changes.

piskvorky · 2016-03-15T09:09:54Z

glove2word2vec.py

+
+    num_lines, dims= get_info(glove_vector_file)
+    gensim_first_line = "{} {}".format(num_lines, dims)
+    model_file=prepend_line(glove_vector_file, output_model_file, gensim_first_line)


PEP8: Spaces around binary operator (here, above, below).

manasRK · 2016-03-24T18:55:19Z

@tmylk Finally managed to get rid of string literal errors with Python 3. And I officially hate Python 3 now. My PR is ready to be merged . :)

Helpful link for anyone who follows this thread later, difference between Python 3 and Python 2.

http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

tmylk · 2016-03-24T19:45:55Z

Great job. Thanks a lot. Did you run pep8 on it?

manasRK · 2016-03-24T20:51:06Z

Yes, checked with PEP8 and PEP257 python packages as well.

piskvorky · 2016-03-25T01:30:20Z

setup.py

@@ -114,6 +104,11 @@ def readfile(fname):
    import wheelhouse_uploader.cmd
    cmdclass.update(vars(wheelhouse_uploader.cmd))

+python_2_6_backports = ''


What does an "empty string" inside setup's install_requires actually do? Is that ok?

This comes from #634 . @tmylk can shed some light on this.

We install argparse only if Python version is 2.6. By default we don't need to install any backports, hence an empty string.

Question was, what does an empty string imply in install_requires. Ignored? Warning? Exception?

The empty string doesn't cause any output in Travis.
https://travis-ci.org/piskvorky/gensim/jobs/118421963#L290

piskvorky · 2016-03-25T05:23:57Z

setup.py

@@ -6,7 +6,6 @@

 """
 Run with:
-


@manasRK I still see many such deleted documentation lines in the diff (top tab, "Files changed").

Re-added deleted blank lines.

manasRK · 2016-03-28T07:19:41Z

@tmylk @piskvorky Codes are now okay? Will close the PR if you are ready to merge :)

piskvorky · 2016-03-28T11:48:33Z

Yes, this is in @tmylk 's hands now, thanks @manasRK !

tmylk · 2016-03-28T17:46:49Z

gensim/scripts/glove2word2vec.py

+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2016 Radim Rehurek <[email protected]>
+# Copyright (C) 2016 Manas Ranjan Kar <[email protected]>


Just checking that you are aware of the legal guidelines of contributing. "By submitting your contribution to be included in the gensim project, you agree to assign joint ownership of your changes (your code patch, documentation fix, whatever) to me, Radim Řehůřek."

https://github.com/piskvorky/gensim/wiki/Developer-page#legal

Yes, surely. That's why I added @piskvorky 's name in the code.

tmylk · 2016-03-28T18:01:55Z

Please add note about this new feature to changelog file.
Also see comments about copyright and removing word2vec_standalone.

If you resubmit the PR against the develop branch then it will be merged in. (Master branch is always the latest pypi release.)

Also please squash the commits into one using rebase command.

piskvorky · 2016-03-29T02:41:34Z

My notes:

Doc fixes in word2vec_standalone are fine. Joint copyright ownership also fine.

If the PR against develop branch or squashing the commits gives you trouble @manasRK, @tmylk can help you. Let's get this merged ASAP.

tmylk · 2016-03-29T09:42:23Z

@manasRK Happy to talk in https://gitter.im/piskvorky/gensim if needed

Rebasing piskvorky#625

tmylk · 2016-04-11T15:52:25Z

Merged in 982dc3c

tmylk assigned gojomo Mar 13, 2016

tmylk assigned tmylk and unassigned gojomo Mar 13, 2016

piskvorky reviewed Mar 13, 2016
View reviewed changes

piskvorky reviewed Mar 14, 2016
View reviewed changes

Update glove2word2vec.py

7a5e9e8

Updated changes with @piskvorky 's suggestions.

manasRK added 2 commits March 14, 2016 15:29

Update glove2word2vec.py

279cd00

More suggestions integrated. Hopefully, didn't miss anything.

Update glove2word2vec.py

a304469

Updated with more suggestions from @piskvorky

piskvorky reviewed Mar 15, 2016
View reviewed changes

manasRK added 5 commits March 24, 2016 23:36

All checks done on Py2 and Py3

e616f87

Update for change in string literals in python 3

6b34a24

Update glove2word2vec.py

ce4a00a

Now backporting functions for Py2.6

20dff12

Reverting to Final commit

66a6744

piskvorky reviewed Mar 25, 2016
View reviewed changes

Making text readable

d90f9d2

piskvorky reviewed Mar 25, 2016
View reviewed changes

manasRK added 2 commits March 25, 2016 11:09

Re-adding deleted blank lines

481d6fa

Update setup.py

6f8365e

piskvorky added the feature Issue described a new feature label Mar 28, 2016

tmylk reviewed Mar 28, 2016
View reviewed changes

manasRK added 2 commits March 29, 2016 14:15

Update CHANGELOG.txt

870b539

Rerun build

cd0097f

manasRK mentioned this pull request Mar 29, 2016

Rebasing #625 manasRK/gensim#2

Merged

manasRK added a commit to manasRK/gensim that referenced this pull request Mar 29, 2016

Merge pull request #2 from manasRK/manas-develop

a09d2f9

Rebasing piskvorky#625

manasRK mentioned this pull request Mar 29, 2016

Rebasing #625 #644

Closed

tmylk added a commit that referenced this pull request Apr 11, 2016

GloVe conversion script #625

dfc8552

tmylk closed this Apr 11, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Easy import of GloVe vectors using Gensim #625

Easy import of GloVe vectors using Gensim #625

manasRK commented Mar 9, 2016

tmylk commented Mar 13, 2016

piskvorky Mar 13, 2016

piskvorky commented Mar 13, 2016

piskvorky commented Mar 13, 2016

manasRK commented Mar 13, 2016

piskvorky commented Mar 13, 2016

manasRK commented Mar 13, 2016

piskvorky commented Mar 14, 2016

manasRK commented Mar 14, 2016

piskvorky Mar 14, 2016

piskvorky commented Mar 14, 2016

manasRK commented Mar 15, 2016

piskvorky Mar 15, 2016

manasRK commented Mar 24, 2016

tmylk commented Mar 24, 2016

manasRK commented Mar 24, 2016

piskvorky Mar 25, 2016

manasRK Mar 25, 2016

tmylk Mar 25, 2016

piskvorky Mar 25, 2016

tmylk Mar 25, 2016

piskvorky Mar 25, 2016

manasRK Mar 25, 2016

manasRK commented Mar 28, 2016

piskvorky commented Mar 28, 2016

tmylk Mar 28, 2016

manasRK Mar 28, 2016

tmylk commented Mar 28, 2016

piskvorky commented Mar 29, 2016

tmylk commented Mar 29, 2016

tmylk commented Apr 11, 2016

Easy import of GloVe vectors using Gensim #625

Easy import of GloVe vectors using Gensim #625

Conversation

manasRK commented Mar 9, 2016

tmylk commented Mar 13, 2016

Choose a reason for hiding this comment

piskvorky commented Mar 13, 2016

piskvorky commented Mar 13, 2016

manasRK commented Mar 13, 2016

piskvorky commented Mar 13, 2016

manasRK commented Mar 13, 2016

piskvorky commented Mar 14, 2016

manasRK commented Mar 14, 2016

Choose a reason for hiding this comment

piskvorky commented Mar 14, 2016

manasRK commented Mar 15, 2016

Choose a reason for hiding this comment

manasRK commented Mar 24, 2016

tmylk commented Mar 24, 2016

manasRK commented Mar 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manasRK commented Mar 28, 2016

piskvorky commented Mar 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmylk commented Mar 28, 2016

piskvorky commented Mar 29, 2016

tmylk commented Mar 29, 2016

tmylk commented Apr 11, 2016