-
-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
online word2vec #900
Merged
Merged
online word2vec #900
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,365 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Online word2vec tutorial\n", | ||
| "\n", | ||
| "So far, word2vec cannot increase the size of vocabulary after initial training. To handle unknown words, not in word2vec vocaburary, you must retrain updated documents over again.\n", | ||
| "\n", | ||
| "In this tutorial, we introduce gensim new feature, online vocaburary update. This additional feature overcomes the unknown word problems. Despite after initial training, we can continuously add new vocaburary to the pre-trained word2vec model using this online feature.\n", | ||
| "\n", | ||
| "This implementation is still beta version at 16/09/04. You can download the beta version of online word2vec implementation in the following repository." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "collapsed": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%%bash\n", | ||
| "git clone -b online-w2v [email protected]:isohyt/gensim.git" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "metadata": { | ||
| "collapsed": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from gensim.corpora.wikicorpus import WikiCorpus\n", | ||
| "from gensim.models.word2vec import Word2Vec, LineSentence\n", | ||
| "from pprint import pprint\n", | ||
| "from copy import deepcopy\n", | ||
| "from multiprocessing import cpu_count" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Download wikipedia dump files\n", | ||
| "\n", | ||
| "We use the past and the current version of wiki dump files as online training." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%%bash\n", | ||
| "wget https://dumps.wikimedia.org/archive/2010/2010-11/enwiki/20101011/enwiki-20101011-pages-articles.xml.bz2\n", | ||
| "wget https://dumps.wikimedia.org/enwiki/20160820/enwiki-20160820-pages-articles.xml.bz2" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Convert two wikipedia dump files\n", | ||
| "To avoid alert when convert old verision of wikipedia dump, you should download alternative wikicorpus.py in my repo." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "old, new = [WikiCorpus('enwiki-{}-pages-articles.xml.bz2'.format(ymd)) for ymd in ['20101011', '20160820']]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "metadata": { | ||
| "collapsed": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "def write_wiki(wiki, name, titles = []):\n", | ||
| " with open('{}.wiki'.format(name), 'wb') as f:\n", | ||
| " wiki.metadata = True\n", | ||
| " for text, (page_id, title) in wiki.get_texts():\n", | ||
| " if title not in titles:\n", | ||
| " f.write(b' '.join(text)+b'\\n')\n", | ||
| " titles.append(title)\n", | ||
| " return titles" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 4, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "old_titles = write_wiki(old, 'old')\n", | ||
| "all_titles = write_wiki(new, 'new', old_titles)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 5, | ||
| "metadata": { | ||
| "collapsed": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "oldwiki, newwiki = [LineSentence(f+'.wiki') for f in ['old', 'new']]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Initial training\n", | ||
| "At first we train word2vec using \"enwiki-20101011-pages-articles.xml.bz2\". After that, we update model using \"enwiki-20160820-pages-articles.xml.bz2\"." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 6, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "CPU times: user 4h 39min 57s, sys: 1min 28s, total: 4h 41min 25s\n", | ||
| "Wall time: 1h 32min 43s\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "%%time\n", | ||
| "model = Word2Vec(oldwiki, min_count = 0, workers=cpu_count())\n", | ||
| "# model = Word2Vec.load('oldmodel')\n", | ||
| "oldmodel = deepcopy(model)\n", | ||
| "oldmodel.save('oldmodel')" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Japanese new idol group, [\"Babymetal\"](https://en.wikipedia.org/wiki/Babymetal), weren't known worldwide in 2010, so that the word, \"babymetal\", is not in oldmodel vocaburary.\n", | ||
| "Note: In recent years, they became the famous idol group not only in Japan. They won many music awards and run world tour." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 7, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "\"word 'babymetal' not in vocabulary\"\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "try:\n", | ||
| " print(oldmodel.most_similar('babymetal'))\n", | ||
| "except KeyError as e:\n", | ||
| " print(e)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Online update\n", | ||
| "To use online word2vec feature, set update=True when you use build_vocab using new documents." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 8, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "CPU times: user 2h 53min 51s, sys: 1min 1s, total: 2h 54min 52s\n", | ||
| "Wall time: 57min 24s\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "%%time\n", | ||
| "model.build_vocab(newwiki, update=True)\n", | ||
| "model.train(newwiki)\n", | ||
| "model.save('newmodel')\n", | ||
| "# model = Word2Vec.load('newmodel')" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Model Comparison\n", | ||
| "By the online training, the size of vocaburaries are increased about 3 millions." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 9, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "The vocabulary size of the oldmodel is 6161170\n", | ||
| "The vocabulary size of the model is 8469444\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "for m in ['oldmodel', 'model']:\n", | ||
| " print('The vocabulary size of the', m, 'is', len(eval(m).vocab))" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### After online training, the word, \"babymetal\", is added in model. This word is simillar with rock and metal bands." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 10, | ||
| "metadata": { | ||
| "collapsed": false | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "[('espairsray', 0.7539531588554382),\n", | ||
| " ('crossfaith', 0.7476214170455933),\n", | ||
| " ('mucc', 0.7363666296005249),\n", | ||
| " ('girugamesh', 0.7309226989746094),\n", | ||
| " ('flumpool', 0.7182492017745972),\n", | ||
| " ('gackt', 0.715751051902771),\n", | ||
| " ('jpop', 0.7055245637893677),\n", | ||
| " ('kuroyume', 0.7049269676208496),\n", | ||
| " ('ellegarden', 0.7018687725067139),\n", | ||
| " ('tigertailz', 0.701062023639679)]\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "try:\n", | ||
| " pprint(model.most_similar('babymetal'))\n", | ||
| "except KeyError as e:\n", | ||
| " print(e)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## The word, \"Zootopia\", become disney movie through the years.\n", | ||
| "In the past, the word, \"Zootopia\", was used just for an annual summer concert put on by New York top-40 radio station Z100, so that the word, \"zootopia\", is simillar with music festival.\n", | ||
| "\n", | ||
| "In 2016, Zootopia is a American 3D computer-animated comedy film released by Walt Disney Pictures. As a result, the word, \"zootopia\", was often used as Animation films." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 11, | ||
| "metadata": { | ||
| "collapsed": false, | ||
| "scrolled": false | ||
| }, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "The count of the word,zootopia, is 24 in oldmodel\n", | ||
| "[('itsekseni', 0.655870258808136),\n", | ||
| " ('baverstam', 0.6502687931060791),\n", | ||
| " ('hachnosas', 0.6450551748275757),\n", | ||
| " ('carrantouhill', 0.631106436252594),\n", | ||
| " ('bugasan', 0.6258121728897095),\n", | ||
| " ('lollapolooza', 0.6192305088043213),\n", | ||
| " ('hutuz', 0.6134281754493713),\n", | ||
| " ('soulico', 0.6122198104858398),\n", | ||
| " ('kabungwe', 0.6060466766357422),\n", | ||
| " ('prischoßhalle', 0.6056506633758545)]\n", | ||
| "\n", | ||
| "The count of the word,zootopia, is 257 in model\n", | ||
| "[('incredibles', 0.7643648386001587),\n", | ||
| " ('antz', 0.7575620412826538),\n", | ||
| " ('spaceballs', 0.7434272766113281),\n", | ||
| " ('pagemaster', 0.730089545249939),\n", | ||
| " ('beetlejuice', 0.7257461547851562),\n", | ||
| " ('coneheads', 0.7239412069320679),\n", | ||
| " ('tarzan', 0.7139339447021484),\n", | ||
| " ('catscratch', 0.7124171257019043),\n", | ||
| " ('boxtrolls', 0.7024375796318054),\n", | ||
| " ('aristocats', 0.7005465030670166)]\n", | ||
| "\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "w = 'zootopia'\n", | ||
| "for m in ['oldmodel', 'model']:\n", | ||
| " print('The count of the word,'+w+', is', eval(m).vocab[w].count, 'in', m)\n", | ||
| " pprint(eval(m).most_similar(w))\n", | ||
| " print('')" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.5.1" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 0 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@isohyt why is this file here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I transform old enwiki dump (2010 in this notebook), Attribution Error was returned as follows.
File "***/lib/python3.5/site-packages/gensim/corpora/wikicorpus.py", line 211, in extract_pages ns = elem.find(ns_path).text AttributeError: 'NoneType' object has no attribute 'text'To avoid this error, I fixed wikicorpus.py.