Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

word2vec accuracy #10

Open
timothywangdev opened this issue Sep 3, 2015 · 7 comments
Open

word2vec accuracy #10

timothywangdev opened this issue Sep 3, 2015 · 7 comments

Comments

@timothywangdev
Copy link
Owner

Collection of experiment results

  • Data
RC_2015-03 (32.6 GB uncompressed)
  • Training
hujie@hujie-GT70-2PE:~/projects/ml/word2vec$ ./word2vec -binary 1 -save-vocab vocab -negative 5 -size 500 -output output -train ../ConversationalAgent/experiments/word2vec/data/data_generated -min-count 50 -threads 8
Starting training using file ../ConversationalAgent/experiments/word2vec/data/data_generated
Vocab size: 203291
Words in train file: 1830337168
Alpha: 0.000005  Progress: 100.00%  Words/thread/sec: 131.63k
  • Testing
hujie@hujie-GT70-2PE:~/projects/ml/word2vec$ ./compute-accuracy output 30000 < questions-words.txt
capital-common-countries:
ACCURACY TOP1: 30.77 %  (56 / 182)
Total accuracy: 30.77 %   Semantic accuracy: 30.77 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 22.35 %  (40 / 179)
Total accuracy: 26.59 %   Semantic accuracy: 26.59 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 1.85 %  (1 / 54)
Total accuracy: 23.37 %   Semantic accuracy: 23.37 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 8.72 %  (90 / 1032)
Total accuracy: 12.92 %   Semantic accuracy: 12.92 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 94.12 %  (288 / 306)
Total accuracy: 27.10 %   Semantic accuracy: 27.10 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 16.77 %  (156 / 930)
Total accuracy: 23.52 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 16.77 % 
gram2-opposite:
ACCURACY TOP1: 46.50 %  (279 / 600)
Total accuracy: 27.72 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 28.43 % 
gram3-comparative:
ACCURACY TOP1: 89.71 %  (1195 / 1332)
Total accuracy: 45.61 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 56.95 % 
gram4-superlative:
ACCURACY TOP1: 77.25 %  (584 / 756)
Total accuracy: 50.07 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 61.19 % 
gram5-present-participle:
ACCURACY TOP1: 79.83 %  (843 / 1056)
Total accuracy: 54.96 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 65.40 % 
gram6-nationality-adjective:
ACCURACY TOP1: 56.38 %  (446 / 791)
Total accuracy: 55.11 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 64.10 % 
gram7-past-tense:
ACCURACY TOP1: 69.16 %  (1025 / 1482)
Total accuracy: 57.51 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 65.18 % 
gram8-plural:
ACCURACY TOP1: 83.33 %  (880 / 1056)
Total accuracy: 60.30 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 67.57 % 
gram9-plural-verbs:
ACCURACY TOP1: 77.78 %  (546 / 702)
Total accuracy: 61.47 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 68.40 % 
Questions seen / total: 10458 19544   53.51 %

We got low accuracy in several tests(currency, city-in-state, capital-world, capital-common-countries)
@timothywangdev
Copy link
Owner Author

Collection of experiment results

  • Data
RC_2015-03 (32.6 GB uncompressed)
  • Training
ubuntu@ip-172-31-6-ive 5 -size 300 -output output -train ../data_generated -min-count 50 -threads 16 -iter 10 -cbow 0
Starting training using file ../data_generated
Vocab size: 203291
Words in train file: 1830337168
Alpha: 0.000002  Progress: 100.00%  Words/thread/sec: 66.75k
  • Testing
[email protected]$ ./compute-accuracy output 30000 < questions-w 
capital-common-countries:
ACCURACY TOP1: 46.15 %  (84 / 182)
Total accuracy: 46.15 %   Semantic accuracy: 46.15 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 46.37 %  (83 / 179)
Total accuracy: 46.26 %   Semantic accuracy: 46.26 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 3.70 %  (2 / 54)
Total accuracy: 40.72 %   Semantic accuracy: 40.72 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 13.47 %  (139 / 1032)
Total accuracy: 21.29 %   Semantic accuracy: 21.29 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 92.81 %  (284 / 306)
Total accuracy: 33.77 %   Semantic accuracy: 33.77 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 23.01 %  (214 / 930)
Total accuracy: 30.04 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 23.01 % 
gram2-opposite:
ACCURACY TOP1: 53.83 %  (323 / 600)
Total accuracy: 34.39 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 35.10 % 
gram3-comparative:
ACCURACY TOP1: 91.07 %  (1213 / 1332)
Total accuracy: 50.75 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 61.15 % 
gram4-superlative:
ACCURACY TOP1: 91.14 %  (689 / 756)
Total accuracy: 56.43 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 67.41 % 
gram5-present-participle:
ACCURACY TOP1: 77.84 %  (822 / 1056)
Total accuracy: 59.95 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 69.77 % 
gram6-nationality-adjective:
ACCURACY TOP1: 75.47 %  (597 / 791)
Total accuracy: 61.65 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 70.59 % 
gram7-past-tense:
ACCURACY TOP1: 62.42 %  (925 / 1482)
Total accuracy: 61.78 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 68.85 % 
gram8-plural:
ACCURACY TOP1: 82.20 %  (868 / 1056)
Total accuracy: 63.99 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 70.61 % 
gram9-plural-verbs:
ACCURACY TOP1: 83.76 %  (588 / 702)
Total accuracy: 65.32 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 71.67 % 
Questions seen / total: 10458 19544   53.51 %

We got low accuracy in several tests(currency, city-in-state, capital-world, capital-common-countries)

@bwuu
Copy link
Collaborator

bwuu commented Sep 5, 2015

What machines are you using to train? My laptop isn't really cutting it. Like 0.5%/hr ...

@timothywangdev
Copy link
Owner Author

i7 4800 MQ or ec2 c4.4xlarge

@bwuu
Copy link
Collaborator

bwuu commented Sep 6, 2015

  • Data
RC_2015-01  (31G uncompressed)

But corpus was generated without any splitting of comments, so each comment's body was fully on one line.

  • Train
time ./word2vec -train data_generated -output vectors.bin -cbow 1 -size 200 -window 5 -negative 5 -hs 0 -threads 4 -binary 1 -iter 5 -min-count 50
  • Testing
osboxes@osboxes:~/Desktop/word2vec/trunk$ ./compute-accuracy vectors.bin 30000 < questions-words.txt 
capital-common-countries:
ACCURACY TOP1: 42.86 %  (90 / 210)
Total accuracy: 42.86 %   Semantic accuracy: 42.86 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 39.29 %  (77 / 196)
Total accuracy: 41.13 %   Semantic accuracy: 41.13 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 8.57 %  (6 / 70)
Total accuracy: 36.34 %   Semantic accuracy: 36.34 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 9.92 %  (84 / 847)
Total accuracy: 19.43 %   Semantic accuracy: 19.43 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 88.89 %  (272 / 306)
Total accuracy: 32.47 %   Semantic accuracy: 32.47 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 17.85 %  (166 / 930)
Total accuracy: 27.16 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 17.85 % 
gram2-opposite:
ACCURACY TOP1: 44.33 %  (266 / 600)
Total accuracy: 30.42 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 28.24 % 
gram3-comparative:
ACCURACY TOP1: 87.69 %  (1168 / 1332)
Total accuracy: 47.41 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 55.90 % 
gram4-superlative:
ACCURACY TOP1: 81.18 %  (755 / 930)
Total accuracy: 53.20 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 62.10 % 
gram5-present-participle:
ACCURACY TOP1: 82.29 %  (869 / 1056)
Total accuracy: 57.94 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 66.50 % 
gram6-nationality-adjective:
ACCURACY TOP1: 52.83 %  (448 / 848)
Total accuracy: 57.35 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 64.47 % 
gram7-past-tense:
ACCURACY TOP1: 66.53 %  (986 / 1482)
Total accuracy: 58.90 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 64.89 % 
gram8-plural:
ACCURACY TOP1: 75.47 %  (797 / 1056)
Total accuracy: 60.67 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 66.25 % 
gram9-plural-verbs:
ACCURACY TOP1: 80.77 %  (567 / 702)
Total accuracy: 62.01 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 67.39 % 
Questions seen / total: 10565 19544   54.06 %

@timothywangdev
Copy link
Owner Author

looks like it's got a higher semantic accuracy, not sure if it's due to a larger word vector dimension. I'm training a much larger dataset right now (entire 2015 dataset), hopefully we will get a better semantic accuracy.

@bwuu
Copy link
Collaborator

bwuu commented Sep 8, 2015

1tb on ec2? isnt the storage alone pretty expensive?

@timothywangdev
Copy link
Owner Author

  • Data
RC_2015-01,02,03,04,05
  • Training
hehe@hehe-Q400A:~/projects/ml/word2vec$ ./word2vec -binary 1 -save-vocab vocab -negative 3 -size 800 -output output -train ../ConversationalAgent/experiments/word2vec/data/data_generated -min-count 100 -threads 8 -iter 5
Starting training using file ../ConversationalAgent/experiments/word2vec/data/data_generated
Vocab size: 360272
Words in train file: 8921611451
Alpha: 0.000036  Progress: 99.93%  Words/thread/sec: 119.62k
  • Testing
hehe@hehe-Q400A:~/projects/ml/word2vec$ ./compute-accuracy output 30000 < questions-words.txt 
capital-common-countries:
ACCURACY TOP1: 45.60 %  (83 / 182)
Total accuracy: 45.60 %   Semantic accuracy: 45.60 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 39.66 %  (71 / 179)
Total accuracy: 42.66 %   Semantic accuracy: 42.66 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 0.00 %  (0 / 54)
Total accuracy: 37.11 %   Semantic accuracy: 37.11 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 17.55 %  (156 / 889)
Total accuracy: 23.77 %   Semantic accuracy: 23.77 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 97.06 %  (297 / 306)
Total accuracy: 37.70 %   Semantic accuracy: 37.70 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 21.40 %  (199 / 930)
Total accuracy: 31.73 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 21.40 % 
gram2-opposite:
ACCURACY TOP1: 49.83 %  (299 / 600)
Total accuracy: 35.19 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 32.55 % 
gram3-comparative:
ACCURACY TOP1: 91.22 %  (1215 / 1332)
Total accuracy: 51.88 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 59.85 % 
gram4-superlative:
ACCURACY TOP1: 85.85 %  (649 / 756)
Total accuracy: 56.79 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 65.28 % 
gram5-present-participle:
ACCURACY TOP1: 84.09 %  (888 / 1056)
Total accuracy: 61.38 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 69.53 % 
gram6-nationality-adjective:
ACCURACY TOP1: 65.11 %  (515 / 791)
Total accuracy: 61.80 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 68.89 % 
gram7-past-tense:
ACCURACY TOP1: 71.93 %  (1066 / 1482)
Total accuracy: 63.55 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 69.54 % 
gram8-plural:
ACCURACY TOP1: 87.78 %  (927 / 1056)
Total accuracy: 66.21 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 71.95 % 
gram9-plural-verbs:
ACCURACY TOP1: 80.34 %  (564 / 702)
Total accuracy: 67.17 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 72.62 % 
Questions seen / total: 10315 19544   52.78 %

Not a significant improvement

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bwuu @timothywangdev