Ignore duplicate entries in dict.txt (and allow explicit special symb…

…s) (#1063)

Summary: Pull Request resolved: fairinternal/fairseq-py#1063

Differential Revision: D20248701

Pulled By: myleott

fbshipit-source-id: 5701539e8728816b2e17382788908fd189c5d0e1

fairseq/data/dictionary.py

@@ -223,9 +223,7 @@ def add_from_file(self, f):
                 )
             word = line[:idx]
             count = int(line[idx + 1 :])
-            self.indices[word] = len(self.symbols)
-            self.symbols.append(word)
-            self.count.append(count)
+            self.add_symbol(word, n=count)
 
     def _save(self, f, kv_iterator):
         if isinstance(f, str):