Support for BPE vocabs + denoising autoencoder in PyTorch Translate (f…

…acebookresearch#362)

Summary:
Pull Request resolved: facebookresearch#362

Pull Request resolved: pytorch/translate#254

This actually uses the fairseq logic which supports BPE cont / end word marker suffixes.

Reviewed By: xianxl

Differential Revision: D12952766

fbshipit-source-id: 35a1bbc38240e4145bec0fc419f2d0a6a73ae2e5

fairseq/data/noising.py

@@ -83,7 +83,7 @@ def noising(self, x, lengths, dropout_prob=0.1, blank_idx=None):
         assert 0 < dropout_prob < 1
 
         # be sure to drop entire words
-        word_idx = self._get_bpe_word_idx(x)
+        word_idx = self.get_word_idx(x)
         sentences = []
         modified_lengths = []
         for i in range(lengths.size(0)):