diff --git a/blog/_posts/2015-09-21-rmva.md b/blog/_posts/2015-09-21-rmva.md index 68fc0c6..edcccc4 100644 --- a/blog/_posts/2015-09-21-rmva.md +++ b/blog/_posts/2015-09-21-rmva.md @@ -366,9 +366,13 @@ Here are some results for the Translated MNIST dataset : For this dataset, the images are of size `1x60x60` where each image contains a randomly placed `1x28x28` MNIST digit. The `3x12x12` glimpse uses a depth of 3 scales where each successive patch is twice the height and width of the previous one. -Training with this dataset was started about 3 days prior to this blog post. -For 7 glimpses, after 193 epochs, we get 1.223% error. Note that the model is still training. -The paper gets 1.22% and 1.2% error for 6 and 8 glimpses, respectively. +After 683 epochs of training on the Translatted MNIST dataset, using 7 glimpses, we obtain 0.92% error. +The paper reaches 1.22% and 1.2% error for 6 and 8 glimpses, respectively. +The exact command used to obtain those results: + +```lua +th examples/recurrent-visual-attention.lua --cuda --dataset TranslatedMnist --unitPixels 26 --learningRate 0.001 --glimpseDepth 3 --maxTries 200 --stochastic --glimpsePatchSize 12 +``` Note : you can evaluate your models with the [evaluation script](https://github.com/Element-Research/rnn/blob/master/scripts/evaluate-rva.lua). It will generate a sample of glimpse sequences and print the confusion matrix results for the test set. diff --git a/blog/_posts/2016-07-25-nce.md b/blog/_posts/2016-07-25-nce.md index 89862a5..58980e5 100644 --- a/blog/_posts/2016-07-25-nce.md +++ b/blog/_posts/2016-07-25-nce.md @@ -4,7 +4,7 @@ title: Language modeling a billion words comments: True author: nicholas-leonard excerpt: Noise contrastive estimation is used to train a multi-GPU recurrent neural network language model on the Google billion words dataset. -picture: https://raw.githubusercontent.com/torch/torch.github.io/master/blog/_posts/images/rnnlm.png +picture: https://raw.githubusercontent.com/torch/torch.github.io/master/blog/_posts/images/rnnlm-small.png --- @@ -18,10 +18,12 @@ picture: https://raw.githubusercontent.com/torch/torch.github.io/master/blog/_po * [Future work](#nce.future) * [References](#nce.ref) +In our last post, we presented a [recurrent model for visual attention](http://torch.ch/blog/2015/09/21/rmva.html) +which combined reinforcement learning with recurrent neural networks. In this Torch blog post, we use noise contrastive estimation (NCE) [[2]](#nce.ref) to train a multi-GPU recurrent neural network language model (RNNLM) on the Google billion words (GBW) dataset [[7]](#nce.ref). -The work presented here is the result of many months of on-and-off work. +The work presented here is the result of many months of on-and-off work at [Element-Research](https://www.discoverelement.com/research). The enormity of the dataset caused us to contribute some novel open-source Torch modules, criteria and even a multi-GPU tensor. We also provide scripts so that you can train and evaluate your own language models. diff --git a/blog/_posts/images/rnnlm-small.png b/blog/_posts/images/rnnlm-small.png new file mode 100644 index 0000000..0cbff63 Binary files /dev/null and b/blog/_posts/images/rnnlm-small.png differ