Attention visualization

[cifkao]

Soft alignment visualization (#108) for the bahdanau branch.

[jlibovicky]

Good job, thanks a lot! Please pay attention to the tests. One of the unit tests is failing because of the TensorBoard summaries (it issue can be also in the test). The integration test is failing because it cannot download the data it should work with. It should get fixed when you rebase with master.

[jlibovicky]

This line is too long. We check the style of the files using pylint. If you run tests/lint_run.sh, it will check all python files get you a detailed report.

[jlibovicky]

I wonder if this is general enough. If you look e.g, at the Recurrent Neural Machine Translation paper, they use GRU net do the attention instead of the softmax weighting and come with a clever way visualizing this attention. i think that reserving the attention_in_time field for the visualization purpose should work even for the recurrent attention, but please check if it is so.

[jindrahelcl]

just FYI, the line length used by me is 80. I will add it to the documentation for developers

[jlibovicky]

Considering that the inputs of NM can be also images (we did image captioning with it previously), name images may be confusing. What about plots, visualizations, ... I don't know, maybe even images are OK.

[cifkao]

What about out_images?

[jindrahelcl]

I'd stick with val_plots.

[jlibovicky]

Why 5?

[jlibovicky]

@cifkao: Can you rebase the branch with master and find out whether it will fix the failing unit tets?

[jindrahelcl]

@jlibovicky @cifkao I think the best solution is maybe to cherry-pick the commit that adds the testing data to the repository: ab0a0ba

[cifkao]

@jindrahelcl I agree, rebasing onto master would be difficult. But maybe it's better to cherry-pick ab0a0ba into bahdanau directly?

Apart from that, I found that the visualization is incorrect because attentions_in_time accumulates attention vectors from both training and runtime decoding (is that intended?). Given that I'm visualizing the validation data, I think I want to use only the second half of the list. But what's the right way to separate it from the rest?

[jindrahelcl]

The data should be already in the bahdanau branch (commit 5d88de3)..

The graph contains the attention mechanism ops twice - one for training and the other one for runtime. During validation, the training ops are not executed, so there should not be a problem.

[jindrahelcl]

@cifkao if you don't want to cherry-pick the commit (which is imho the best option), merge the bahdanau branch to this one. (Or rebase this on bahdanau.. But rebasing something that is already pushed to github should be avoided)

[cifkao]

The problem is that the same Attention object is used for constructing both the training and runtime parts of the graph, and therefore attentions_in_time contains tensors from both parts. Because I'm using attentions_in_time for visualization, I was getting images like this:

This was, of course, easily fixed by taking only the last len(decoder_inputs) elements of the list (i.e. the ones that were just appended).

[jindrahelcl]

Huh, this looks very much like some sort of bug. I'll look into it.

2016-11-08 8:30 GMT+00:00 Ondřej Cífka [email protected]:

The problem is that the same Attention object is used for constructing
both the training and runtime parts of the graph, and therefore
attentions_in_time contains tensors from both parts. Because I'm using
attentions_in_time for visualization, I was getting images like this:

[image: tensorboard]
https://cloud.githubusercontent.com/assets/8046580/20091403/dac7641a-a592-11e6-9777-ce513c1e0bb1.png

This was, of course, easily fixed by taking only the last
len(decoder_inputs) elements of the list.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#121 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABwcs1uiXk8b1HogwqbBU8OOoyH4Lq4Gks5q8DMwgaJpZM4KkZYF
.

[jlibovicky]

It's definitely a bug, but luckily, it does not influence performance of most of the models, but it is probably a reason why the coverage model worked so poorly.