How to get intermediate values of pretrained models?

[snie2012]

Is it possible to get the intermediate values of the hidden layers for the pretrained sequence tagging models when making prediction? For example, the output of the LSTM layer of a pretrained POS tagging model. If so, how?

[alanakbik]

Hello @snie2012 that's a good idea - unfortunately there is currently no in-built way to do this, but this is something we should really add. I'll add a 'feature' tag to this issue.

For now, one thing you could do is to modify the forward() method of the SequenceTagger so that it returns the intermediate tensors as well and get the embeddings from there.

[snie2012]

Thanks for your reply @alanakbik . It'll be great if there is built in method to do this. Currently, I manage to use register_forward_hook to get the intermediate values. It works something like this:

sentence = Sentence('I love Berlin .')
pos_tagger = SequenceTagger.load('pos')

output = torch.zeros(4, 1, 53)  # The intermediate value will be stored in this variable 
def hook(m, i, o): output.copy_(o.data)  # The hook that copies intermediate value during forward pass
pos_tagger.linear.register_forward_hook(hook)  # Register the hook

pos_tagger.predict(sentence)  # Run the model on a sentence
print(output)  # The intermediate value is now in output

[alanakbik]

Ah this is great and might be a good blueprint for a convenience method that uses hooks like this to get the embeddings. So thanks for sharing this solution!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.