Option to output "test predictions" text file with each checkpoint in run_seq2seq.py

[kingpalethe]

Further to this discussion:

https://discuss.huggingface.co/t/how-to-output-test-generations-txt-with-run-seq2seq-py/3825

The prior incarnation of this script would output test generations at each checkpoint, which was very useful for understanding the progress of model training.

The current script...

https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py

Seems to only output this text file once, at the end of the last epoch.

If there was a way to enable the previous behavior, I am guessing that would be widely useful.

thanks

[LysandreJik]

May be of interest to @patil-suraj @stas00 @sgugger

[stas00]

Yes, as I replied in the forums, this functionality was dropped - not sure why it was done, as I wasn't part part of the planning discussion.

I think it was not intentional, the devs were probably unaware it was used and given that the example tests were dropped too it's not surprising it was missed. I propose the dropped examples tests are restored (which will require porting to the new script) which will expose some of the functionality that was removed with it.

Practically, let's identify what else might have been removed and create separate issues besides this one and may be ask the community to help restore/backport the previously working things to the new script(s)?

e.g. one such important thing is the tests that were moved to legacy, so this script is no longer being tested.

p.s. this should be of help restoring/porting the example tests #10036

[stas00]

@bhadreshpsavani, please let us know if you're inspired to take care of this in:
#10337 (comment)
Thank you.

[bhadreshpsavani]

Sure @stas00,
I can take care of this with a separate PR or if possible in the same PR,
Thanks

[stas00]

Correction, as I was refactoring run_seq2seq.py I can see now that the code wasn't removed - it's exactly the same. Someone decided to rename the resulting file instead. So the feature hasn't been removed, just renamed.

I'm not attached to either,

1. the original was saving it as "test_generations.txt"
2. the new one as "test_preds_seq2seq.txt"

I think the original name is the most intuitive one.

@sgugger, do you have an opinion here?

[stas00]

@bhadreshpsavani, so please hold a moment while we are re-modelling run_seq2seq.py and then I will update you when the model example is ready to be synced. Thank you!

[stas00]

PR to restore the original functionality: #10428

[stas00]

OK, the original name has been restored as it used to be, @kingpalethe

As I mentioned in #10428 if you'd like to request a new feature to do this on each check point please don't hesitate to make such request.

[kingpalethe]

@stas00 thanks -- apologies, you are correct. I had hallucinated this behavior. I made a new issue: #10439

[stas00]

All is good.

and now I see that my PR made that script inconsistent with other scripts, but perhaps all scripts should use the same filename for test_generations.txt. I can't quite see the point of it having a different name in each script.