Related to 392

[pythonometrist]

I am running into the same issue as #392.

Using APEX over cross validation leads to OOM.

I just wanted to check the work around solution.

Instantiate model, optimizer.

apex initialize

for k in cross_vals:
         train(model, optimizer)
         reset( model) 

Is that correct?

Problem is that I am using a transformer, so resetting the model indicates losing the transfer learning.

I wasnt clear on how to use the previous version of APEX either. Any suggestions would be appreciated.

[ptrblck]

Hi @pythonometrist,

could you explain a bit, what you mean by "losing the transfer learning"?
If you are finetuning the model, you could reset the model after each fold to this pretrained status again (instead of randomly initializing).

[mcarilli]

The recently merged native Amp API should fix this issue and straightforwardly accommodate cross-validation:
https://pytorch.org/docs/master/amp.html
https://pytorch.org/docs/master/notes/amp_examples.html

See #439 (comment) for details.