[FEATURE] LR_DECAY adjustment from 0.007 to 0.0045 seems to help model convergence

[2-dor]

Hey Steve,

Minor thing but it seems that using the LR_DECAY value of 0.0045 in the trainer benefits model convergence over the 0.007 value in the trainer by default.

I have been using this value for a few months now & its ESR slope is slightly better. If you feel this is a noteworthy improvement, maybe having it included in the next release?

[sdatkinson]

Neat.

I'm potentially interested, but would want more data points.

It would be nice to assemble a set of test cases. I assume this is a high-gain model? Orange is better, but it's still over 0.01. What happens when it's a clean model, or when there's a cab? etc

[2-dor]

Neat.

I'm potentially interested, but would want more data points.

It would be nice to assemble a set of test cases. I assume this is a high-gain model? Orange is better, but it's still over 0.01. What happens when it's a clean model, or when there's a cab? etc

Yes. It's a very hairy high gain model.
I will run a new set of tests today to ilustrate how the 0.0045 decay compares against the 0.007 across clean, gain & miced-up reamps.

[2-dor]

This is how the training looks like for a "CLEAN" reamp:

[2-dor]

This is how the training looks like for a "HIGH GAIN" direct (no cab) reamp:

[2-dor]

This is how the training looks like for a "HIGH GAIN" miced-up (SM57 on cab) reamp:

[2-dor]

This archive contains the reamps (all were done using the v3_0_0.wav), the resulting NAM models and the lightning_logs folder with all its contents: https://mega.nz/file/GhwwVbaJ#95_cYIMwLW4AeQN9lgXlG1VZ9YkDvpUqhPO1Ntw-XKs

[2-dor]

LR 0.004 and LR_DECAY 0.004 seems to be slightly less jagged and converges comparatively (ESR & time) to LR_DECAY 0.007

Anyway, this is a deep rabbit hole..

[yovelop]

Absolutely agree that LR_decay better to be smaller than 0.007
I done many-many tests before and now use 0.002
It depends on how much epochs can you make. If less than 300 - then 0.007 is normal, if 1000-2000 - 0.003 will be great, if 2000-5000 - use 0.002, if more - 0.001

But initial lr sometimes need to set lower value or loss will jump many times (jump from local minimum to worse place)