Reasoning / Intuition behind Model Averaging

[teowenshen]

Is there any reason or intuition for why the models are averaged in checkpoint.py by using this formula, instead of averaging over a weighted sum?

(I just copied the equation from

icefall/icefall/checkpoint.py

Line 400 in 1d5c03f

(1) avg = (model_end * end - model_start * start) / interval.

, where b refers to batch_idx and m refers to model params.)

It seemed quite counterintuitive to me that the earlier model is subtracted from a later model, even if indeed the weights for $m_{\text{end}}$ and $m_{\text{start}}$ do sum to 1.

Is this actually an established model averaging method..? Or, was there any Github discussion regarding this model averaging method?

Thanks!

[csukuangfj]

Please have a look at

• Model average #344
• Model averaging #337

[yaozengwei]

In training, we maintain the model_avg, which is the average of all periodically sampled models (model_1, model_2,...,model_n) from the start:

During decoding, we want to use the averaged model at the interval [p+1, p+2, ..., q]:

[teowenshen]

I see. So in effect, Icefall is already averaging over every mini-batch (approximated by sampling every params.average_period) along the way during training, which is why the actual averaging itself for decoding is just a subtraction of the earlier models.

Thank you so much for pointing me towards the correct direction and for connecting the dots for me.