Question about the abc parametrization across the two papers

[thomasdulacatos]

Hi. First, thanks for your work.

Reading your two papers (https://arxiv.org/pdf/2011.14522.pdf and https://arxiv.org/pdf/2203.03466.pdf), I've seen some differences that I don't understand. On the first paper, you define the abc-Parametrization, and you write (page 2):

On page 9, you specified the Mup parametrization c is equal to 0, so the learning rate is not proportional to n.

But, on your second paper, page 5, when specifying the Mup parametrization, the learning rate are not the same for the different layers, and some are proportional to n:

I don't understand the reason of those differences ... Thanks for your time

[thegregyang]

Hi Thomas,

Thanks for your interest.

These two descriptions are equivalent. The former paper (TP4) notes the following symmetry

which remains true if you replace c with c_l (learning rate scaling for individual layer l) above. In other words, in the (a_l, b_l, c_l) for each layer, there is a 1-dimensional redundancy. Up to this redundancy, the two descriptions are the same.

The reason we differ in our presentation is that the former paper (TP4) is aimed at theoretists who tend to think about gradient descent with no "ad hoc" modification to the gradient (like per-layer learning rate), so we use a single learning rate for the entire network, at the expense of needing to scale multipliers (the a_ls) for different layers. However, the latter paper (TP5) is aimed at practitioners, who are very comfortable with per-layer learning rate, for example, through parameter groups in PyTorch optimizers. In contrast, they are less comfortable with changing model definition by inserting and changing multipliers. So we opt to absorb all multipliers into per-layer learning rates.

More details are given in the latter paper (TP5):