master branch warmup learning rate is not bias lr falls from 0.1 to lr0

[challow0]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

No response

Bug

• when adding to optimizer groups, g[2] is not the last to add. But in warmup it is still j == 2

• This is the master branch
  

  yolov5/train.py

  Lines 153 to 171 in 0537e8d

  	g = [], [], [] # optimizer parameter groups
  	bn = tuple(v for k, v in nn.__dict__.items() if 'Norm' in k) # normalization layers, i.e. BatchNorm2d()
  	for v in model.modules():
  	if hasattr(v, 'bias') and isinstance(v.bias, nn.Parameter): # bias
  	g[2].append(v.bias)
  	if isinstance(v, bn): # weight (no decay)
  	g[1].append(v.weight)
  	elif hasattr(v, 'weight') and isinstance(v.weight, nn.Parameter): # weight (with decay)
  	g[0].append(v.weight)
  	if opt.optimizer == 'Adam':
  	optimizer = Adam(g[2], lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	elif opt.optimizer == 'AdamW':
  	optimizer = AdamW(g[2], lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	else:
  	optimizer = SGD(g[2], lr=hyp['lr0'], momentum=hyp['momentum'], nesterov=True)
  	optimizer.add_param_group({'params': g[0], 'weight_decay': hyp['weight_decay']}) # add g0 with weight_decay
  	optimizer.add_param_group({'params': g[1]}) # add g1 (BatchNorm2d weights)

• This is the v6.1 tag branch
  

  yolov5/train.py

  Lines 156 to 173 in 3752807

  	g0, g1, g2 = [], [], [] # optimizer parameter groups
  	for v in model.modules():
  	if hasattr(v, 'bias') and isinstance(v.bias, nn.Parameter): # bias
  	g2.append(v.bias)
  	if isinstance(v, nn.BatchNorm2d): # weight (no decay)
  	g0.append(v.weight)
  	elif hasattr(v, 'weight') and isinstance(v.weight, nn.Parameter): # weight (with decay)
  	g1.append(v.weight)
  	if opt.optimizer == 'Adam':
  	optimizer = Adam(g0, lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	elif opt.optimizer == 'AdamW':
  	optimizer = AdamW(g0, lr=hyp['lr0'], betas=(hyp['momentum'], 0.999)) # adjust beta1 to momentum
  	else:
  	optimizer = SGD(g0, lr=hyp['lr0'], momentum=hyp['momentum'], nesterov=True)
  	optimizer.add_param_group({'params': g1, 'weight_decay': hyp['weight_decay']}) # add g1 with weight_decay
  	optimizer.add_param_group({'params': g2}) # add g2 (biases)

• This is the master branch warmup
  

  yolov5/train.py

  Lines 336 to 338 in 0537e8d

  	for j, x in enumerate(optimizer.param_groups):
  	# bias lr falls from 0.1 to lr0, all other lrs rise from 0.0 to lr0
  	x['lr'] = np.interp(ni, xi, [hyp['warmup_bias_lr'] if j == 2 else 0.0, x['initial_lr'] * lf(epoch)])

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[glenn-jocher]

@challow0 thanks for the bug report! It looks like you are correct, 338 should have been updated and has not been.