Hyperparameter evolution with optimizer key in hyps dict

[Zacchaeus00]

❔Question

When I evolve hyperparameters, tb_writer seems to be None.

Additional context

Apex recommended for faster mixed precision training: https://github.com/NVIDIA/apex
Namespace(batch_size=2, bucket='', cache_images=False, cfg='models/yolov5x.yaml', data='config/config.yaml', device='', epochs=10, evolve=True, hyp='', img_size=[1024], multi_scale=False, name='yolov5x_hyp', noautoanchor=False, nosave=False, notest=False, rect=False, resume=False, single_cls=False, weights='')
Using CUDA device0 _CudaDeviceProperties(name='Tesla P100-PCIE-16GB', total_memory=16280MB)

Hyperparameters {'optimizer': 'SGD', 'lr0': 0.01, 'momentum': 0.937, 'weight_decay': 0.0005, 'giou': 0.05, 'cls': 0.58, 'cls_pw': 1.0, 'obj': 1.0, 'obj_pw': 1.0, 'iou_t': 0.2, 'anchor_t': 4.0, 'fl_gamma': 0.0, 'hsv_h': 0.014, 'hsv_s': 0.68, 'hsv_v': 0.36, 'degrees': 0.0, 'translate': 0.0, 'scale': 0.5, 'shear': 0.0}
Traceback (most recent call last):
File "train.py", line 447, in
results = train(hyp.copy())
File "train.py", line 47, in train
log_dir = tb_writer.log_dir # run directory
AttributeError: 'NoneType' object has no attribute 'log_dir'

[glenn-jocher]

@Zacchaeus14 yes, we've recently updated this region of the code, but we have not tested hyperparameter evolution with it. I will try to look over this.

[glenn-jocher]

@Zacchaeus14 I've patched this specific problem, but hyperparameter evolution is not going to work the way things are currently due to the presence of the optimizer choice in the hyp dictionary. I'll leave this open with a TODO, but I don't have a timeline for you. In the meantime, do not evolve.

[Zacchaeus00]

Got it, thanks

[glenn-jocher]

@Zacchaeus14 --evolve functionality has been restored in 8056fe2. We fixed the bug affecting it and added a few new features, allowing you to specify per-parameter evolution gains and limits. Please let us know if you run into anymore issues.

[Zacchaeus00]

Really appreciate it! Your work definitely pushed the field forward!

[glenn-jocher]

@Zacchaeus14 thanks! We should be adding a hyperparameter evolution tutorial as well in the next few days. I'm running it now on the coco128 dataset and it seems to be working well.

I also just realized we should probably automatically saved the final result as evolved_hyps.yaml, and then you can train with this directly using the existing argparser updates:

python train.py --hyp evolved_hyps.yaml

yolov5/train.py

Line 402 in 8056fe2

parser.add_argument('--hyp', type=str, default='', help='hyp.yaml path (optional)')