How to test each saved weight?

[xiaohai12]

❓ Questions and Help

After I saved several weights from training, I would like to evaluate each saved weights just in case overfitting.
When I tried to run python -m torch.distributed.launch --nproc_per_node=$NGPUS /path_to_maskrcnn_benchmark/tools/test_net.py --config-file "configs/e2e_mask_rcnn_R_50_FPN_1x.yaml" TEST.IMS_PER_BATCH 16
it only evaluated the last saved weights.
So, is it possible to evaluate each saved weights ?
By the way, it costed around 20 mins to evaluate 8000 images on GPU V100,
is it normal or not?

[leijiezhang]

Yes the code only evaluate the final check out point, the codes is in /engine/inference.py
if you want to check out other saved weights, you have to re-write the codes, it relates to many python files, be careful. I have tried however failed

[xiaohai12]

Yes the code only evaluate the final check out point, the codes is in /engine/inference.py
if you want to check out other saved weights, you have to re-write the codes, it relates to many python files, be careful. I have tried however failed

Actually, I checked what is the context inside the last checkpoint file and I found that it was just the path of the saved model weights, which means we only need to modified the last checkpoint file context by using the path of the model weights you want to test.

[obendidi]

I haven't tested it, but you can probably just modify the MODEL.WEIGHT in the config file you used with the full path to the model you want to test

[wangg12]

In maskrcnn_benchmark/utils/checkpoint.py, it will always load the last checkpoint in load().
I think you need to change the code to support evaluating on other checkpoints.

[wangg12]

Now you can test with any checkpoint you saved.

[xiaohai12]

Now you can test with any checkpoint you saved.

Thanks! It is very useful!