Regression in Training Performance after Migration to MMLab2.0

[npurson]

I am currently in the process of migrating a mmdet3d plugin project from its previous dependencies (mmcv-full==1.5.3, mmdet==2.25.1, mmdet3d==1.0.0rc4) to the latest versions (mmengine==0.10.2, mmcv==2.1, mmdet==3.3, mmdet3d==1.4).

Following the migration, I have diligently adjusted the altered APIs and configurations. However, I have encountered a concerning regression in training performance. Specifically, the mIoU results has decreased from 30 to 28.

Notably, when employing the pre-trained weights from the previous version for evaluation on the migrated version, consistent results are obtained, suggesting correctness in the implementation of both model and data loading procedures.

Given the complexity of the project, I refrain from sharing the entire codebase. Instead, I seek insights into potential general causes of such a performance drop, under the assumption that my implementation is correct. Leveraging your expertise in the development of MMLab projects, I hope to gain valuable perspectives on this matter.

Thank you in advance for your assistance and guidance.

[CFZ1]

I am currently in the process of migrating a mmdet3d plugin project from its previous dependencies (mmcv-full==1.5.3, mmdet==2.25.1, mmdet3d==1.0.0rc4) to the latest versions (mmengine==0.10.2, mmcv==2.1, mmdet==3.3, mmdet3d==1.4).

Following the migration, I have diligently adjusted the altered APIs and configurations. However, I have encountered a concerning regression in training performance. Specifically, the mIoU results has decreased from 30 to 28.

Notably, when employing the pre-trained weights from the previous version for evaluation on the migrated version, consistent results are obtained, suggesting correctness in the implementation of both model and data loading procedures.

Given the complexity of the project, I refrain from sharing the entire codebase. Instead, I seek insights into potential general causes of such a performance drop, under the assumption that my implementation is correct. Leveraging your expertise in the development of MMLab projects, I hope to gain valuable perspectives on this matter.

Thank you in advance for your assistance and guidance.

Why does this phenomenon occur? Have you found the reason? Thanks a lot.

[npurson]

Sadly no. Did you meet the same issue?

[CFZ1]

Yes, I used Tianchi aluminum materials for object detection. In 2021, I used mmdet 2.x, and the mAP@50 could reach 80%. However, after migrating to mmdet 3.x, with the same data split and similar parameter configurations, I can only achieve 69%-70% now. Switching back to mmdet 2.x seems to make it easier to achieve higher accuracy. It's strange, maybe I've made a mistake somewhere.