GPU memory increasing, then eventually OOM

[DangChuong-DC]

I tried to run Rotated-Faster-RCNN on dota-1.5 dataset.
I found that GPU memory is increasing throughout the training and eventually OOM.
Below image is my run with 1 GPU.

Image

[yangxue0827]

Please run python mmrotate/utils/collect_env.py to collect necessary environment information and paste it here.

[yangxue0827]

Also, please provide your configs in the path of work_dirs.

[williamcorsel]

Experienced the same issue with Oriented R-CNN. I fixed it by setting the gpu_assign_thr flag following the suggestion here

[DangChuong-DC]

Thank you @yangxue0827 for prompt response and your dedicated works.
@williamcorsel I confirmed that setting gpu_assign_thr=100 resolved my issues.

Great works, thank you very much.