[Bug]

[pphgood]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (master) or latest version (1.x).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

master branch https://github.com/open-mmlab/mmrotate

Environment

Install according to the official documents

Reproduces the problem - code sample

def min_area_polygons(pointsets: torch.Tensor) -> torch.Tensor:
"""Find the smallest polygons that surrounds all points in the point sets.

Args:
    pointsets (Tensor): point sets with shape  (N, 18).

Returns:
    torch.Tensor: Return the smallest polygons with shape (N, 8).
"""
polygons = pointsets.new_zeros((pointsets.size(0), 8))
ext_module.min_area_polygons(pointsets, polygons)
return polygons

Reproduces the problem - command or script

python tools/test.py or.py work_dirs/or1/epoch_12.pth --format-only --eval-options submission_dir=work_dirs/Task1_results

Reproduces the problem - error message

[>>>>>>>>>>>>>>>>>>>>>>>>> ] 5579/10833, 9.4 task/s, elapsed: 592s, ETA: 557sException ignored in: <function _MultiProcessingDataLoaderIter.del at 0x00000187DA88D4C0>
Traceback (most recent call last):
File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1481, in del
self._shutdown_workers()
File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 1469, in _shutdown_workers
w.terminate()
File "C:\Users\pph.conda\envs\pytorch\lib\multiprocessing\process.py", line 133, in terminate
self._popen.terminate()
File "C:\Users\pph.conda\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate
_winapi.TerminateProcess(int(self._handle), TERMINATE)
PermissionError: [WinError 5] 拒绝访问。
Traceback (most recent call last):
File "tools/test.py", line 257, in
main()
File "tools/test.py", line 222, in main
outputs = single_gpu_test(model, data_loader, args.show, args.show_dir,
File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmdet\apis\test.py", line 29, in single_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "d:\mmrotate\mmrotate\mmrotate\models\detectors\single_stage.py", line 101, in simple_test
bbox_list = self.bbox_head.get_bboxes(
File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmcv\runner\fp16_utils.py", line 208, in new_func
return old_func(*args, **kwargs)
File "d:\mmrotate\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 1125, in get_bboxes
results = self._get_bboxes_single(cls_score_list, point_pred_list,
File "d:\mmrotate\mmrotate\mmrotate\models\dense_heads\oriented_reppoints_head.py", line 1216, in _get_bboxes_single
polys = min_area_polygons(pts)
File "C:\Users\pph.conda\envs\pytorch\lib\site-packages\mmcv\ops\min_area_polygons.py", line 19, in min_area_polygons
ext_module.min_area_polygons(pointsets, polygons)
RuntimeError: CUDA error: an illegal memory access was encountered
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Users\pph.conda\envs\pytorch\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate
_winapi.TerminateProcess(int(self._handle), TERMINATE)
PermissionError: [WinError 5] 拒绝访问。

Additional information

It's fine when I'm training, but when I'm using the test it happens。The same error happened when I was running on my server。
But when I tested the validation set, this error did not occur。

[zytx121]

Hi @pphgood, this may be because the min_area_polygons operator crashed because no target was detected on a picture in the test set.

• You can add try, except in _get_bboxes_single to skip it when it crashes.
• Try some potentially useful solutions mentioned in CUDA error: an illegal memory access was encountered #405.

BTW the maintenance of the old version has stopped. Welcome to the new version: https://github.com/open-mmlab/mmrotate/tree/1.x

[pphgood]

Hi @pphgood, this may be because the min_area_polygons operator crashed because no target was detected on a picture in the test set.

• You can add try, except in _get_bboxes_single to skip it when it crashes.
• Try some potentially useful solutions mentioned in CUDA error: an illegal memory access was encountered #405.

BTW the maintenance of the old version has stopped. Welcome to the new version: https://github.com/open-mmlab/mmrotate/tree/1.x
Thank you for your answer, but I have another problem during the experiment. I changed the loss function. Finally, the experimental effect increased somewhat in the training set and verification set, but it was worse in the test set. May I ask if this is the cause of overfitting?