ValueError: No proposal boxes available for one of the images during training

[botcs]

❓ Questions and Help

In #169 the case is discussed where no annotations are available.
During training on small objects it is quite frequent that the RPN does not generate any proposal, what can I do now?

Reducing NMS_THRESH and FG_IOU_THRESHOLD does not seem to help.

[fmassa]

Hi,

I think this error should only happen if there are no boxes output by the RPN, and not that the boxes do not overlap, which is very weird.

Did you change anything else in the codebase?

[botcs]

No, the coco-training is running perfectly, it is rather the database.

[fmassa]

Oh, I see, this error is not happening when matching the instances for the RPN, but when preparing the instances for the heads.
This is weird, because we normally append the ground-truth box together with the proposals during training, so this shouldn't be happening.

[botcs]

The thing is that it runs for a while with an average number of proposals of 25 (not showing any kind of decrease) and suddenly when 0 proposals are made, then crash occurs, and I am not so sure at which point should I catch this error. Could you give some hints?

[fmassa]

are you sure that all the images have valid annotations? This sounds like there is an image that has problematic annotations

[botcs]

I have printed all the instance labels for every image loaded, and no, every empty entries are removed from the dataset

[fmassa]

Same thing for boxes which have less than 1 size?

[botcs]

printed the boxes' size as well, I remove any instance that is less than 100 pixels in area (sum reduced binary masks) not a single problem like this

[fmassa]

One thing I don't understand: you mentioned that there are 25 proposals on average: this seems very low to me, the default parameters should make something like 512. Why is that the case?

[botcs]

try out printing proposals on the COCO dataset as well. It is higher than 25 but definitely not close to 512

[fmassa]

So, I think there is a confusion between proposals and ground-truth boxes.

The proposals are the ones returned by the RPN, and are in the order of 1000s. We select a subset of them to feed to the heads.
Then there are the GT boxes, which are in much smaller number, in average 20 or so per image.

The error you are mentioning says that there are no proposals: this is very weird and should not happen in general.

[botcs]

If you'd like, I can get the number of proposals during a regular COCO training as well and report it here.

Indeed, the error is quite odd, any tips on workaround?
How this case should be handled?

[fmassa]

I'd first try to understand if your RPN is outputting the right things. It should output around 2000 proposals (that's the value we specify in the config). If that's lower than that then there is a problem there already

[botcs]

I have found the source of this problem.
The proposals were cut away by keep_only_positive_boxes since my contiguous_id started from 0 instead of 1, while 0 should be kept for background.

[hm1119]

I have found the source of this problem. The proposals were cut away by keep_only_positive_boxes since my contiguous_id started from 0 instead of 1, while 0 should be kept for background.

hello I also have this issue; and how do you solved it. thanks!