RoIHeads: don't remove empty boxes in postprocess

pytorch#1019 introduced a change that removes empty boxes in
postprocessing on the basis that, previously, empty boxes would actually
reach this postprocessing step, and that the model could therefore
output empty boxes (even though NMS would've most likely filtered them
out). It's essentially a safety check that'd be seldom needed.

However, that filtering causes dynamicity on the TPU (because the number
of empty boxes, if any, would be unknown). And in any case, we recently
introduced a change in PR pytorch#4 that purposefully pads the boxes tensor
with empty boxes, to avoid dynamicity. Therefore there's no point trying
to remove boxes that we use as padding, we'll just filter those out of
the output on the CPU.

torchvision/models/detection/roi_heads.py

@@ -603,10 +603,6 @@ def postprocess_detections(self, class_logits, box_regression, proposals, image_
             scores = torch.where(score_mask, scores, torch.tensor(0.0, device=device))
             labels = torch.where(score_mask, labels, torch.tensor(-1, device=device))
 
-            # remove empty boxes
-            keep = box_ops.remove_small_boxes(boxes, min_size=1e-2)
-            boxes, scores, labels = boxes[keep], scores[keep], labels[keep]
-
             # non-maximum suppression, independently done per class
             keep = box_ops.batched_nms(boxes, scores, labels, self.nms_thresh, self.detections_per_img)