Using with latest version of Detectron2

[Shadercloud]

Is it possible to use your model with the latest version of detectron2? I'm currently using detectron2 with good results (but not good for occluded objects).

I'm using CUDA 11, but your repo appears to be built on an older version of detectron2 with cuda 10.

Is it possible to just merge certain files into my existing detectron2 files and then use your model configs? (If so what files to merge?)

[lkeab]

You could download the detectron2 with support to CUDA 11, and then you could replace its mask head file with ours and some dataset reading parts. Note that there will be some minor difference you need to adapt due to the version/api differences of detectron2.

[Shadercloud]

Ok, i've downloaded a fresh latest copy of Detectron2. I then started merging BCNet into it, by doing the following steps:

1. Add new files:

new file:   configs/Base-FCOS-test-2x.yaml
new file:   configs/Base-FCOS-test-3x.yaml
new file:   configs/Base-FCOS.yaml
new file:   configs/fcos/fcos_imprv_R_101_FPN.yaml
new file:   configs/fcos/fcos_imprv_R_50_FPN.yaml
new file:   detectron2/layers/boundary.py
new file:   detectron2/layers/iou_loss.py
new file:   detectron2/layers/misc.py
new file:   detectron2/layers/scale.py

2. Update detectron2/config/defaults.py to include the FCOS Options section from BCNet

3. Update detectron2/layers/__init__.py to import all the newly added layer files

4. Update detectron2/modeling/__init__.py to remove imports to functions that no longer exist (will post file once it's working)

5. Overwrite entire detectron2/modeling/meta_arch/ directory with the BCNet directory

6. Overwrite entire detectron2/modeling/roi_heads/ directory with the BCNet directory

** EDITED ** Ignore previous error i've now gotten further with following steps:

7. Remove any reference to c_iter and max_iter in detectron2/modeling/meta_arch/fcos.py

8. Replace in detectron2/modeling/meta_arch/inference_fcos.py:
   pre_nms_top_n = candidate_inds.view(1, -1).sum(1)
   with
   pre_nms_top_n = candidate_inds.reshape(1, -1).sum(1)

9. Update detectron2/data/detection_utils.py and detectron2/data/datasets/coco.py to add code related to bg_object_segmentation

10. Added new file detectron2/data/datasets/process_dataset.py (and successfully used this to generate .json annotation files with the bg_object_segmentation

** EDITED ** Fixed final training error by replacing this code in detectron2/modeling/roi_heads/mask_head.py :

    if cls_agnostic_mask:
        pred_mask_logits_gt = pred_mask_logits[:, 0]
        pred_bo_mask_logits = pred_mask_bo_logits[:, 0]
        pred_boundary_logits_bo = pred_boundary_logits_bo[:, 0]
        pred_boundary_logits = pred_boundary_logits[:, 0]
    else:
        indices = torch.arange(total_num_masks)
        gt_classes = cat(gt_classes, dim=0)

with:

    if cls_agnostic_mask:
        pred_mask_logits_gt = pred_mask_logits[:, 0]
        pred_bo_mask_logits = pred_mask_bo_logits[:, 0]
        pred_boundary_logits_bo = pred_boundary_logits_bo[:, 0]
        pred_boundary_logits = pred_boundary_logits[:, 0]
    else:
        indices = torch.arange(total_num_masks)
        gt_classes = cat(gt_classes, dim=0)
        pred_mask_logits_gt = pred_mask_logits[indices, gt_classes]
        pred_bo_mask_logits = pred_mask_bo_logits[indices, gt_classes]
        pred_boundary_logits_bo = pred_boundary_logits_bo[indices, gt_classes]
        pred_boundary_logits = pred_boundary_logits[indices, gt_classes]

I will run it now and if it works I can put together a repo with this.

[Shadercloud]

I have created a Detectron2 (version 0.6) repo with BCNet merged into it: https://github.com/Shadercloud/detectron2

[lkeab]

Thanks for the help. 👍 :)

[kieranatkins]

Hi,
I am trying to use @Shadercloud 's merged D2 library, and after copying the BCNet detection_utils.py annotations_to_instances method to the new library I could begin training, however after 50 iterations the loss_cls reaches infinity and an exception is caused. What might cause the the loss to reach infinity/0 like this?

This is being run on a custom dataset, that has been run through the preprocessing script included. Thanks.

FloatingPointError: Loss became infinite or NaN at iteration=61! loss_dict = {'loss_cls': inf, 'loss_reg': 0.0, 'loss_centerness': 0.0, 'loss_mask': 0.7118306756019592, 'loss_mask_bo': 0.0, 'loss_boundary_bo': 0.0, 'loss_boundary': 0.5922521352767944}

[ttaggg]

@Shadercloud
I think several modifications to the code in your repo should be done to load pretrained weights for fcos_imprv_R_101_FPN.yaml.

1. in detectron2.detectron2.modeling.backbone.fpn.LastLevelP6P7: self.in_feature = "p5"
2. in detectron2.detectron2.modeling.backbone.fpn.LastLevelP6P7: self.p6 = nn.Conv2d(out_channels, out_channels, 3, 2, 1)
3. in detectron2.detectron2.config.defaults: _C.MODEL.ROI_MASK_HEAD.CLS_AGNOSTIC_MASK = True