The shape of the mask [268569] at index 0 does not match the shape of the indexed tensor [0] at index 0

[jpainam]

Problem with Instance Segmentation

Hi, I trained a maskrcnn model without your augmented data/annotation and it worked well. Using your data/annotation folder and file. I got this error

Traceback (most recent call last):
  File "tools/train_net.py", line 171, in <module>
    main()
  File "tools/train_net.py", line 164, in main
    model = train(cfg, args.local_rank, args.distributed)
  File "tools/train_net.py", line 73, in train
    arguments,
  File "/home/eldad/maskscoring_rcnn/maskrcnn_benchmark/engine/trainer.py", line 66, in do_train
    loss_dict = model(images, targets)
  File "/home/eldad/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eldad/maskscoring_rcnn/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 49, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/home/eldad/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eldad/maskscoring_rcnn/maskrcnn_benchmark/modeling/rpn/rpn.py", line 100, in forward
    return self._forward_train(anchors, objectness, rpn_box_regression, targets)
  File "/home/eldad/maskscoring_rcnn/maskrcnn_benchmark/modeling/rpn/rpn.py", line 119, in _forward_train
    anchors, objectness, rpn_box_regression, targets
  File "/home/eldad/maskscoring_rcnn/maskrcnn_benchmark/modeling/rpn/loss.py", line 91, in __call__
    labels, regression_targets = self.prepare_targets(anchors, targets)
  File "/home/eldad/maskscoring_rcnn/maskrcnn_benchmark/modeling/rpn/loss.py", line 62, in prepare_targets
    labels_per_image[~anchors_per_image.get_field("visibility")] = -1
RuntimeError: The shape of the mask [268569] at index 0 does not match the shape of the indexed tensor [0] at index 0

So i guess, the error is coming from the fact that, for a rotation, the shape of the images does not match anymore.
I used this tutorial https://colab.research.google.com/github/joheras/CLoDSA/blob/master/notebooks/CLODSA_Instance_Segmentation.ipynb

[joheras]

Hi,

So, the problem is happenning with the maskrcnn library that you are using, right?

Could you indicate me which implementation of MaskRcnn are you using so I can try to reproduce the error? Could you also send me the augmentations that you applied to your dataset?

Best

[jpainam]

Hi, this is the augmentations i applied to my dataset

import matplotlib.pyplot as plt
from clodsa.augmentors.augmentorFactory import createAugmentor
from clodsa.transformers.transformerFactory import transformerGenerator
from clodsa.techniques.techniqueFactory import createTechnique
import cv2

PROBLEM = "instance_segmentation"
ANNOTATION_MODE = "coco"
INPUT_PATH = "/home/eldad/data/VOCdevkit/VOC2007/JPEGImages/"
GENERATION_MODE = "linear"
OUTPUT_MODE = "coco"
OUTPUT_PATH = "/home/eldad/data/maskrcnn_train_augmented/"

augmentor = createAugmentor(PROBLEM, ANNOTATION_MODE, OUTPUT_MODE,
                            GENERATION_MODE, INPUT_PATH,
                            {"outputPath": OUTPUT_PATH}
                            )
transformer = transformerGenerator(PROBLEM)

for angle in [90, 180]:
    rotate = createTechnique("rotate", {"angle": angle})
    augmentor.addTransformer(transformer(rotate))

flip = createTechnique("flip", {"flip": 1})
augmentor.addTransformer(transformer(flip))

none = createTechnique("none", {})
augmentor.addTransformer(transformer(none))

augmentor.applyAugmentation()

I used maskscoring which is built on top of maskscrcnn. The core of maskscoring is till maskrcnn as you can see in the Traceback error. Here the repository
https://github.com/zjhuang22/maskscoring_rcnn
Thanks

[joheras]

I have never used that library, could you provide me a minimal example to try it? Best, Jónathan El lun., 14 sept. 2020 a las 9:11, Jean-Paul AINAM (< [email protected]>) escribió:

…

Hi, this is the augmentations i applied to my dataset import matplotlib.pyplot as pltfrom clodsa.augmentors.augmentorFactory import createAugmentorfrom clodsa.transformers.transformerFactory import transformerGeneratorfrom clodsa.techniques.techniqueFactory import createTechniqueimport cv2 PROBLEM = "instance_segmentation"ANNOTATION_MODE = "coco"INPUT_PATH = "/home/eldad/data/VOCdevkit/VOC2007/JPEGImages/"GENERATION_MODE = "linear"OUTPUT_MODE = "coco"OUTPUT_PATH = "/home/eldad/data/maskrcnn_train_augmented/" augmentor = createAugmentor(PROBLEM, ANNOTATION_MODE, OUTPUT_MODE, GENERATION_MODE, INPUT_PATH, {"outputPath": OUTPUT_PATH} )transformer = transformerGenerator(PROBLEM) for angle in [90, 180]: rotate = createTechnique("rotate", {"angle": angle}) augmentor.addTransformer(transformer(rotate)) flip = createTechnique("flip", {"flip": 1})augmentor.addTransformer(transformer(flip)) none = createTechnique("none", {})augmentor.addTransformer(transformer(none)) augmentor.applyAugmentation() I used maskscoring which is based on maskscrcnn. The core of maskscoring is till maskrcnn. Here the repository https://github.com/zjhuang22/maskscoring_rcnn Thanks — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJRAG4F7JCXSCNETLLFBWLSFW6Y3ANCNFSM4RKREHCQ> .

[jpainam]

Hi, it's a little bit difficult to provide you with a minimal example. We have generated more than 4000 images using your tool and we don't really know which image is causing this problem.
Since rotation, and flipping change the width and height of the image, i would like to know if your tool also rotates or flips the annotation values?

Thank you.
Meanwhile, we will try our best to provide you with a minimal example.

[joheras]

Hi, Yes, the tool rotates and flips the annotation values to obtain the correct annotation. I will try to see how the tool works so I can try to reproduce the problem. Best, Jónathan El mar., 15 sept. 2020 a las 11:13, Jean-Paul AINAM (< [email protected]>) escribió:

…

Hi, it's a little bit difficult to provide you with a minimal example. We have generated more than 4000 images using your tool and we don't really know which image is causing this problem. Since rotation, and flipping change the width and height of the image, i would like to know if your tool also rotates or flips the annotation values? Thank you. Meanwhile, we will try our best to provide you with a minimal example. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJRAG3FO5UIPM6KVRUVWEDSF4V33ANCNFSM4RKREHCQ> .

[Eldad27]

@joheras I've been following this.

1. can you please provide a brief explanation of how this augmentation process (for instance segmentation) handle images in a dataset without annotations (no ground truth)? What's your advice on augmenting images without ground truth (negative samples)?

2. Could there be a possibility that negative coordinates are generated as part of the augmentation through some faulty files during the augmentation process?

[jpainam]

@joheras @Eldad27 I found out that, though I have images without annotations. (i.e, the segmentation field is [[]], these images do not appear in the new json file or the output folder. I guess CLoDSA removes them before creating the new json file.

Empty segmentation fields are also represented as [[]]. I didn't find a single one in my annotation file

[joheras]

Ok, I see. I didn't consider empty annotations. I will release a new version, hopefully tomorrow, solving that issue. I'll keep you posted. Best, Jónathan El dom., 20 sept. 2020 8:54, Jean-Paul AINAM <[email protected]> escribió:

…

@joheras <https://github.com/joheras> @Eldad27 <https://github.com/Eldad27> I found out that, though I have images without annotations. (i.e, the segmentation field is [[]], these images do not appear in the new json file or the output folder. I guess CLoDSA removes them before creating the new json file. Empty segmentation fields are also represented as [[]]. I didn't find a single one in my annotation file — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJRAG76SGE4W5K23ZWTIELSGWRJLANCNFSM4RKREHCQ> .

[joheras]

Hi, I am trying to understand the problem. I have checked that when the json file contains an image, but that image does not have any annotation, that is, when in the list associated with the "annotations" key of the json file it does not appear the reference to the image, clodsa augments those images correctly. However, you indicated that the segmentation field contains [[]] but this does not seem to follow the COCO convention. Am I missing something? Could you send me an example of your annotation file to know what is happening? Best, Jónathan El dom., 20 sept. 2020 a las 9:51, Jónathan Heras (<[email protected]>) escribió:

…

Ok, I see. I didn't consider empty annotations. I will release a new version, hopefully tomorrow, solving that issue. I'll keep you posted. Best, Jónathan El dom., 20 sept. 2020 8:54, Jean-Paul AINAM ***@***.***> escribió: > @joheras <https://github.com/joheras> @Eldad27 > <https://github.com/Eldad27> I found out that, though I have images > without annotations. (i.e, the segmentation field is [[]], these images > do not appear in the new json file or the output folder. I guess CLoDSA > removes them before creating the new json file. > > Empty segmentation fields are also represented as [[]]. I didn't find a > single one in my annotation file > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#17 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABJRAG76SGE4W5K23ZWTIELSGWRJLANCNFSM4RKREHCQ> > . >

[jpainam]

Hi, I found the problem. It is related with maskrcnn. So, i'm going to close this issuee.
When bbox is empty, it raises this error, i fixed it by removing all images without bbox as shown in this issue
facebookresearch/maskrcnn-benchmark#31

This is what i did to make the training works

#data/datasets/coco.py
        ids_to_remove = []
        for img_id in self.ids:
            ann_ids = self.coco.getAnnIds(imgIds=img_id)
            anno = self.coco.loadAnns(ann_ids)
            if all(
                any(o <= 1 for o in obj['bbox'][2:])
                for obj in anno
                if obj['iscrowd'] == 0
            ):
                ids_to_remove.append(img_id)
        self.ids = [img_id for img_id in self.ids if img_id not in ids_to_remove]

Maybe, it is also related with the way CLoDSA deals with empty bbox . You can check that too