Import coco annotations with keypoints and boxes

[twurdwz]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I have images with two types of annotations (keypoints and bounding boxes) for each instance in each frame. I have a similar scenario as #5904 or #1421 .

When exporting everything works as expected and I get the outside bounding box in the annotation object. The issue comes when I import annotations to get the same/similar results.

Ideally, I would like to directly import the box and the keypoints as a group. If that is not possible, can I import them separately and then manually group them?

I have 2 categories: person, and person_bbox.

I have tried...

1. Importing as coco keypoints 1.0 the box in the annotation object: { ..., keypoints: [...], bbox: [...], ...]. --> This ignores the bbox data.
2. Importing two different annotations (coco and coco keypoint) --> This overwrites annotations.
3. Importing two annotation objects per instance (with different ids) one with category person and another one with person_bbox (this one with no keypoints. --> This throws and error when importing because keypoints are None type.

*Note: I am using cvat.ai online tool.

Expected Behaviour

The expected behavior is to import a keypoint dataset with different bounding box than the one defined by the keypoints. If not, be able to import Groups.

Current Behaviour

Possible Solution

Context

Your Environment

• I am using cvat.ai online tool.
• All my annotations are in coco format

[zhiltsov-max]

Hi, I agree that it's inconvenient. You can use this workaround until it's implemented:

import os.path as osp
import sys
from glob import glob
from typing import Dict, Optional
import datumaro as dm
from datumaro.plugins.cvat_format.converter import _SubsetWriter, XmlAnnotationWriter
from unittest import mock

class RemapBboxLabels(dm.ItemTransform):
    def __init__(self, extractor: dm.IExtractor, new_label_id: int):
        super().__init__(extractor)
        self._new_label_id = new_label_id

    def transform_item(self, item: dm.DatasetItem) -> Optional[dm.DatasetItem]:
        updated_annotations = []
        for ann in item.annotations:
            if isinstance(ann, dm.Bbox):
                ann = ann.wrap(label=self._new_label_id)
            updated_annotations.append(ann)

        return self.wrap_item(item, annotations=updated_annotations)

class PatchedCvatSubsetWriter(_SubsetWriter):
    # CVAT will require 'outside' property on the skeleton points,
    # but it is missing in the datumaro export in CVAT format
    # Here we fix this by monkey-patching the export method.

    def _write_shape(self, shape, item):
        xml_writer = self._writer

        def patched_open_points(points: Dict):
            if isinstance(shape, dm.Points):
                points['outside'] = str(int(shape.visibility[0] == dm.Points.Visibility.absent))
                points['occluded'] = str(int(shape.visibility[0] == dm.Points.Visibility.hidden))

            XmlAnnotationWriter.open_points(xml_writer, points)

        with mock.patch.object(self._writer, 'open_points', patched_open_points):
            return super()._write_shape(shape, item)

def prepare_import_dataset(kp_dataset_dir: str, dst_dir: str):
    kp_dataset_annotation_filename = next(
        fn for fn in glob(osp.join(kp_dataset_dir, 'annotations', '*.json'))
        if 'person_keypoints' in osp.basename(fn)
    )
    bbox_dataset = dm.Dataset.import_from(kp_dataset_annotation_filename, 'coco_instances')
    kp_dataset = dm.Dataset.import_from(kp_dataset_annotation_filename, 'coco_person_keypoints')

    # Boxes need to have a separate label in CVAT,
    # but they will be parsed with the same label as skeletons,
    # since they are read from the same annotation. So, we just remap the labels.
    resulting_labels = kp_dataset.categories()[dm.AnnotationType.label]
    bbox_dataset.transform('project_labels', dst_labels=resulting_labels)
    bbox_label_id = resulting_labels.find('person_bbox')[0] # <<<< use your bbox label name here
    assert bbox_label_id is not None

    output_dataset = dm.Dataset.from_extractors(bbox_dataset, kp_dataset)
    output_dataset.transform(RemapBboxLabels, new_label_id=bbox_label_id)

    with mock.patch('datumaro.plugins.cvat_format.converter._SubsetWriter', PatchedCvatSubsetWriter):
        output_dataset.export(dst_dir, 'cvat', save_images=True)


if __name__ == '__main__':
    prepare_import_dataset(sys.argv[1], sys.argv[2])

You'll need to install Datumaro which is used in CVAT now:

python3 -m virtualenv venv
. ./venv/bin/activate
pip install 'datumaro[default]@git+https://github.com/cvat-ai/datumaro.git@ff83c00c2c1bc4b8fdfcc55067fcab0a9b5b6b11'

Convert your COCO dataset using the script above:

python ./convert.py <path/to/coco/dataset/dir> <output/dir>

And import the produced annotation file in CVAT task using the CVAT format.

[twurdwz]

Thanks a lot! converting the annotations to cvat 1.1 format and then importing those annotations to cvat worked.

[twurdwz]

Closing this issue