[GSoC2024] Fix import of outside track shapes in Datumaro-based formats

[Yeek020407]

Motivation and context

Main Issue: #7571
Related Issue: #2339

I've reproduced the issue mentioned in #7571 when exporting and importing annotations using both the Datumaro and Coco 1.0 formats. Specifically, the "Switch outside" attribute isn't being applied as expected. After some investigation, I pinpointed the root cause to be the absence of the "outside" attribute in the exported annotations.

To address this, I've made adjustments to the binding.py file to bypass the track_id during annotation import. This modification appears to solve the issue regarding the "Switch outside" attribute. However, it introduces a new concern: the potential loss of information, including keyframes and track_id.

While this workaround offers a temporary fix, I'm contemplating a more holistic approach that entails properly handling the "outside" attribute during both the export and import processes of annotations. This method could preserve the integrity of the annotations while ensuring the functionality of the "Switch outside" attribute.

I'm reaching out for feedback or suggestions on my proposed solution. Is there a preference for one of these approaches, or might there be another avenue I haven't considered?

Looking forward to your insights.

Checklist

• I submit my changes into the develop branch
• I have created a changelog fragment
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues (see GitHub docs)
• I have increased versions of npm packages if it is necessary
  (cvat-canvas,
  cvat-core,
  cvat-data and
  cvat-ui)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.

[zhiltsov-max]

Hi, thank you for sending the PR! Just disabling tracks in the export doesn't seem to be a proper solution, as the whole point of using the outside property is to use it with tracks. Probably, something along these lines should be adjusted, as this is the common logic for several formats. The handling logic (and so, the solution) should be similar to this fragment.

[Yeek020407]

Hi, thank you for sending the PR! Just disabling tracks in the export doesn't seem to be a proper solution, as the whole point of using the outside property is to use it with tracks. Probably, something along these lines should be adjusted, as this is the common logic for several formats. The handling logic (and so, the solution) should be similar to this fragment.

Thanks for your advice. Will make necessary changes!

[Yeek020407]

Hi @zhiltsov-max , I have made necessary changes and add a test to it. Please have a look

[Yeek020407]

Hii, @zhiltsov-max, I have transferred the code to tests/python and fixed the bug that arose. I have checked with all the REST API, SDK and CLI tests. Please take a look.

[zhiltsov-max]

Hi, please fix the issues reported by the CI.

[Yeek020407]

Hi, please fix the issues reported by the CI.

Hope the changes will fix the issues flagged by the CI.

[zhiltsov-max]

Please update the tests, and we can merge this.

[Drise13]

@Drise13, you wrote about some code snippet in #7669 (comment). Could you provide information about how you could obtain it? Specifically, the points field is expected to be empty in skeleton annotations.

By snippet, I meant "subset of the json data file". If the skeleton points are supposed to be empty, why are there both ones with and without points? Additionally, when running an export to CVAT, it tries to interpolate the tracks of the skeleton which has no points and throws a broadcast error. My intuition tells me that skeleton tracks should not try to be interpolated and only use the points. That seems to work correctly when I cleaned out the skeletons with no points. It may be that it fixed it because there were no other keyframes to try to interpolate from, but I'm not certain on that.

As for the purpose of this PR, if this is not the right PR for this investigation, should I open a new issue instead?

[zhiltsov-max]

@Drise13,

By snippet, I meant "subset of the json data file".

What is this file?

Additionally, when running an export to CVAT, it tries to interpolate the tracks of the skeleton which has no points and throws a broadcast error. My intuition tells me that skeleton tracks should not try to be interpolated and only use the points.

Agree, it seems like the source of the problem, according to your findings.

As for the purpose of this PR, if this is not the right PR for this investigation, should I open a new issue instead?

There is the issue which you mentioned before. If it describes the problem you faced, then it should be discussed there.

[zhiltsov-max]

Please add a more complex test like this
task_test_7669_outside_in_dm_formats_backup_2024_04_17_18_55_18.zip
job_11805_annotations_2024_04_17_16_43_29_coco keypoints 1.0.zip

[Yeek020407]

@zhiltsov-max Thanks for the suggestion. I have added the new test for this.

[Yeek020407]

Hi @zhiltsov-max , I have added the example you provided into our test and added some checking on the number of outside = True in each elements.

[zhiltsov-max]

Thank you for contributing into the project!

[NicolasPetermann134]

Hi @zhiltsov-max !
I'm facing the same issue: ValueError: operands could not be broadcast together with shapes (0,) (6,)

I found two annotations in backup where "points":[] for a skeleton track.

I removed them and tried to create from backup but I am facing this error - KeyError: "There is no item named 'task.json' in the archive"

I am 100% that in my zip file there is a task.json tough.
my zip has following structure:
dataset

• annotations.json
• task.json
• data
  • all images

Would highly appreciate some help.
Regards

[zhiltsov-max]

@NicolasPetermann134,

I don't see how it's related to this PR or issues focused.

my zip has following structure:

What is dataset, a directory? If it's a directory, it should not be present in the backup archive.

[NicolasPetermann134]

@zhiltsov-max
I commented on this case as I saw here #7669 (comment) the idea of deleting the instances with empty points...

dataset is the zipped file.
I have here shared my backups https://1drv.ms/f/s!AiG5XaoBkxRVm9YYwD1G7yHEUrE01g?e=tPhDNN

One is the original backup which is not modified - this one I can use to create a new task without any issues.
For the other one I only modified annotation.json - I deleted there the instances with empty points, I didn't touch the task.json but I get the error KeyError: "There is no item named 'task.json' in the archive"

What could be the issue?

[zhiltsov-max]

@Eldies , please look at the issue.

[NicolasPetermann134]

hi @Eldies, any news on this?

[NicolasPetermann134]

ftr: I could solve the problem. I could export the dataset easily without errors when I tried it on the self hosted version.