Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide custom task name for CVAT #2353

Merged
merged 3 commits into from
Dec 5, 2022

Conversation

Rusteam
Copy link
Contributor

@Rusteam Rusteam commented Nov 28, 2022

What changes are proposed in this pull request?

Closes #1753

  1. Custom task name can be passed for labelling data in CVAT such as dataset.annotate("anno_key", backend="cvat", task_name="Custom task name", ...)
  2. Default task name for CVAT to include an annotation key such as FiftyOne_{dataset_name}_{annotation_key}

How is this patch tested? If it is not, please explain why.

added few lines to existing cvat unit-tests.

Release Notes

Is this a user-facing change that should be mentioned in the release notes?

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release
    notes for FiftyOne users.

A custom task name can be provided with CVAT annotation backend. Default CVAT task name now includes an annotation key.

What areas of FiftyOne does this PR affect?

  • App: FiftyOne application changes
  • Build: Build and test infrastructure changes
  • Core: Core fiftyone Python library changes
  • Documentation: FiftyOne documentation changes
  • Other

@Rusteam
Copy link
Contributor Author

Rusteam commented Nov 28, 2022

@ehofesmann just have a couple of notes about this implementation:

  1. I had to make a slight workaround to get an annotation key from annotation_backend.upload_samples() in order to get the latest annotation run key without changing the main api for all backends.
  2. I was unable to build docs to check that the document update generated nicely because of an error in bash generate_docs.bash. Should I file an issue? Here is the traceback:
Generating model zoo listing page
Traceback (most recent call last):
  File "/Users/rustem.galiullin/Projects/fiftyone/docs/scripts/make_model_zoo_docs.py", line 17, in <module>
    import fiftyone.zoo as foz
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/zoo/__init__.py", line 9, in <module>
    from .datasets import *
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/zoo/datasets/__init__.py", line 20, in <module>
    import fiftyone.utils.data as foud
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/utils/data/__init__.py", line 10, in <module>
    from .base import *
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/utils/data/base.py", line 20, in <module>
    import fiftyone.core.fields as fof
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/core/fields.py", line 21, in <module>
    import fiftyone.core.odm as foo
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/core/odm/__init__.py", line 10, in <module>
    from .database import (
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/core/odm/database.py", line 64, in <module>
    class DatabaseConfigDocument(Document):
  File "/Users/rustem.galiullin/Projects/fiftyone/fiftyone/core/odm/database.py", line 70, in DatabaseConfigDocument
    version = fof.StringField()
AttributeError: partially initialized module 'fiftyone.core.fields' has no attribute 'StringField' (most likely due to a circular import)

_dataset_name = samples_batch._dataset.name.replace(
" ", "_"
)
latest_anno_key = _get_latest_anno_key(samples)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a clever workaround to get the anno_key, but now that config.task_name is supported, I think it would be best to just rely on the user putting their anno_key in the task name if that's what they want.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is unfortunate that we didn't include anno_key in either the upload_annotations() method or in the AnnotationBackendConfig, because it's definitely reasonable to want to use it to generate default values like this 🤦

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it's enough for me that I can provide a custom task name to differentiate between tasks for the same dataset.

Do you think that will require a lot of/breaking changes to pass anno_key to upload_annotations?

Copy link
Contributor

@brimoor brimoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In cases where there are more than one task in the run, what do you think about treating tasks_name as a base name and appending 1, 2, 3, etc to it?

@Rusteam
Copy link
Contributor Author

Rusteam commented Dec 1, 2022

Would not that require getting the count of annotation keys as well?

@brimoor
Copy link
Contributor

brimoor commented Dec 1, 2022

No I'm suggesting adding a separate enhancement to this PR: a single annotation run may create multiple tasks (eg 1 per video for video collections, or if the task_size parameter is provided for image collections). I'm suggesting that you update the logic so that the tasks within the run are named task_name1, task_name2`, etc.

@Rusteam
Copy link
Contributor Author

Rusteam commented Dec 2, 2022

Oh yeah that can be done.

How about the default task name? Do we include an annotation key or not?

@brimoor
Copy link
Contributor

brimoor commented Dec 2, 2022

How about the default task name? Do we include an annotation key or not?

I don't think we should include annotation key in the default task name because there's no official way to get that information into upload_samples() right now without making a breaking change to the annotation API.

@Rusteam Rusteam force-pushed the feature/cvat-custom-task-name branch from c0968c7 to 12208fb Compare December 5, 2022 05:55
Copy link
Contributor

@brimoor brimoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution!

@brimoor brimoor merged commit 0c11bb0 into voxel51:develop Dec 5, 2022
@Rusteam Rusteam deleted the feature/cvat-custom-task-name branch December 5, 2022 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FR] Allow task names to be provided when annotating with CVAT
3 participants