Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tus for task annotations import #4327

Merged
merged 16 commits into from
Mar 11, 2022
Merged

Tus for task annotations import #4327

merged 16 commits into from
Mar 11, 2022

Conversation

klakhov
Copy link
Contributor

@klakhov klakhov commented Feb 11, 2022

Motivation and context

Continue TUS integration to provide chunk upload for large annotation files in Tasks and Jobs
Resolved #964

How has this been tested?

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

@klakhov klakhov changed the title [WIP] Tus for import Tus for task annotations import Feb 15, 2022
@klakhov klakhov changed the title Tus for task annotations import [WIP] Tus for task annotations import Feb 15, 2022
@klakhov klakhov changed the title [WIP] Tus for task annotations import Tus for task annotations import Feb 17, 2022
@klakhov
Copy link
Contributor Author

klakhov commented Feb 17, 2022

@bsekachev @nmanovic Could you please take a look at this patch?

cvat/apps/iam/filters.py Outdated Show resolved Hide resolved
cvat/apps/engine/views.py Outdated Show resolved Hide resolved
# UploadMixin method
def get_upload_dir(self, data_type):
db_model = self.get_object()
if data_type == 'annotations':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the context self is the view. Thus we have self.action. I don't think that we need to pass data_type at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes,self.action can help us on init/finish upload. But when we are uploading file with chunks, we use universal append_tus_chunk action where there is no direct information about data_type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@klakhov , probably it is because append_tus_chunk should be a method, not an action.

cvat/apps/engine/views.py Outdated Show resolved Hide resolved
@klakhov
Copy link
Contributor Author

klakhov commented Feb 21, 2022

I've encountered strange problem with test_tasks_delete cli test. After new functionality in this pr was added, self.mock_stdout.getvalue() in this test started returning:

on upload finished test_task
Task ID 1 deleted

instead of

Task ID 1 deleted

This test checked presence of test_task in those lines, so it failed. But eventually task was deleted, therefore I think it's better to check for Task ID 1 deleted, so I've changed the test too.

@bsekachev
Copy link
Member

Hmm, uploading annotations to an empty task does not work for me. UI says "Annotations have been loaded", but the task is still empty.
Please, check.

@bsekachev
Copy link
Member

@klakhov Could you please resolve conflicts?

Looks like the same issue is still reproducable for me, or maybe did I miss something? (pull, restart server, restart UI, ... ?)

bsekachev
bsekachev previously approved these changes Mar 3, 2022
@klakhov
Copy link
Contributor Author

klakhov commented Mar 5, 2022

@nmanovic I've removed data_type as we discussed and added several actions for each upload type

@nmanovic nmanovic merged commit 1fa2676 into develop Mar 11, 2022
@nmanovic nmanovic deleted the kl/tus-for-import branch March 11, 2022 20:13
@k1won k1won mentioned this pull request Mar 14, 2022
2 tasks
mikhail-treskin pushed a commit to retailnext/cvat that referenced this pull request Jun 22, 2022
* add data_type to tus mixin

* added tus for task annotations import

* added tus for jobs annotations import

* applied comments

* fix test

* fix incorrect upload endpoint

* add location creation based on origin

* remove unused import

* remove data_type

* remove unused comment

* update changelog to new release

Co-authored-by: Nikita Manovich <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Timeout when uploading a big file with annotations
3 participants