Share without copying & mount cloud storages #2377

Marishka17 · 2020-10-30T17:36:44Z

Motivation and context

Related issue: #204

This PR will allow not copy files inside CVAT when creating a task using share files.
Also PR contains documentation about mounting cloud storage(e.g AWS S3, Azure container, Google Drive) as FUSE.

How has this been tested?

Manually, tests

Checklist

fix uploaded archive/zip
check automount using fstab
check automount using systemd

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

nmanovic · 2020-11-02T07:26:34Z

@Marishka17 , please check the following use case:

Mount a local directory to CVAT server with photos
Try to classify images. For example, if the directory has your family photos, you can have labels me, mother, father, brother, etc.

Expected results:

Task should be created mostly immediately
Navigation by the task should be without significant delays
You can annotate these photos with the annotation speed more than 1000 tags/hour.
Photos are not copied into the CVAT container (one copy of data).
The cache with chunks can be zero size.

coveralls · 2020-11-05T17:17:53Z

Coverage increased (+0.01%) to 61.21% when pulling fcc15ce on mk/share_without_copying_ into d6ac8cc on develop.

cvat-ui/src/components/create-task-page/advanced-configuration-form.tsx

cvat-ui/src/components/create-task-page/create-task-content.tsx

cvat-ui/src/components/create-task-page/advanced-configuration-form.tsx

nmanovic · 2020-11-11T04:55:45Z

cvat/apps/engine/models.py

@@ -54,6 +54,18 @@ def choices(cls):
    def __str__(self):
        return self.value

+class UploadedDataStorageLocationChoice(str, Enum):


I will vote for a shorter name like StorageChoice, DataLocationChoice or something like that.

bsekachev · 2020-11-11T09:53:38Z

@Marishka17 I've just tested the PR in local development.
I have a share directory with some files. I selected on of files and set copy_data to True in advanced dialog.
After that I created the task, went to share directory, renamed the file and everything still works well. Shoudn't this PR allow to avoid copying?
Also in data/data/<id>/raw I see the file, so, it was copied. Am I wrong with understanding something?

Marishka17 · 2020-11-11T10:39:25Z

@bsekachev, by default files are not copied to ../data/id/raw/ and customer works with files from share. If customer selects the copy data checkbox, then files are copied to ../data/id/raw/ and he works with files inside CVAT.
Is there something wrong?

bsekachev · 2020-11-11T10:41:04Z

@bsekachev, by default files are not copied to ../data/id/raw/ and customer works with files from share. If customer selects the copy data checkbox, then files is copied to ../data/id/raw/ and he works with files inside CVAT.
Is there something wrong?

Oh, sure, got it.

bsekachev · 2020-11-11T14:32:02Z

@Marishka17
One enhancement I could suggest is when we have a task with data on the share and file is not available on the share anymore (I just renamed a file to emulate it), we have infinite loading of an annotation view. No any errors occur client side, client just receives empty zip archive. I believe it would be better if UI says that the file disappear.

It could be a different PR I suppose.

nmanovic · 2020-11-17T05:21:06Z

@Marishka17
One enhancement I could suggest is when we have a task with data on the share and file is not available on the share anymore (I just renamed a file to emulate it), we have infinite loading of an annotation view. No any errors occur client side, client just receives empty zip archive. I believe it would be better if UI says that the file disappear.

It could be a different PR I suppose.

I think we need to address it in the PR. It is a bug and could lead to a lot of issues which will difficult to investigate.

nmanovic · 2020-11-30T06:33:53Z

@azhavoro , could you please look at the PR and provide your decision?

cvat-core/src/server-proxy.js

azhavoro · 2020-12-01T07:44:22Z

@Marishka17 PR is ok for me, please fix the conflict with views.py

iraadit · 2021-02-22T13:40:16Z

Is the "Share without copying" working with the cli?

Marishka17 · 2021-02-24T14:41:35Z

@iraadit, there is currently no way to use this feature using the cli. Could you please create the issue?

iraadit · 2021-02-24T20:50:06Z

@iraadit, there is currently no way to use this feature using the cli. Could you please create the issue?

Hi, I created the new issue #2862

Thank you

bsekachev and others added 7 commits October 27, 2020 15:14

tmp

aecbb5a

Added checkbox 'copy_data'

5762fdd

Updated versions

dac6128

Added location of uploaded data

b97a6cd

temp

a1b32b4

Added guide how mount cloud storages as fuse

c378989

Added some test

6e5648f

Marishka17 requested review from bsekachev and azhavoro October 30, 2020 17:36

Marishka17 requested a review from nmanovic as a code owner October 30, 2020 17:36

nmanovic closed this Nov 2, 2020

azhavoro reopened this Nov 2, 2020

Marishka17 added 4 commits November 5, 2020 12:42

Fix disappearing of the checkbox when the button is pressed

89d572d

fix

fb8f5ff

Added tests

0bf3153

Added automount using /etc/fstab

cf406f5

Marishka17 added 3 commits November 9, 2020 14:16

Fixed codacy issues

39c2d17

Merge branch 'develop' into mk/share_without_copying_

f542199

Fixed typo

7309d99

Marishka17 changed the title ~~[WIP] Share without copying & mount cloud storages~~ Share without copying & mount cloud storages Nov 9, 2020

bsekachev reviewed Nov 10, 2020

View reviewed changes

Fixes

143fe28

nmanovic reviewed Nov 11, 2020

View reviewed changes

Renaming

d2e186a

bsekachev previously approved these changes Nov 11, 2020

View reviewed changes

Marishka17 changed the title ~~Share without copying & mount cloud storages~~ [WIP] Share without copying & mount cloud storages Nov 20, 2020

Marishka17 added 4 commits November 26, 2020 13:37

Fix: server returns a response with 404 code

ddf2268

Added handling of the case when a shared files are no longer available

0835f26

Merge branch 'develop' into mk/share_without_copying_

c1da108

Updated migration dependency

3096111

Marishka17 dismissed bsekachev’s stale review via 3096111 November 26, 2020 11:56

Marishka17 changed the title ~~[WIP] Share without copying & mount cloud storages~~ Share without copying & mount cloud storages Nov 27, 2020

Marishka17 requested a review from bsekachev November 27, 2020 08:27

bsekachev reviewed Nov 30, 2020

View reviewed changes

cvat-core/src/server-proxy.js Outdated Show resolved Hide resolved

fix

ee8ce64

Marishka17 added 2 commits December 1, 2020 10:55

Merge branch 'develop' into mk/share_without_copying_

d52d5ba

Updated CHANGELOG

0292fbb

azhavoro previously approved these changes Dec 1, 2020

View reviewed changes

bsekachev previously approved these changes Dec 1, 2020

View reviewed changes

Merged develop

fcc15ce

bsekachev dismissed stale reviews from azhavoro and themself via fcc15ce December 2, 2020 06:51

bsekachev merged commit 004cb64 into develop Dec 2, 2020

bsekachev deleted the mk/share_without_copying_ branch December 2, 2020 09:05

bsekachev mentioned this pull request Feb 16, 2021

Any way to avoid copying and compressing files when creating new task? #204

Closed

Marishka17 linked an issue Feb 16, 2021 that may be closed by this pull request

Any way to avoid copying and compressing files when creating new task? #204

Closed

iraadit mentioned this pull request Feb 24, 2021

Add possibility to not copy files when using share files and cli #2862

Closed

2 tasks

shortcipher3 mentioned this pull request Aug 11, 2021

Add option to not copy files when using share files and web ui/cli #3542

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Share without copying & mount cloud storages #2377

Share without copying & mount cloud storages #2377

Marishka17 commented Oct 30, 2020 •

edited by bsekachev

Loading

nmanovic commented Nov 2, 2020 •

edited

Loading

coveralls commented Nov 5, 2020 •

edited

Loading

nmanovic Nov 11, 2020 •

edited

Loading

Marishka17 Nov 11, 2020

bsekachev commented Nov 11, 2020

Marishka17 commented Nov 11, 2020 •

edited

Loading

bsekachev commented Nov 11, 2020

bsekachev commented Nov 11, 2020

nmanovic commented Nov 17, 2020

nmanovic commented Nov 30, 2020

azhavoro commented Dec 1, 2020

iraadit commented Feb 22, 2021

Marishka17 commented Feb 24, 2021

iraadit commented Feb 24, 2021

Share without copying & mount cloud storages #2377

Share without copying & mount cloud storages #2377

Conversation

Marishka17 commented Oct 30, 2020 • edited by bsekachev Loading

Motivation and context

How has this been tested?

Checklist

Checklist

License

nmanovic commented Nov 2, 2020 • edited Loading

coveralls commented Nov 5, 2020 • edited Loading

nmanovic Nov 11, 2020 • edited Loading

Choose a reason for hiding this comment

Marishka17 Nov 11, 2020

Choose a reason for hiding this comment

bsekachev commented Nov 11, 2020

Marishka17 commented Nov 11, 2020 • edited Loading

bsekachev commented Nov 11, 2020

bsekachev commented Nov 11, 2020

nmanovic commented Nov 17, 2020

nmanovic commented Nov 30, 2020

azhavoro commented Dec 1, 2020

iraadit commented Feb 22, 2021

Marishka17 commented Feb 24, 2021

iraadit commented Feb 24, 2021

Marishka17 commented Oct 30, 2020 •

edited by bsekachev

Loading

nmanovic commented Nov 2, 2020 •

edited

Loading

coveralls commented Nov 5, 2020 •

edited

Loading

nmanovic Nov 11, 2020 •

edited

Loading

Marishka17 commented Nov 11, 2020 •

edited

Loading