Skip to content

Commit

Permalink
Update documentation. Added tutorial about attach google cloud storage (
Browse files Browse the repository at this point in the history
#3995)

* add google cloud storage tutorial

* update attach-cloud-storage.md

* apply suggestion

* apply suggestion v 2

* update images

* fix mistakes

* Update site/content/en/docs/manual/basics/attach-cloud-storage.md

Co-authored-by: Maria Khrustaleva <[email protected]>

* fix linter error

Co-authored-by: Maria Khrustaleva <[email protected]>
  • Loading branch information
TOsmanov and Maria Khrustaleva authored Dec 10, 2021
1 parent e375c70 commit e39a17b
Show file tree
Hide file tree
Showing 6 changed files with 92 additions and 4 deletions.
96 changes: 92 additions & 4 deletions site/content/en/docs/manual/basics/attach-cloud-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,11 +164,72 @@ Prepare the dataset as in the point [prepare dataset](#prepare-dataset).
to the files in the container. Click `select a file` and select manifest file, in order to upload file to the root
of the container leave blank `upload to folder` field.

Now you can [attach new cloud storage into CVAT](#attach-new-cloud-storage).

## Using Google Cloud Storage

### Create Google account

First, create a Google account, go to [account login page](https://accounts.google.com/) and click `Create account`.
After, go to the [Google Cloud page](https://cloud.google.com), click `Get started`, enter the required data
and accept the terms of service (you'll need credit card information to register).

### Create a bucket

Your first project will be created automatically, you can see it on the [cloud resource manager page](https://console.cloud.google.com/cloud-resource-manager).
To create a bucket, go to the [cloud storage page](https://console.cloud.google.com/storage/browser)
and press `Create bucket`. Next, enter the name of the bucket, add labels if necessary, select the type of location
for example region and the location nearest to you, select storage class, when selecting access control
you can enable `Enforce public access prevention on this bucket` (if you plan to have anonymous access to your bucket,
it should be disabled) you can select `Uniform` or `Fine-grained` access control, if you need protection of your
object data select protect object data type. When all the information is entered click `Create` to create the bucket.

![](/images/google_cloud_storage_tutorial1.jpg)

### Upload

Prepare the dataset as in the point [prepare dataset](#prepare-dataset).

To upload files, you can simply drag and drop files and folders into a browser window
or use the `upload folder` and/or `upload files`.

### Access permissions

To access Google Cloud Storage from CVAT you will need a `Project ID`
you can find it by going to [cloud resource manager page](https://console.cloud.google.com/cloud-resource-manager)

![](/images/google_cloud_storage_tutorial5.jpg)

#### Create a service account and key file

To access your bucket you need a key file and a service account. To create a service account,
go to `IAM & Admin`/`Service Accounts` and press `Create Service Account`. Enter your account
name and click `Create And Continue`. Select a role for example `Basic`/`Viewer`.
Next, you can give access rights to the service account, to complete click `Done`.

![](/images/google_cloud_storage_tutorial2.jpg)

The account you created will appear in the service accounts list, open it and go to the `Keys` tab.
To create a key, click `ADD` and select `Create new key`, next you need to choose the key type `JSON` and select `Create`.
The key file will be downloaded automatically.

![](/images/google_cloud_storage_tutorial3.jpg)

[Learn more about creating keys](https://cloud.google.com/docs/authentication/getting-started).

#### Anonymous access

To configure anonymous access, open your bucket and go to the permissions tab click `ADD` to add new principals.
In `new principals` field specify `allUsers`, select role for example `Cloud Storage Legacy`/`Storage Legacy Bucket Reader`
and press `SAVE`.

![](/images/google_cloud_storage_tutorial4.jpg)

Now you can attach new cloud storage into CVAT.

## Attach new cloud storage

After you upload the dataset and manifest file to AWS-S3 or Azure Blob Container
After you upload the dataset and manifest file to AWS-S3, Azure Blob Container or Google Cloud Storage
you will be able to attach a cloud storage. To do this, press the `Attach new cloud storage`
button on the `Cloud storages` page and fill out the following form:

Expand All @@ -179,7 +240,7 @@ button on the `Cloud storages` page and fill out the following form:
of an item on cloud storages page.
- `Provider` - choose provider of the cloud storage:

- [AWS-S3](#using-aws-s3):
- [**AWS-S3**](#using-aws-s3):

- [`Bucket`](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket) - cloud storage bucket name.

Expand All @@ -191,14 +252,14 @@ button on the `Cloud storages` page and fill out the following form:
- `ACCESS KEY ID`
- `SECRET ACCESS KEY ID`

- `Anonymous access` - For anonymous access, you need to enable public access to bucket.
- `Anonymous access` - for anonymous access, you need to enable public access to bucket.

- `Region` - here you can choose a region from the list or add a new one. To get more information click
on [`?`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions).

</br>

- [Azure Blob Container](https://docs.microsoft.com/en-us/azure/storage/blobs/):
- [**Azure Blob Container**](https://docs.microsoft.com/en-us/azure/storage/blobs/):

- `Container name` - name of the cloud storage container.

Expand All @@ -215,6 +276,33 @@ button on the `Cloud storages` page and fill out the following form:
- `Account name` - storage account name.

</br>

- [**Google Cloud**](https://cloud.google.com/docs):

- [`Bucket name`](https://cloud.google.com/storage/docs/creating-buckets) - cloud storage bucket name,
you can find the created bucket on the [storage browser page](https://console.cloud.google.com/storage/browser).

- `Authorization type`:

- [`Key file`](#create-a-service-account-and-key-file) - you can drag a key file to the area `attach a file`
or click on the area to select the key file through the explorer. If the environment variable
`GOOGLE_APPLICATION_CREDENTIALS` is specified for an environment with a deployed CVAT instance, then it will
be used if you do not attach the key file
([more about `GOOGLE_APPLICATION_CREDENTIALS`](https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable)).

- [`Anonymous access`](#anonymous-access) - for anonymous access, you need to enable public access to bucket.

- `Prefix` - used to filter data from the bucket.

- [`Project ID`](https://cloud.google.com/resource-manager/docs/creating-managing-projects) - you can find
the created project on the [cloud resource manager page](https://console.cloud.google.com/cloud-resource-manager),
note that the project name does not match the project ID.

- `Location` - here you can choose a region from the list or add a new one. To get more information click
on [`?`](https://cloud.google.com/storage/docs/locations#available-locations).

</br>

- `Manifest` - the path to the manifest file on your cloud storage.
You can add multiple manifest files using the `Add manifest` button.
You can find on how to prepare dataset manifest [`here`](/docs/manual/advanced/dataset_manifest/).
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit e39a17b

Please sign in to comment.