Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK layer 2 - cover RC1 usecases #4813

Merged
merged 151 commits into from
Aug 26, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
e34731d
Fix generated OpenAPI schema
zhiltsov-max Jun 20, 2022
4f1940b
Update changelog
zhiltsov-max Jun 20, 2022
cf759f5
Merge branch 'develop' into zm/fix-openapi-schema
zhiltsov-max Jun 20, 2022
aad0ccc
Make data-related fields in the Task optional
zhiltsov-max Jun 21, 2022
50959a4
Allow null organization in Task, Project, Clouds
zhiltsov-max Jun 21, 2022
f43d47a
Add a task to generate schema
zhiltsov-max Jun 21, 2022
a16d103
Fix attribute values declaration - it is a list of strings
zhiltsov-max Jun 21, 2022
38b7d82
Merge branch 'develop' into zm/fix-openapi-schema
zhiltsov-max Jun 22, 2022
7248a7f
add generator config
zhiltsov-max Jun 24, 2022
4bda27e
Update generator config and templates
zhiltsov-max Jun 27, 2022
e953a23
Add generated code
zhiltsov-max Jun 27, 2022
6d9f43e
Add tasks tests
zhiltsov-max Jun 27, 2022
e73828e
Add sample
zhiltsov-max Jun 27, 2022
35b764a
Fixes to some methods
zhiltsov-max Jun 27, 2022
4acffb5
Add org parameters in schema globally
zhiltsov-max Jun 28, 2022
8c5f2fc
Fix tasks patch tests
zhiltsov-max Jun 28, 2022
aa8d8f0
Enable file uploading in task data
zhiltsov-max Jul 1, 2022
d69af23
Merge branch 'develop' into zm/add-cvat-sdk
zhiltsov-max Jul 1, 2022
5210bce
Merge with develop, fix issues
zhiltsov-max Jul 1, 2022
aa0a8ff
Fix schema generation error
zhiltsov-max Jul 1, 2022
1e5f915
Add introspection to models and apis, embed apis into client, add cod…
zhiltsov-max Jul 4, 2022
78eb73e
Add debug task for api code processing
zhiltsov-max Jul 4, 2022
b580feb
Improve type annotations in api
zhiltsov-max Jul 4, 2022
a491203
Add design document
zhiltsov-max Jul 5, 2022
ac41fd4
Fix launch,json
zhiltsov-max Jul 5, 2022
03ddd56
Simplify imports
zhiltsov-max Jul 5, 2022
122635c
Fix tests
zhiltsov-max Jul 5, 2022
4ca10bc
Move operation method after api tag
zhiltsov-max Jul 5, 2022
1519d3a
Rename _preload_content to _parse_response
zhiltsov-max Jul 5, 2022
a123d6f
Regularize operation return types in API
zhiltsov-max Jul 7, 2022
13e9644
Add default _configuration arg in model creation, fix some type annot…
zhiltsov-max Jul 7, 2022
347e414
Update tests
zhiltsov-max Jul 8, 2022
c9d15d0
Add schema for page_size parameter
zhiltsov-max Jul 8, 2022
97d5beb
Enable users tests
zhiltsov-max Jul 11, 2022
c44f6af
Fix schema for tus-related endpoints (PATCH)
zhiltsov-max Jul 12, 2022
611d7fb
ProjectsApi: fix serializers, fix tests
zhiltsov-max Jul 12, 2022
6e22c63
Merge branch 'develop' into zm/add-cvat-sdk
zhiltsov-max Jul 12, 2022
17d790c
Update changelog
zhiltsov-max Jul 12, 2022
2b8a460
Dont run pylint on generated api client
zhiltsov-max Jul 12, 2022
3784e79
Exclude cvat-api-client from bandit and remark checks
zhiltsov-max Jul 12, 2022
0a991ed
Update CI for cvat-api-client dependency
zhiltsov-max Jul 12, 2022
525feb5
fix linter
zhiltsov-max Jul 12, 2022
af76558
Move remarkignore
zhiltsov-max Jul 12, 2022
434517b
Update docs, fix linter
zhiltsov-max Jul 12, 2022
cd5e1b9
Fix package installation
zhiltsov-max Jul 13, 2022
24837d8
Fix dev instructions
zhiltsov-max Jul 14, 2022
b2c362a
Revert serializer change
zhiltsov-max Jul 14, 2022
e913491
Merge branch 'develop' into zm/add-cvat-sdk
zhiltsov-max Jul 14, 2022
07069c4
Rename api-client to sdk
zhiltsov-max Jul 14, 2022
31034f1
Merge branch 'zm/add-cvat-sdk' of https://github.com/cvat-ai/cvat int…
zhiltsov-max Jul 14, 2022
d0e9075
Format files
zhiltsov-max Jul 14, 2022
06481e1
Add core client base to sdk
zhiltsov-max Jul 14, 2022
aa8672e
Style fixes
zhiltsov-max Jul 14, 2022
cb3576f
Implement task creation
zhiltsov-max Jul 15, 2022
d18271e
Update license headers, add task creation test
zhiltsov-max Jul 18, 2022
0de20f1
remove sample
zhiltsov-max Jul 18, 2022
63ce027
Fix overlap field schema
zhiltsov-max Jul 19, 2022
2af421a
Fix task creation test
zhiltsov-max Jul 19, 2022
bf57543
Manage session cookies in ApiClient
zhiltsov-max Jul 19, 2022
3e4e859
Don't overwrite gitignore
zhiltsov-max Jul 19, 2022
905dd92
Fix type annotation
zhiltsov-max Jul 19, 2022
c27b494
Add list fetching utility
zhiltsov-max Jul 20, 2022
55c120b
Fix users list response
zhiltsov-max Jul 20, 2022
e060f53
Fix example formatting
zhiltsov-max Jul 20, 2022
4a8dc99
Fix license headers
zhiltsov-max Jul 20, 2022
e6529cd
Update bulk task removal
zhiltsov-max Jul 20, 2022
64aecbe
Add progressreporter model, remove tqdm dependency
zhiltsov-max Jul 20, 2022
db96c6b
Fix enpoint names for cloudstorages
zhiltsov-max Jul 20, 2022
11e0ad0
Fix links in api contacts
zhiltsov-max Jul 20, 2022
c6349f4
Move l2 sources
zhiltsov-max Jul 20, 2022
16f09b3
Merge branch 'develop' into zm/sdk-l2
zhiltsov-max Jul 20, 2022
c24dd44
Change contact
zhiltsov-max Jul 20, 2022
9ef4b47
Fix linter error
zhiltsov-max Jul 20, 2022
84cfe84
Extract class for uploading
zhiltsov-max Jul 20, 2022
2f23938
Split cvat client into usecases
zhiltsov-max Jul 21, 2022
412d0fb
Enable sdk tests
zhiltsov-max Jul 26, 2022
2f31269
Move to OO-like approach
zhiltsov-max Jul 26, 2022
c1760fa
Fix test, clarify model interfaces
zhiltsov-max Jul 27, 2022
f05482a
Enable more task tests, introduce config
zhiltsov-max Jul 27, 2022
7bdce47
Enable task file downloading tests
zhiltsov-max Jul 27, 2022
80b36da
Enable frame downloading tests
zhiltsov-max Jul 27, 2022
65a3f9d
Fix annotation and backup uploading
zhiltsov-max Jul 28, 2022
e1bbc5f
Fix backup uploading
zhiltsov-max Jul 29, 2022
dece1bd
Clean imports
zhiltsov-max Jul 29, 2022
f4333f9
Fix param
zhiltsov-max Jul 29, 2022
cdb1be3
Rename CvatClient to Client
zhiltsov-max Aug 1, 2022
b776f18
Move types component
zhiltsov-max Aug 1, 2022
3414a68
Refactor cli code
zhiltsov-max Aug 1, 2022
4d01fb8
Refactor module placement
zhiltsov-max Aug 1, 2022
45c02c7
Add missing args
zhiltsov-max Aug 1, 2022
57e4116
Fix cookie handling
zhiltsov-max Aug 1, 2022
10875ab
Implement cli tests
zhiltsov-max Aug 1, 2022
d9a6e82
Add cli tests task in launch.json
zhiltsov-max Aug 1, 2022
ed2ae4b
Merge branch 'develop' into zm/sdk-l2
zhiltsov-max Aug 1, 2022
5b51bb2
Reorganize python test placement
zhiltsov-max Aug 2, 2022
8eaaf18
Fix tests after merge
zhiltsov-max Aug 2, 2022
89c2725
Add file uploading cli tests
zhiltsov-max Aug 2, 2022
1bc7cf6
Update launch.json
zhiltsov-max Aug 2, 2022
a05beec
Update github workflows
zhiltsov-max Aug 2, 2022
81233fc
Fix login problem
zhiltsov-max Aug 3, 2022
c2a5f3e
Fix imports
zhiltsov-max Aug 3, 2022
b17acc5
Fix linter errors
zhiltsov-max Aug 3, 2022
b8a7087
Fix ci
zhiltsov-max Aug 3, 2022
a4e51d4
Add a test for failed task creation
zhiltsov-max Aug 3, 2022
fb1e358
Add auth tests
zhiltsov-max Aug 3, 2022
a23ff17
Update issue tests to use sdk
zhiltsov-max Aug 3, 2022
39f4715
Remove PUT methods on model-specific endpoints
zhiltsov-max Aug 4, 2022
6798791
Fix auth tests
zhiltsov-max Aug 4, 2022
9da9436
Add issue removal tests
zhiltsov-max Aug 4, 2022
af6c598
Add jobs and downloading
zhiltsov-max Aug 4, 2022
6ef21e3
Fix potential logging problem
zhiltsov-max Aug 4, 2022
e3ea337
Refactor downloading and uploading
zhiltsov-max Aug 4, 2022
7a5df89
Fix tests, add Issues and Comments, move group operations to proxy cl…
zhiltsov-max Aug 5, 2022
00fdcf6
Merge branch 'develop' into zm/sdk-l2
zhiltsov-max Aug 8, 2022
e9c3113
Add TUS dep in SDK
zhiltsov-max Aug 8, 2022
db80b3e
Update cvat-sdk file layout
zhiltsov-max Aug 12, 2022
ee84cc9
Upgrade pylint and clean files
zhiltsov-max Aug 12, 2022
52c3e0f
fix check
zhiltsov-max Aug 12, 2022
5acea16
Update black and isort on CI
zhiltsov-max Aug 12, 2022
f7d786a
fix black and isort on ci
zhiltsov-max Aug 12, 2022
fd3220d
fix exit code
zhiltsov-max Aug 12, 2022
1b66328
fix linter error
zhiltsov-max Aug 12, 2022
a397cbd
Merge branch 'zm/sdk-l2' into zm/sdk-l2-me
zhiltsov-max Aug 12, 2022
dc4b551
Move helpers module
zhiltsov-max Aug 12, 2022
2b6e5f3
Fix generator version, enable sdk building on CI
zhiltsov-max Aug 12, 2022
08c3adf
fix exit code
zhiltsov-max Aug 12, 2022
5aaa323
Fix linter error
zhiltsov-max Aug 12, 2022
d696eb6
Merge branch 'zm/sdk-l2' into zm/sdk-l2-me
zhiltsov-max Aug 12, 2022
57620fc
Add support for OpenApiTypes in PolymorphicProxySerializer, fix job/a…
zhiltsov-max Aug 15, 2022
5afa6b5
Update job rest api tests to use sdk, fix job and job annotations end…
zhiltsov-max Aug 15, 2022
f7541b8
Introduce repositories in sdk level2
zhiltsov-max Aug 16, 2022
f023d11
Fix annotations and datasets schema, add rest api tests
zhiltsov-max Aug 16, 2022
12e5218
Merge branch 'develop' into zm/sdk-l2-me
zhiltsov-max Aug 16, 2022
2adcaad
Remove extra file
zhiltsov-max Aug 16, 2022
3fadfa5
Implement jobs sdk, add missing task methods. Add projects, issues, c…
zhiltsov-max Aug 17, 2022
aec4c29
Refactor sdk code, add job commits and issues
zhiltsov-max Aug 18, 2022
54e415c
Fix partial_update schemas, add issues and comments tests
zhiltsov-max Aug 19, 2022
a7c03a5
Make some parameters optional for annotation serializers
zhiltsov-max Aug 19, 2022
770fa1d
Treat message optional
zhiltsov-max Aug 19, 2022
6540888
Fix other model list endpoints in views
zhiltsov-max Aug 19, 2022
20a1e1b
Add missing tests and annotations methods in proxies
zhiltsov-max Aug 19, 2022
af1e15b
Fix show empty tasks (#100)
Marishka17 Aug 19, 2022
75ba69b
[Snyk] Upgrade dotenv-webpack from 7.1.1 to 8.0.0 (#98)
azhavoro Aug 19, 2022
7b492ee
Disable bandit for tests
zhiltsov-max Aug 22, 2022
f81f982
Fix bandit checks
zhiltsov-max Aug 22, 2022
30b60df
Fix bandit exclude
zhiltsov-max Aug 22, 2022
58e4fe5
Merge branch 'develop' into zm/sdk-l2-me
zhiltsov-max Aug 22, 2022
4f0aac5
Update server tests for task and jobs PUT method
zhiltsov-max Aug 22, 2022
b6369f7
Update changelog
zhiltsov-max Aug 22, 2022
24c65cc
Fix operation names in schema
zhiltsov-max Aug 23, 2022
edb3a77
Merge branch 'develop' into zm/sdk-l2-me
nmanovic Aug 26, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .bandit
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@
# B406 : import_xml_sax
# B410 : import_lxml
skips: B101,B102,B320,B404,B406,B410
exclude: **/tests/**,tests
2 changes: 1 addition & 1 deletion .github/workflows/bandit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:

echo "Bandit version: "$(bandit --version | head -1)
echo "The files will be checked: "$(echo $CHANGED_FILES)
bandit $CHANGED_FILES --exclude '**/tests/**' -a file --ini ./.bandit -f html -o ./bandit_report/bandit_checks.html
bandit -a file --ini .bandit -f html -o ./bandit_report/bandit_checks.html $CHANGED_FILES
deactivate
else
echo "No files with the \"py\" extension found"
Expand Down
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Possibility to display tags on frame
- Support source and target storages (server part)
- Tests for import/export annotation, dataset, backup from/to cloud storage
- Added Python SDK package (`cvat-sdk`)
- Added Python SDK package (`cvat-sdk`) (<https://github.com/opencv/cvat/pull/4813>)
- Previews for jobs
- Documentation for LDAP authentication (<https://github.com/cvat-ai/cvat/pull/39>)
- OpenCV.js caching and autoload (<https://github.com/cvat-ai/cvat/pull/30>)
Expand All @@ -28,7 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- Bumped nuclio version to 1.8.14
- Simplified running REST API tests. Extended CI-nightly workflow
- REST API tests are partially moved to Python SDK (`users`, `projects`, `tasks`)
- REST API tests are partially moved to Python SDK (`users`, `projects`, `tasks`, `issues`)
- cvat-ui: Improve UI/UX on label, create task and create project forms (<https://github.com/cvat-ai/cvat/pull/7>)
- Removed link to OpenVINO documentation (<https://github.com/cvat-ai/cvat/pull/35>)
- Clarified meaning of chunking for videos
Expand All @@ -51,6 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Image search in cloud storage (<https://github.com/cvat-ai/cvat/pull/8>)
- Reset password functionality (<https://github.com/cvat-ai/cvat/pull/52>)
- Creating task with cloud storage data (<https://github.com/cvat-ai/cvat/pull/116>)
- Show empty tasks (<https://github.com/cvat-ai/cvat/pull/100>)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhiltsov-max , I'm not sure why we have the line in the PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure either. Probably, we need to review changelog links.


### Security
- TDB
Expand Down
21 changes: 10 additions & 11 deletions cvat-cli/src/cvat_cli/cli.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Copyright (C) 2020-2022 Intel Corporation
# Copyright (C) 2022 CVAT.ai Corporation
#
# SPDX-License-Identifier: MIT
Expand All @@ -11,7 +10,7 @@
import tqdm
from cvat_sdk import Client, models
from cvat_sdk.core.helpers import TqdmProgressReporter
from cvat_sdk.core.types import ResourceType
from cvat_sdk.core.proxies.tasks import ResourceType


class CLI:
Expand All @@ -26,7 +25,7 @@ def __init__(self, client: Client, credentials: Tuple[str, str]):

def tasks_list(self, *, use_json_output: bool = False, **kwargs):
"""List all tasks in either basic or JSON format."""
results = self.client.list_tasks(return_json=use_json_output, **kwargs)
results = self.client.tasks.list(return_json=use_json_output, **kwargs)
if use_json_output:
print(json.dumps(json.loads(results), indent=2))
else:
Expand All @@ -50,7 +49,7 @@ def tasks_create(
"""
Create a new task with the given name and labels JSON and add the files to it.
"""
task = self.client.create_task(
task = self.client.tasks.create_from_data(
spec=models.TaskWriteRequest(name=name, labels=labels, **kwargs),
resource_type=resource_type,
resources=resources,
Expand All @@ -66,7 +65,7 @@ def tasks_create(

def tasks_delete(self, task_ids: Sequence[int]) -> None:
"""Delete a list of tasks, ignoring those which don't exist."""
self.client.delete_tasks(task_ids=task_ids)
self.client.tasks.remove_by_ids(task_ids=task_ids)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhiltsov-max , I don't know all use cases, but probably tasks.delete(task_ids) is better. First argument can be array of ids or array of tasks. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its almost just a copy-paste from what CLI had. I can't say I prefer one way over another. Both work well and both are convertible to each other. I can say that your variant is more focused on manual typing, while the other is better for passing a container without the need of unpacking. I'm not sure if delete(id1, id2, id3) is more frequent pattern in the client code than delete(ids).


def tasks_frames(
self,
Expand All @@ -80,11 +79,11 @@ def tasks_frames(
Download the requested frame numbers for a task and save images as
task_<ID>_frame_<FRAME>.jpg.
"""
self.client.retrieve_task(task_id=task_id).download_frames(
self.client.tasks.retrieve(obj_id=task_id).download_frames(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if it is possible or not, but it will be great if the first argument is ID. Thus we can just write tasks.retrieve(task_id).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I got the question. It is ID. Did you mean rename id to id?

frame_ids=frame_ids,
outdir=outdir,
quality=quality,
filename_pattern="task_{task_id}_frame_{frame_id:06d}{frame_ext}",
filename_pattern=f"task_{task_id}" + "_frame_{frame_id:06d}{frame_ext}",
)

def tasks_dump(
Expand All @@ -99,7 +98,7 @@ def tasks_dump(
"""
Download annotations for a task in the specified format (e.g. 'YOLO ZIP 1.0').
"""
self.client.retrieve_task(task_id=task_id).export_dataset(
self.client.tasks.retrieve(obj_id=task_id).export_dataset(
format_name=fileformat,
filename=filename,
pbar=self._make_pbar(),
Expand All @@ -112,7 +111,7 @@ def tasks_upload(
) -> None:
"""Upload annotations for a task in the specified format
(e.g. 'YOLO ZIP 1.0')."""
self.client.retrieve_task(task_id=task_id).import_annotations(
self.client.tasks.retrieve(obj_id=task_id).import_annotations(
format_name=fileformat,
filename=filename,
status_check_period=status_check_period,
Expand All @@ -121,13 +120,13 @@ def tasks_upload(

def tasks_export(self, task_id: str, filename: str, *, status_check_period: int = 2) -> None:
"""Download a task backup"""
self.client.retrieve_task(task_id=task_id).download_backup(
self.client.tasks.retrieve(obj_id=task_id).download_backup(
filename=filename, status_check_period=status_check_period, pbar=self._make_pbar()
)

def tasks_import(self, filename: str, *, status_check_period: int = 2) -> None:
"""Import a task from a backup file"""
self.client.create_task_from_backup(
self.client.tasks.create_from_backup(
filename=filename, status_check_period=status_check_period, pbar=self._make_pbar()
)

Expand Down
2 changes: 1 addition & 1 deletion cvat-cli/src/cvat_cli/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
import os
from distutils.util import strtobool

from cvat_sdk.core.types import ResourceType
from cvat_sdk.core.proxies.tasks import ResourceType

from .version import VERSION

Expand Down
2 changes: 1 addition & 1 deletion cvat-sdk/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -74,4 +74,4 @@ cvat_sdk/api_client/
requirements/
docs/
setup.py
README.md
README.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you put setup.py and README.md into .gitignore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are also generated automatically. We decided to remove such files. Moreover, setup.py is meaningless without REST client code.

Loading