Simple Review Pipeline (Server)

[bsekachev]

Motivation and context

The pull request introduces simple validation pipeline to annotations quality control and some basic features for that

New REST API endpoints:
DELETE /api/v1/reviews/<review_id>
POST /api/v1/reviews

DELETE /api/v1/issues/<issue_id>
GET /api/v1/issues/<issue_id>/comments
PATCH /api/v1/issues/<issue_id>

POST /api/v1/comments
PATCH /api/v1/comments/<comment_id>
DELETE /api/v1/comments/<comment_id>

GET /api/v1/jobs/<job_id>/issues
GET /api/v1/jobs/<job_id>/reviews

How has this been tested?

Not tested yet

Checklist

• I submit my changes into the develop branch
• I have added description of my changes into CHANGELOG file
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues (read github docs)
• I have increased versions of npm packages if it is necessary (cvat-canvas,
  cvat-core, cvat-data and cvat-ui)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

[coveralls]

Coverage increased (+0.4%) to 63.547% when pulling 3cdaf1e on bs/review_pipeline into 6ef3ff2 on develop.

[Loc-Vo]

May I know when this feature will be available?
Is there any way I can try this feature?

[bsekachev]

May I know when this feature will be available?
Is there any way I can try this feature?

You can try the feature building cvat from the branch bs/review_pipeline_2

[nmanovic]

@bsekachev , let's discuss REST API design principles which are described in the PR: #2427. I will vote to finalize them and use in the PR.

• DELETE /api/v1/reviews/<review_id>. It is OK.

• DELETE /api/v1/issues/<issue_id>. It is OK

• POST /api/v1/issues/<issue_id>/comments/create. It should be POST /api/v1/comments.

• PATCH /api/v1/issues/<issue_id>/resolve. It should be PATCH /api/v1/issues/<issue_id>.

• PATCH /api/v1/issues/<issue_id>/unresolve. It should be PATCH /api/v1/issues/<issue_id>.

• GET /api/v1/issues/<issue_id>/comments. It should be GET /api/v1/comments?issue=id (but it is OK with the current REST API design).

• PUT /api/v1/comments/<comment_id>. Do you really need PUT or PATCH is enough?

• PATCH /api/v1/comments/<comment_id>. It is OK.

• DELETE /api/v1/comments/<comment_id>. It is OK.

• GET /api/v1/jobs/<job_id>/issues. It should be GET /api/v1/issues?job=id (but it is OK with the current REST API design).

• GET /api/v1/jobs/<job_id>/reviews. It should be GET /api/v1/reviews?job=id (but it is OK with the current REST API design).

• GET /api/v1/jobs/<job_id>/reviews/summary. Probably it should be removed.

• POST /api/v1/jobs/<job_id>/reviews/create. It should be POST /api/v1/reviews.

[bsekachev]

@nmanovic

PUT /api/v1/comments/<comment_id>. Do you really need PUT or PATCH is enough?

Actually it was auto generated with mixins.UpdateModelMixin

GET /api/v1/jobs/<job_id>/reviews/summary

Do not we need to have review summary feature on UI? I explained why we should not perform this logic client-side requesting all the reviews above. Or do you believe it doesn't matter?

Will try to fix others.

[nmanovic]

@nmanovic

PUT /api/v1/comments/<comment_id>. Do you really need PUT or PATCH is enough?

Actually it was auto generated with mixins.UpdateModelMixin

Need to have only PATCH in the case. Please find a workaround. I believe in the past I already did something similar. Probably you can list which methods are supported by a view.

[bsekachev]

@nmanovic

Updated, new list of methods in description.

[chaoyifei]

HI, when this feature will be available

[bsekachev]

@nmanovic Can we merge the PR?

[nmanovic]

@bsekachev , I will do that today if I don't see any other issues.

[Viditagarwal7479]

#2338 (comment)

Do not we need to have review summary feature on UI? I explained why we should not perform this logic client-side requesting all the reviews above. Or do you believe it doesn't matter?

Hi @bsekachev ,
Is this the UI connected to the review pipeline mentioned above?

If we consider the above image as the UI for reviewing. The reviewer will go through the image in case they find any issues in an image they will mark the issue and reject the whole job. This might be fine when only one person is reviewing a part of the data (referred as a job in cvat), i.e. not crowdsourced.