Incremental save of annotation. #120

azhavoro · 2018-10-05T13:57:08Z

No description provided.

bsekachev

++: /* exported createExportContainer ExportType */
Also need add these symbols into eslintrc.conf.js global variables

bsekachev · 2018-10-08T06:53:27Z

cvat/apps/engine/static/engine/js/shapes.js

@@ -3150,6 +3147,25 @@ class PointsView extends PolyShapeView {
    }
 }

+function createExportContainer() {


I believe it is better to move this function into base.js
The same for ExportType object

bsekachev · 2018-10-08T07:35:04Z

cvat/apps/engine/annotation.py

@@ -970,13 +1090,13 @@ def _save_paths_to_db(self):
                # for Postgres bulk_create will return objects with ids even ids
                # are auto incremented. Thus we will not be inside the 'if'.
                if shape_type == 'polygon_paths':
-                    db_paths = list(self.db_job.objectpath_set.filter(shapes="polygons"))
+                    db_paths = list(self.db_job.objectpath_set.exclude(id__in=db_paths_ids).filter(shapes="polygons"))


If we use exclude() with indexes which have existed before bulk_create(), we need only one row:
db_paths = list(self.db_job.objectpath_set.exclude(id__in=db_paths_ids)
Right?

cvat/apps/engine/annotation.py

bsekachev · 2018-10-08T07:41:25Z

cvat/apps/engine/models.py

@@ -145,6 +145,7 @@ class Meta:
 class Shape(models.Model):
    occluded = models.BooleanField(default=False)
    z_order = models.IntegerField(default=0)
+    client_id = models.IntegerField(default=-1)


Probably better add client_id into Annotation class instead of Shape and ObjectPath

bsekachev · 2018-10-08T07:43:03Z

cvat/apps/engine/annotation.py

-        self._save_shapes_to_db()
-        self._save_paths_to_db()
+    def _get_relaited_obj_field_name(self, shape_type):
+        'polygon_paths', 'polyline_paths', 'points_paths', 'box_paths'


Is it extra line?

bsekachev · 2018-10-08T08:39:44Z

cvat/apps/engine/static/engine/js/shapeCollection.js

@@ -51,6 +57,8 @@ class ShapeCollectionModel extends Listener {
        this._colorIdx = 0;
        this._filter = new FilterModel(() => this.update());
        this._splitter = new ShapeSplitter();
+        this._erased = false;
+        this._initialShapes = [];


This field is list, but in this.updateHash() the same name is used as dictionary

bsekachev · 2018-10-08T08:43:29Z

cvat/apps/engine/static/engine/js/shapeCollection.js

@@ -221,49 +229,84 @@ class ShapeCollectionModel extends Listener {
        return this;
    }

+    _getExportTargetContainer(export_type, shape_type, container) {


Are you sure it is should be a method? It don't uses any "this" context.

bsekachev · 2018-10-08T08:54:24Z

cvat/apps/engine/static/engine/js/shapeCollection.js

@@ -801,6 +878,10 @@ class ShapeCollectionModel extends Listener {
    get shapes() {
        return this._shapes;
    }
+
+    get erased() {


unusable getter

bsekachev · 2018-10-08T08:56:38Z

cvat/apps/engine/static/engine/js/shapeCollection.js

+        for (const shape of this._shapes) {
+            let target_export_container = undefined;
+            if (!shape._removed) {
+                if (!(shape.id in this._initialShapes)) {


if (!(shape.id in this._initialShapes) || this._erased) {

bsekachev · 2018-10-08T08:58:22Z

cvat/apps/engine/static/engine/js/shapeCollection.js

+                    continue;
+                }
+            }
+            else if (shape.id in this._initialShapes) {


(shape.id in this._initialShapes && !this._erased)

nmanovic · 2018-10-08T09:00:17Z

cvat/apps/engine/static/engine/js/annotationParser.js

@@ -77,7 +79,8 @@ class AnnotationParser {

        let occluded = +shape.getAttribute('occluded');
        let z_order = shape.getAttribute('z_order') || '0';
-        return [points, occluded, +z_order];
+        let client_id = box.getAttribute('client_id') || '-1'


@azhavoro ,

Need to check somehow that client_id is valid. For example, we don't have the same client id for different boxes or all of them are -1. The check should be on client side when we parse annotation and on server side when the user presses down key.

nmanovic · 2018-10-08T09:01:53Z

cvat/apps/engine/static/engine/js/annotationUI.js


    const data = {
-        annotation: exportedData,
+        annotation: JSON.stringify(exportedData),


What is the reason to send it as string?

I think cause of issue was hash function, that used early. I've not changed request format for backward capability and suggest to change it in new MR.

nmanovic · 2018-10-08T09:10:02Z

cvat/apps/engine/static/engine/js/shapeCollection.js

+            shape_container_target = export_action_container.polyline_paths;
+        }
+
+        if (!shape_container_target) {


Does switch has 'default'? Does it make sense to move the statement into default?

nmanovic · 2018-10-08T09:34:05Z

cvat/apps/engine/static/engine/js/shapeCollection.js

+        this._erased = false;
+    }
+
+    export() {


How long does the function works on big tasks like FP/FN?

About 600ms for 90k objects

…gic; fixed comments from review

nmanovic · 2018-10-10T09:23:52Z

cvat/apps/engine/annotation.py

+        client_ids = set()
+        def extract_and_check_clinet_id(obj):
+            if 'client_id' not in box:
+                    raise Exception('No client_id field in received data')


Please fix indentation

nmanovic · 2018-10-10T09:24:42Z

cvat/apps/engine/annotation.py

+            if 'client_id' not in box:
+                    raise Exception('No client_id field in received data')
+            client_id = obj['client_id']
+            if obj['client_id'] in client_ids:


client_id in client_ids

nmanovic · 2018-10-10T09:26:58Z

cvat/apps/engine/annotation.py

+            client_ids.add(client_id)
+            return client_id
+
+        for action in ['create', 'update', 'delete']:


Let's roll the loop

nmanovic · 2018-10-10T09:27:28Z

cvat/apps/engine/annotation.py

@@ -1707,7 +2065,8 @@ def _flip_shape(shape, im_w, im_h):
                                        ("ytl", "{:.2f}".format(shape.ytl)),
                                        ("xbr", "{:.2f}".format(shape.xbr)),
                                        ("ybr", "{:.2f}".format(shape.ybr)),
-                                        ("occluded", str(int(shape.occluded)))
+                                        ("occluded", str(int(shape.occluded))),
+                                        ("client_id", "{}".format(shape.client_id)),


str(shape.client_id)

nmanovic · 2018-10-10T09:29:23Z

cvat/apps/engine/annotation.py

@@ -1726,7 +2085,8 @@ def _flip_shape(shape, im_w, im_h):
                                                "{:.2f}".format(float(p.split(',')[1]))
                                            )) for p in shape.points.split(' '))
                                        )),
-                                        ("occluded", str(int(shape.occluded)))
+                                        ("occluded", str(int(shape.occluded))),
+                                        ("client_id", "{}".format(shape.client_id)),


Please use str

nmanovic · 2018-10-10T09:29:33Z

cvat/apps/engine/annotation.py

@@ -1773,7 +2133,8 @@ def _flip_shape(shape, im_w, im_h):
                    for path in path_list:
                        dump_dict = OrderedDict([
                            ("id", str(path_idx)),
-                            ("label", path.label.name)
+                            ("label", path.label.name),
+                            ("client_id", "{}".format(path.client_id)),


Please use str

…terpolation shapes

bsekachev · 2018-10-12T15:46:50Z

@azhavoro
I found a bug in this patch. I tried next scenario:

I opened a annotation task.
I remove all annotation by click "Remove annotation" button
I created any bounding box
I saved it and got 400 error.
I saved it again and it was successful.
I updated page and I saw old annotation together with new annotation.

bsekachev · 2018-10-12T16:00:28Z

@azhavoro
Also after client parsed an annotation file without client_id field, we have situation when all objects have index -1. I think it isn't right.

bsekachev · 2018-10-12T16:17:08Z

@azhavoro
Other bug in a scenario:

I opened task (it was a interpolation task, but I think it doesn't matter)
I uploaded annotation file without client_id fields.
I saved job. It was successful.
I moved any box. (UPD: actually it isn't necessary step)
I saved it again and I got 400 error.
Each next save also returned 400 code.

bsekachev · 2018-10-12T16:28:24Z

@azhavoro
ReID doesn't work with this patch. I checked it.

The patch has critical bugs

* Bug has been fixed: impossible to lock/occlude object in AAM * Bug has been fixed: invisible points actually are visible * Bug has been fixed: impossible to close points after editing (#98) * doc: grammatical cleanup of README.md (#107) * Add info about development environment into CONTRIBUTING.md (#110) * Now we store virtual URL instead of update it in the browser address bar (#112) * Copy URL, Frame URL and object URL functionality in a context menu * Bug has been fixed: label UIs don't update after changelabel (#109) * Common escape button for exit from creating/groupping/merging/pasting/aam * Switch outside/keyframe shortkeys * Fix django vulnerability (#121) * Add analytics component (#118) * Incremental save of annotations (#120) * Create task timeout 1h -> 4h. (#136) * OpenVino integration (#134) * Update README.md (#138) * Add an extra field into meta section of a dump file (#149) * Job status was implemented (#153) * Back link to task from annotation view (#156) * Change a task with labels and attributes in admin panel (#157) * Permissions per tasks and jobs (#185) * Fix context menu, text visibility for small images (#202) * Fixed: both context menu are opened simultaneously * Fixed: shape can be unavailable behind text * Fixed: invisible text outside frame * Fix upload big xml files for tasks (#199) * Add Questions section to Readme.md (#226) * Fixed labels order (#242) * Propagate behaviour has been updated in cases with a different resolution (#246) * Updated the guide and images (#241) * Fix number attribute for float numbers. (#258)

Andrey Zhavoronkov added 6 commits September 19, 2018 15:53

improved logic of synchronization with server

73246ed

fixed 'save annotation for the task' feature

989b7cc

using shapes id generated by cleint instead db to update annotations

4210880

removed md5.js, code cleanup

26f6cda

fixed update_paths function; code cleanup;

a5aefec

fixed import existing tasks without client_id

2f26ac7

azhavoro assigned nmanovic Oct 5, 2018

azhavoro requested review from bsekachev and nmanovic October 5, 2018 13:57

bsekachev reviewed Oct 8, 2018

View reviewed changes

nmanovic reviewed Oct 8, 2018

View reviewed changes

Andrey Zhavoronkov added 3 commits October 10, 2018 10:15

added some sanity checks for cleint_id; improved annotation parser lo…

75958d0

…gic; fixed comments from review

added missed migration

9c8d185

Merge remote-tracking branch 'origin' into az/anno_synchronization

3954c0f

nmanovic reviewed Oct 10, 2018

View reviewed changes

Andrey Zhavoronkov added 3 commits October 10, 2018 17:12

fixed comments

7c2cdaa

changed logic of update shapes method

83a97a9

fixed dumping and uploading annotation if annotation task contains in…

a7c74c4

…terpolation shapes

bsekachev previously approved these changes Oct 12, 2018

View reviewed changes

preparation for reid

a5bc4d2

nmanovic approved these changes Oct 15, 2018

View reviewed changes

nmanovic merged commit 7badd24 into develop Oct 15, 2018

nmanovic deleted the az/anno_synchronization branch October 15, 2018 13:19

nmanovic mentioned this pull request Oct 15, 2018

How to keep track id? #21

Closed

yzchaogmail mentioned this pull request Jan 31, 2019

Release 0.3 (#260) yzchaogmail/cvat#3

Merged

TOsmanov pushed a commit to TOsmanov/cvat that referenced this pull request Aug 23, 2021

Add missing method (cvat-ai#120)

5cc0ca2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incremental save of annotation. #120

Incremental save of annotation. #120

azhavoro commented Oct 5, 2018

bsekachev left a comment

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018 •

edited

Loading

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018

bsekachev Oct 8, 2018

nmanovic Oct 8, 2018

nmanovic Oct 8, 2018

azhavoro Oct 10, 2018

nmanovic Oct 8, 2018

nmanovic Oct 8, 2018

azhavoro Oct 10, 2018

nmanovic Oct 10, 2018

nmanovic Oct 10, 2018

nmanovic Oct 10, 2018

nmanovic Oct 10, 2018

nmanovic Oct 10, 2018

nmanovic Oct 10, 2018

bsekachev commented Oct 12, 2018 •

edited

Loading

bsekachev commented Oct 12, 2018

bsekachev commented Oct 12, 2018 •

edited

Loading

bsekachev commented Oct 12, 2018

Incremental save of annotation. #120

Incremental save of annotation. #120

Conversation

azhavoro commented Oct 5, 2018

bsekachev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsekachev Oct 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsekachev commented Oct 12, 2018 • edited Loading

bsekachev commented Oct 12, 2018

bsekachev commented Oct 12, 2018 • edited Loading

bsekachev commented Oct 12, 2018

bsekachev Oct 8, 2018 •

edited

Loading

bsekachev commented Oct 12, 2018 •

edited

Loading

bsekachev commented Oct 12, 2018 •

edited

Loading