Automatic annotation YOLO error

[whiteocean016]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

Expected Behaviour

Using Yolo-v3 AI tool should give the detected bboxes as annotations in CVAT.

Current Behaviour

After clicking "Annotate" I get this error in the CVAT UI:

Detection error occured
Error: "state label" is expected to be Label, but "undefined" has been got.

Possible Solution

Possibly CVAT error in handling the response from nuclio?

Steps to Reproduce (for bugs)

1. Open a video labeling job in Chrome e.g. http://localhost:8080/tasks/1/jobs/1
2. Click "AI Tools" -> Detectors -> set "person" and "cell phone" labels mapping
3. Click "Annotate"

Context

I am trying to use Yolo-v3 via nuclio for automatic annotation of videos. I followed the instructions for setting up
automatic annotation. Confirmed that the Yolo nuclio function is working by using advice from this issue: #2641 (comment)
In addition to this the automatic annotation for the whole video (job) also doesn't work. It shows the progress bar and finishes
without errors, but then there are no annotations made.

Your Environment

• using CVAT v1.2.0
• Docker version 20.10.2
• docker-compose version 1.24.1
• Ubuntu 18.04.05

[AlexandrMoruk]

I have same problem for all automatic segmentation models

[bsekachev]

Hi @whiteocean016

I was not able to reproduce the issue on both my local setup & cvat.org. Could you please provide more details? Task labels, images, nuclio response, etc. More details -> more chances to reproduce and fix.

[whiteocean016]

Hi
Thanks for trying it out. Here is some more information.
If some other info is needed I will provide it.

• labels constructor I use for the project:

  {
    "name": "phone",
    "id": 1,
    "color": "#fa3253",
    "attributes": []
  },
  {
    "name": "person",
    "id": 2,
    "color": "#33ddff",
    "attributes": []
  }
]

• screenshot of error reproduction
  
  
• docker logs of some containers when using the AI tool (wasn't sure which ones are important)
  • nuclio-nuclio-openvino-omz-public-yolo-v3-tf container

    21.02.02 16:01:06.815 sor.http.w1.python.logger (I) Run yolo-v3-tf model {"worker_id": "1"}
    /opt/nuclio/model_handler.py:129: DeprecationWarning: shape property of IENetLayer is deprecated. Please use shape property of DataPtr instead objects returned by in_data or out_data property to access shape of input or output data on corresponding ports
    out_blob = out_blob.reshape(self.model.layers[self.model.layers[layer_name].parents[0]].shape)
    /opt/nuclio/model_handler.py:129: DeprecationWarning: shape property of IENetLayer is deprecated. Please use shape property of DataPtr instead objects returned by in_data or out_data property to access shape of input or output data on corresponding ports
    out_blob = out_blob.reshape(self.model.layers[self.model.layers[layer_name].parents[0]].shape)
    /opt/nuclio/model_handler.py:129: DeprecationWarning: shape property of IENetLayer is deprecated. Please use shape property of DataPtr instead objects returned by in_data or out_data property to access shape of input or output data on corresponding ports
    out_blob = out_blob.reshape(self.model.layers[self.model.layers[layer_name].parents[0]].shape)
    

  • nuclio container: here (created with docker logs nuclio --tail 1 -f > nuclio.log because it spits out a lot)
  • cvat container

    2021-02-02 16:28:26,096 DEBG 'rqworker_low' stderr output:
    DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
    
    2021-02-02 16:28:26,398 DEBG 'rqworker_default_1' stderr output:
    DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
    

• nuclio response to the image seen above with cat /tmp/input.json | nuctl invoke openvino-omz-public-yolo-v3-tf -c 'application/json'

21.02.02 16:53:05.599    nuctl.platform.invoker (I) Executing function {"method": "POST", "url": "http://:49154", "headers": {"Content-Type":["application/json"],"X-Nuclio-Log-Level":["info"],"X-Nuclio-Target":["openvino-omz-public-yolo-v3-tf"]}}
21.02.02 16:53:05.950    nuctl.platform.invoker (I) Got response {"status": "200 OK"}
21.02.02 16:53:05.950                     nuctl (I) >>> Start of function logs
21.02.02 16:53:05.950 ino-omz-public-yolo-v3-tf (I) Run yolo-v3-tf model {"time": 1612281185609.2192, "worker_id": "0"}
21.02.02 16:53:05.950                     nuctl (I) <<< End of function logs

> Response headers:
Server = nuclio
Date = Tue, 02 Feb 2021 15:53:05 GMT
Content-Type = application/json
Content-Length = 1128

> Response body:
[
    {
        "confidence": "0.9896554",
        "label": "person",
        "points": [
            710,
            695,
            779,
            894
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.9857917",
        "label": "suitcase",
        "points": [
            1156,
            662,
            1440,
            1044
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.9857152",
        "label": "person",
        "points": [
            964,
            323,
            1216,
            1034
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.9848279",
        "label": "person",
        "points": [
            457,
            635,
            527,
            884
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.92583245",
        "label": "backpack",
        "points": [
            1017,
            373,
            1227,
            662
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.8154599",
        "label": "person",
        "points": [
            102,
            760,
            152,
            880
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.72792083",
        "label": "person",
        "points": [
            799,
            699,
            847,
            897
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.6802303",
        "label": "tvmonitor",
        "points": [
            1421,
            576,
            1480,
            619
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.67955214",
        "label": "person",
        "points": [
            783,
            684,
            823,
            879
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.61954117",
        "label": "person",
        "points": [
            219,
            750,
            271,
            888
        ],
        "type": "rectangle"
    },
    {
        "confidence": "0.5773699",
        "label": "person",
        "points": [
            2,
            725,
            38,
            910
        ],
        "type": "rectangle"
    }
]

[AlexandrMoruk]

All models working in Firefox, but chrome is not

[whiteocean016]

I get the same error in Firefox and Chrome.

[bsekachev]

Hi, @whiteocean016

Sorry for delay.
From data you provided I get back to CVAT 1.2.0, created task with these labels:

[ {
    "name": "phone",
    "color": "#fa3253",
    "attributes": []
  },
  {
    "name": "person",
    "color": "#33ddff",
    "attributes": []
  }
]

and your image, but I still was not able to reproduce the bug.

As you can see on the screenshot, from server I get only "person" labels, no any "suitcase" (because actually these labels are expected to be filtered by server since they are not in the task). I do get them from the serverless function BTW, what is expected.

The error: Error: "state label" is expected to be Label, but "undefined" has been got. makes me suspicions that on the client you get objects with labels which the task does not contain. So, could you please provide also response data which you get from the server?

Also I would be interested in request payload. All these data can be found in chrome dev tools on Network tab. For example in my case I see followings:

[whiteocean016]

Hi @bsekachev

Np, thanks a lot for helping.
It turns out I get the same payload and response as you. Here is the payload:

And here is the response:

[bsekachev]

@whiteocean016

Hmm.. not exactly. I see request payload is the same, but the response is different.
In my case I have only persons, you also have suitcase, tvmonitor, backpack.
And it is a reason why UI fails, because it doesn't expect these labels to appear at all.

Let me know one more thing, is the task created within a project?

[whiteocean016]

Aha you are right about the labels, I totally missed that.

Yes the task was created within a project.

[bsekachev]

@whiteocean016

So, finally I was able to reproduce the issue and glad to say that it has been already fixed in #2725.
Please, update CVAT from develop branch and thanks for your support.

Also one more PR has been recently submitted #2780 to fix quite serious usability issue with AI Tools #2753 and I would recommend you to wait it to be merged (I hope we will do it tomorrow).

@AlexandrMoruk FYI

I'll close the issue, but if you still have questions, please ask.

[sajadalipour]

I solved this problem.
For creating a task, you must define labels exclusively for that task and the task must not belong to any project.