Add a "Skeleton Tool" for human pose labeling

[chunter18]

My actions before raising this issue

• [*] Read/searched the docs
• [*] Searched past issues

Context/summary

Human pose labeling is relatively common task in computer vision - while I have been using CVAT for a number of labeling tasks, pose labeling is a pain point. From my perspective the best way currently to do pose labeling currently is with the points tool, but to label each point with an informative tag like "shoulder" or "knee" makes tagging very slow, and just tagging each point with a generic label means that there needs to be a refinement step on the generated tags which isn't ideal.

I think a nice feature would be to implement a "Skeleton Tool", similar to what can be seen here: https://youtu.be/q17gqr0EIUQ?t=103.

In short, a user can create a bounding box that comes pre-populated with a number of points which can be moved after placing. The output format would include the coordinates of the box as well as coordinates of each point in the skeleton. Users can create an select from a number of skeletons.

I think CVAT is an outstanding tool, and I would prefer to see features like this added than to use another system. Thanks to the team for all the great work.

Related Issues

#494
#1421

[zhiltsov-max]

Related #1476

[alainrivard]

Hello, I need a free annotation tool with skeleton support to integrate with one project and this is the feature left out in all free version. So I took a look at integrating it in CVAT (I don't need many feature since it is a closed annotation project and only export to COCO). Now it is a early prototype, but I need to move on other tasks for a while, so I'm sharing the code as a starting point for the discussions on this issue.

You can see a demo at this link (downloading the file provides a much better video quality): https://1drv.ms/v/s!All8gCWBqdOlgRCq-6UgTjdBWR0Y?e=6QfU7x

Here is the high level summary of the patch provided:

• Adding a skeleton was braking the Any-Tool to Any-Label pattern, since for a skeleton the shape information is embedded in the label itself
  • So, skeleton tool only show label with skeleton information
  • So, skeleton object cannot change label after instantiated
• In the label editor, we now have to possibility to add attributes and one skeleton object
  • A skeleton object has a One-to-One relationship with a label in the project
  • A skeleton editor will be need, but in the meantime we can add the skeleton information in the raw editor
  • A simple skeleton format has be defined (see below)
  • Attributes now have a category ('manual', 'auto', 'hidden')
  • A new hidden attribute named __internal_shape_orientation is added to every label
• A SkeletonSpecs model had to be defined
  • Serialization of the skeleton has been added at most place needed (need to be confirmed)
• The canvas now support a new skeleton shape
  • A skeleton shape is drawn as a box with opacity 0
  • A skeleton is using the resize event to remap the skeleton inside the box at draw time
  • A skeleton is using the drawstop event to remap the bonding box to skeleton points
  • A skeleton shape don't use shapeSizeElement, it uses shapeInfoElement to handle (Front, Back, Up, Down)
  • getFinalRectCoordinates now support an optional cursorPosition argument to handle first point second point drawn
  • A skeleton update its internal hidden attribute add onDrawDone time
  • A skeleton is added as a polyline with opacity 0
  • A skeleton is using the active object resizing event to the handle links defined by the template

I'm new to the CVAT code base, React, Redux and SVG, so there may be misconception in the code from the patch.
What I think is missing to do: complete the specifications, finalize the development, validation. test cases, skeleton editor and datasets export support. Hope this can help you.

The person 15 points skeleton template is defined as this, all coordinates are relative, they are normalized at the remap. This is what you need to paste in the template array in the raw editor:
{"name": "Head", "point": [50, 99], "links": ["Neck"]},
{"name": "Neck", "point": [50, 87], "links": ["R_shoulder", "L_shoulder"]},
{"name": "R_shoulder", "point": [28, 85], "links": ["R_elbow"]},
{"name": "L_shoulder", "point": [72, 85], "links": ["L_elbow"]},
{"name": "R_elbow", "point": [23, 72], "links": ["R_hand"]},
{"name": "L_elbow", "point": [77, 72], "links": ["L_hand"]},
{"name": "R_hand", "point": [23, 50], "links": []},
{"name": "L_hand", "point": [77, 50], "links": []},
{"name": "Back", "point": [50, 50], "links": ["Neck"]},
{"name": "R_hip", "point": [35, 48], "links": ["Back"]},
{"name": "L_hip", "point": [65, 48], "links": ["Back"]},
{"name": "R_knee", "point": [33, 25], "links": ["R_hip"]},
{"name": "L_knee", "point": [67, 25], "links": ["L_hip"]},
{"name": "R_feet", "point": [33, 1], "links": ["R_knee"]},
{"name": "L_feet", "point": [67, 1], "links": ["L_knee"]}

To try it, but you cannot export dataset, just save annotated projects (skeleton_demo.zip):
git clone --depth 1 --branch v1.7.0 https://github.com/openvinotoolkit/cvat
cd cvat && mkdir logs keys
python3 -m venv .env
. .env/bin/activate
pip install -U pip wheel setuptools
pip install -r cvat/requirements/development.txt
git switch -c experimental
git apply skeleton_demo.patch

python manage.py migrate
python manage.py collectstatic
python manage.py createsuperuser
code
npm ci && npm run start:cvat-ui

Go in code and start: server: debug

[nmanovic]

@alainrivard , thanks for sharing your patch. It will be great to work together and merge the feature into develop branch in the future.

[csfankun]

hi,when will this skeleton tool been published?
Is there any approximate time?
I think everyone look forward to this magnificent function..

[nmanovic]

@csfankun , it is not in our roadmap yet. If nobody from community helps, it will not be in H1'2022. In general it is possible to annotate human pose, it is just inconvenient and slow. Vote for the issue. If we see that it is the most important feature for CVAT community, we will take it and implement for future releases.

[guillaumevr]

@alainrivard thank you for this amazing patch
@nmanovic can you tell me if it's possible to draw the different parts of a polyline in different colors? The objective is to differentiate the right and left parts of the skeleton. Thx

[ghost]

We definitly need this feature asap. Thanks for the input @alainrivard

[bsekachev]

The feature has been implemented here #1

It also was merged to current repository, you need to update CVAT from develop branch if you want to use it.
It is initial implementation and probably will be improved in the future, so, we are glad to get any feedbacks.

Thank you.