Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to train rokoko dataset in mmpose? #1243

Open
fredyu opened this issue Mar 15, 2022 · 4 comments
Open

Is it possible to train rokoko dataset in mmpose? #1243

fredyu opened this issue Mar 15, 2022 · 4 comments

Comments

@fredyu
Copy link

fredyu commented Mar 15, 2022

Thanks for your feature request and we will review and plan for it when necessary.
If you feel we have helped you, give us a STAR! 😆

Steps

  1. Check if the feature has been requested in the meta issue, and if so, click thumb up button.
  2. Post the feature request in the meta issue, if it is new.

Describe the feature

Motivation

A clear and concise description of the motivation of the feature.

  1. Ex1. It is inconvenient when [....].
  2. Ex2. There is a recent paper [....], which is very helpful for [....].

Related resources

If there is an official code released or third-party implementations, please also provide the information here, which would be very helpful.

Additional context

Add any other context or screenshots about the feature request here.
If you would like to implement the feature and create a PR, please leave a comment here and that would be much appreciated.

@jin-s13
Copy link
Collaborator

jin-s13 commented Mar 16, 2022

Could you please provide more details about rokoko dataset?

@fredyu
Copy link
Author

fredyu commented Mar 16, 2022

Thanks Jin-s13 for the quick response. The ROKOKO dataset is saved in a csv file, format as following:

Timestamp Root_P_x Root_P_y Root_P_z Root_Q_x Root_Q_y Root_Q_z Root_Q_w Hips_P_x Hips_P_y Hips_P_z Hips_Q_x Hips_Q_y Hips_Q_z. .............
0 0 0 0 6.657903e-08 0 0 1 -0.007210765 0.909949 -0.002047792 0.0149896 0.007403604 0.001145249
0 0 0 0 6.657903e-08 0 0 1 -0.007210765 0.909949 -0.002047792 0.0149896 0.007403604 0.001145249
0 0 0 0 6.657903e-08 0 0 1 -0.007210765 0.909949 -0.002047792 0.0149896 0.007403604 0.001145249
10.00977 0 0 0 6.657903e-08 0 0 1 -0.007251054 0.9095271 -0.002021573 0.01500696 0.007402699 0.001134259
20.01953 0 0 0 6.657903e-08 0 0 1 -0.007284701 0.909778 -0.001934813 0.01497277 0.007376918 0.001162205
30.0293 0 0 0 6.657903e-08 0 0 1 -0.007382753 0.9097246 -0.001832062 0.0150325 0.007342283 0.001128938
39.79492 0 0 0 6.657903e-08 0 0 1 -0.007468735 0.9097522 -0.001758403 0.01517074 0.007407371 0.001001961
49.80469 0 0 0 6.657903e-08 0 0 1 -0.007545079 0.9097492 -0.001727905 0.01532079 0.007468593 0.0009430593

Each line includes 183 values:
Timestamp | Root_P_x | Root_P_y | Root_P_z | Root_Q_x | Root_Q_y | Root_Q_z | Root_Q_w | Hips_P_x | Hips_P_y | Hips_P_z | Hips_Q_x | Hips_Q_y | Hips_Q_z | Hips_Q_w | LeftThigh_P_x | LeftThigh_P_y | LeftThigh_P_z | LeftThigh_Q_x | LeftThigh_Q_y | LeftThigh_Q_z | LeftThigh_Q_w | LeftShin_P_x | LeftShin_P_y | LeftShin_P_z | LeftShin_Q_x | LeftShin_Q_y | LeftShin_Q_z | LeftShin_Q_w | LeftFoot_P_x | LeftFoot_P_y | LeftFoot_P_z | LeftFoot_Q_x | LeftFoot_Q_y | LeftFoot_Q_z | LeftFoot_Q_w | LeftToe_P_x | LeftToe_P_y | LeftToe_P_z | LeftToe_Q_x | LeftToe_Q_y | LeftToe_Q_z | LeftToe_Q_w | LeftToeTip_P_x | LeftToeTip_P_y | LeftToeTip_P_z | LeftToeTip_Q_x | LeftToeTip_Q_y | LeftToeTip_Q_z | LeftToeTip_Q_w | RightThigh_P_x | RightThigh_P_y | RightThigh_P_z | RightThigh_Q_x | RightThigh_Q_y | RightThigh_Q_z | RightThigh_Q_w | RightShin_P_x | RightShin_P_y | RightShin_P_z | RightShin_Q_x | RightShin_Q_y | RightShin_Q_z | RightShin_Q_w | RightFoot_P_x | RightFoot_P_y | RightFoot_P_z | RightFoot_Q_x | RightFoot_Q_y | RightFoot_Q_z | RightFoot_Q_w | RightToe_P_x | RightToe_P_y | RightToe_P_z | RightToe_Q_x | RightToe_Q_y | RightToe_Q_z | RightToe_Q_w | RightToeTip_P_x | RightToeTip_P_y | RightToeTip_P_z | RightToeTip_Q_x | RightToeTip_Q_y | RightToeTip_Q_z | RightToeTip_Q_w | Spine1_P_x | Spine1_P_y | Spine1_P_z | Spine1_Q_x | Spine1_Q_y | Spine1_Q_z | Spine1_Q_w | Spine2_P_x | Spine2_P_y | Spine2_P_z | Spine2_Q_x | Spine2_Q_y | Spine2_Q_z | Spine2_Q_w | Spine3_P_x | Spine3_P_y | Spine3_P_z | Spine3_Q_x | Spine3_Q_y | Spine3_Q_z | Spine3_Q_w | Spine4_P_x | Spine4_P_y | Spine4_P_z | Spine4_Q_x | Spine4_Q_y | Spine4_Q_z | Spine4_Q_w | LeftShoulder_P_x | LeftShoulder_P_y | LeftShoulder_P_z | LeftShoulder_Q_x | LeftShoulder_Q_y | LeftShoulder_Q_z | LeftShoulder_Q_w | LeftArm_P_x | LeftArm_P_y | LeftArm_P_z | LeftArm_Q_x | LeftArm_Q_y | LeftArm_Q_z | LeftArm_Q_w | LeftForeArm_P_x | LeftForeArm_P_y | LeftForeArm_P_z | LeftForeArm_Q_x | LeftForeArm_Q_y | LeftForeArm_Q_z | LeftForeArm_Q_w | LeftHand_P_x | LeftHand_P_y | LeftHand_P_z | LeftHand_Q_x | LeftHand_Q_y | LeftHand_Q_z | LeftHand_Q_w | Neck_P_x | Neck_P_y | Neck_P_z | Neck_Q_x | Neck_Q_y | Neck_Q_z | Neck_Q_w | Head_P_x | Head_P_y | Head_P_z | Head_Q_x | Head_Q_y | Head_Q_z | Head_Q_w | RightShoulder_P_x | RightShoulder_P_y | RightShoulder_P_z | RightShoulder_Q_x | RightShoulder_Q_y | RightShoulder_Q_z | RightShoulder_Q_w | RightArm_P_x | RightArm_P_y | RightArm_P_z | RightArm_Q_x | RightArm_Q_y | RightArm_Q_z | RightArm_Q_w | RightForeArm_P_x | RightForeArm_P_y | RightForeArm_P_z | RightForeArm_Q_x | RightForeArm_Q_y | RightForeArm_Q_z | RightForeArm_Q_w | RightHand_P_x | RightHand_P_y | RightHand_P_z | RightHand_Q_x | RightHand_Q_y | RightHand_Q_z | RightHand_Q_w

Based on this ROKOKO file, we have rendered out PNG file from Maya for each line (timestamp):

V33-1F_1 0009
V33-1F_1 0011
V33-1F_1 0008

@fredyu
Copy link
Author

fredyu commented Mar 16, 2022

and we can visualize it like:
image

The 3d to 2d projection is not exactly matched since we don't the camera parameters when rendering on Maya.

@ly015
Copy link
Member

ly015 commented Apr 14, 2022

Thanks for using MMPose. Please refer to our tutorial about how to add a new dataset in MMpose. This dataset seems to include 2D/3D keypoints, rendered synthetic images, and camera parameters. You can refer to the implementations of COCO dataset the Human3.6M dataset for 2D keypoint detection and 3D keypoint detection respectively.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants