Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support single depth to 3D hand keypoints baseline, NYU Hand Pose Dataset and AWR #1487

Open
wants to merge 1 commit into
base: dev-0.29
Choose a base branch
from

Conversation

walsvid
Copy link
Contributor

@walsvid walsvid commented Jul 14, 2022

Motivation

Integrate single depth to 3D hand keypoints into mmpose.

Modification

Add an algorithm for estimating 3D gestures from a single depth image to mmpose
Includes the following changes:

  • Add 3d_kpt_sview_depth_img common dataset NYU Hand Pose dataset
  • Add 3d_kpt_sview_depth_img basic baseline algorithm AWR (AAAI2020) network architecture
  • Add data transformation pipeline for depth image data pre-processing

🚧 The evaluation code is still WIP, code review are required for above part.
🚨 More unit test are needed. Maybe it could be discussed during the code review stage.

BC-breaking (Optional)

  • Modified TopDownGenerateTargetRegression to adapt the calculation method of the GT target coordinates of the algorithm that uses the depth image as input

Use cases (Optional)

Checklist

Before PR:

  • I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
  • Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
  • Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
  • New functionalities are covered by complete unit tests. If not, please add more unit tests to ensure correctness.
  • The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

  • CLA has been signed and all committers have signed the CLA in this PR.

@CLAassistant
Copy link

CLAassistant commented Jul 14, 2022

CLA assistant check
All committers have signed the CLA.

@jin-s13 jin-s13 requested a review from ly015 July 14, 2022 13:02
@jin-s13
Copy link
Collaborator

jin-s13 commented Jul 14, 2022

Thanks a lot for your contribution! We will review the PR ASAP.
@walsvid Please sign the CLA so that we can proceed the code merging.

@walsvid
Copy link
Contributor Author

walsvid commented Jul 18, 2022

Hi, @jin-s13 . I have signed the CLA. I will continuously submit some commits in this branch recently to improve the PR.

@walsvid
Copy link
Contributor Author

walsvid commented Sep 2, 2022

Code will be significantly refactored to fit v1.x, so it will take some extra time.

@ly015
Copy link
Member

ly015 commented Sep 2, 2022

Code will be significantly refactored to fit v1.x, so it will take some extra time.

Thank you very much. We will maintain both master and 1.x branches until 2023, and still release v0.x versions with new features in this period. So it would be great to create a new pull request for the dev-1.x branch to add this feature to 1.x version. And this PR will still be merged into the master branch after review.

@jin-s13
Copy link
Collaborator

jin-s13 commented Jan 4, 2023

Hi @walsvid, hope you're well. Thank you again for the contribution!

Is this PR ready for review? Or shall we wait for your further updates/commits before code review?

BTW, have you already trained a few models on NYU dataset? Would you like to share the checkpoints and training logs with us?

@walsvid
Copy link
Contributor Author

walsvid commented Jan 5, 2023

Hi, I think this PR still need updates for code review. As for NYU dataset, I have checkpoints of AWR model, I can provide link. But the log is messy, this may not help. I'll update thie PR ASAP.

@jin-s13
Copy link
Collaborator

jin-s13 commented Jan 5, 2023

Hi, I think this PR still need updates for code review. As for NYU dataset, I have checkpoints of AWR model, I can provide link. But the log is messy, this may not help. I'll update thie PR ASAP.

Well understood. Please let us know if you have met any problems/questions. Thanks.

@Enderfga
Copy link

Enderfga commented Jan 5, 2023

Hi @walsvid ,I'm doing some nyu hand dataset and awr related research and would like to use mmpose, would you be so kind as to provide the code and checkpoints, thank you very much!

@walsvid
Copy link
Contributor Author

walsvid commented Jan 10, 2023

Hi @walsvid ,I'm doing some nyu hand dataset and awr related research and would like to use mmpose, would you be so kind as to provide the code and checkpoints, thank you very much!

Hi, @Enderfga @jin-s13. I upload the ckpt and nyu data meets the coco format annotations to Google Drive. The current Pull Request still needs to improve the code related to evaluation. It would be better if @Enderfga can help to improve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants