[EPIC] Motion Capture

[mrhegemon]

Top Priorities

• New UX
• Fix face and hands
• Video Segment to VRM
• QA

Overall goal is to fuse our pose Skeletal IK from many sources such as multiple camera inputs and or pose capture and clients into a full body model.

Avatar System

• Add Full Body IK MVP with a simple squat mechanic #7486
• [Epic] VRM - Rebuild Avatar Source Rig #7592
• Rewrite animation state machine to be way simpler & data oriented
• Additive FK/IK blending instead of slerp
• All motion capture and IK solvers operate in world space rather than local space #8790
• Avatar scaling for different size bodies #7519
• Add arm twist bones to retargeting source skeleton #6700
• Networked WebXR Finger Tracking #5650
• [Story] Expanded bone matching #8503
• Regression testing for different avatars for bone matching, retargeting & IK
• [Epic] WebGPU supports vertex animation textures #7943
• Extensible avatar animation sets API #8502
• Show hand models when avatar is hidden in WebXR #7590
• Pill colliders moving with each bone

Motion Capture System V1

• Mediapipe pose tracking
• hip position assessment
• hip rotation assessment
• Add Sitting Animation and State #9133
• Seated mocap #9134
• landmark noise reduction using kalman or other filters https://github.com/freemocap/freemocap/tree/main
• landmark return to a rest pose or have some strategy for when data is not present
• ground assessment - may need a fancy strategy
• Motion Capture Calibration UI #9045
• Motion Capture - Add promptings to UI to ensure user is properly framed in video #8563
• Mediapipe face tracking
• Mediapipe hand tracking #9047
• Allow users to change their video feed quality and crop
• Crop pose video and pass to hands and face capture models

Motion Capture System V2

• Build a pretty CV Motion Capture Control UX
• Multi point / Multi camera client CV Fusion! #9048
• Add WebGPU backend to TensorFlow.js and MediaPipe #7983

Human Assessment

• foot placement
• torso mode (seated position, wheelchairs)
• height assessment of avatar - may need various strategies such as xr inputs
• wingspan assessment and fine grained body fitting
• Face Tracking Visimes #7495
• Build the Skeletal IK Settings UX

MoCap Recording & Playback

• Serialize (Record) the Skeletal IK
• Deserialize (Play back) the Skeletal IK
• Build Client Record / Playback UX
• Build Editor Record / Playback UX
• Build the Admin UX

Later nice features

• Custom camera position, FOV and Frustum, intrinsics and so on especially for multiple cameras
• AR Magic Window [EPIC]:Advanced camera. perspective AR w/ facial tracking / Magic Window AR / Projection Mapping. #5303
• MR Capture
• Add WebGPU backend to TensorFlow.js and MediaPipe #7983

NOTES

CV Tracking OPTION 1 : MediaPipe

Human js
https://github.com/vladmandic/human
https://google.github.io/mediapipe/getting_started/javascript.html

HOLISTIC BODY CAPTURE - WEB DEMO
https://gist.github.com/mrhegemon/2832793cae560c6980d2ed40048cf1e4

SIMPLE POSE CAPTURE - WEB DEMO
https://gist.github.com/mrhegemon/7931bf2cf8be283f3aef0b2bb7a5cbe4

CV Tracking OPTION 2 : Others

Light Tensorflow
https://codesandbox.io/s/9rhlj

OpenPose
https://cmu-perceptual-computing-lab.github.io/openpose/web/html/doc/index.html

AR Magic Window

https://andykong.org/projects/eyecatcher/
https://karan-ahuja.com/fun_projects/eyecatcher/

MediaPipe + Three js

• https://github.com/mrhegemon/RiggingJs
• https://github.com/spite/FaceMeshFaceGeometry
• https://techtee.medium.com/real-time-face-mesh-point-cloud-with-three-js-tensorflow-js-and-typescript-1f37ae844e1f
• https://orth.uk/club2d/
• https://handpose-demos.glitch.me/
• https://github.com/orgs/jeeliz/repositories
• Gesture Detection https://juejin.cn/post/7051508361015066654
• https://blog.tensorflow.org/2021/11/3D-handpose.html
• https://note.com/tech_hikky/n/n1345d2ca0539
• https://raspberrypi4u.blogspot.com/2020/04/raspberry-pi-tensorflowjs-facemesh.html
• https://github.com/hiukim/mind-ar-js
• https://glitch.com/~networked-hand-3js-tf174-handv1
• https://selvamsubbiah.com/mediapipe-facemesh-triangles-in-tensorflow-js-part-2/
• eye tracking https://webgazer.cs.brown.edu/
• https://www.ar-watches.ru/

Mediapipe + Unity

https://github.com/Kariaro/VRigUnity

BlazePoseWithUnity
https://github.com/joonb14/BlazePoseWithUnity
https://github.com/digital-standard/ThreeDPoseUnityBarracuda/tree/master
https://github.com/digital-standard/ThreeDPoseUnityBarracuda/blob/master/Assets/Scripts/VNectModel.cs

Final IK Unity

http://root-motion.com/
https://assetstore.unity.com/packages/tools/animation/final-ik-14290
https://www.youtube.com/user/SnuffThePuppy/playlists

Web IK

Osso

https://twitter.com/sketchpunklabs
https://www.patreon.com/posts/63305737
osso Tests
https://www.youtube.com/@SketchpunkLabs/videos

OTHER EXAMPLES

https://www.kdnuggets.com/2020/08/3d-human-pose-estimation-experiments-analysis.html

8th wall
https://www.8thwall.com/8thwall/face-effects-threejs/code/README.md

https://github.com/imerso/facecap

DIRECT POSENET TO VR
https://gitlab.com/posenet-vr

Another - TDPToVR
https://nponta-tanuki.booth.pm/items/2492481

https://github.com/TrackLab/ViRe

Real-time pose estimation vr tracker by web camera in VRChat [2021-0102]
https://www.youtube.com/watch?v=cf5moO_1TWY

ThreeDPoseTracker
https://mobile.twitter.com/search?q=%23ThreeDPoseTracker
https://store.steampowered.com/app/1366950/Driver4VR/
https://www.youtube.com/channel/UCVEw7o4VIuX3ZvWoQx_n3Fg
https://www.youtube.com/watch?v=5at0Gw_mEmc

Kaldo
https://npm.io/package/kalidokit
https://kalidoface.com/
https://kalidoface.com/

VRChat UX Flow
https://docs.vrchat.com/docs/full-body-tracking

VRChat OSC
https://docs.vrchat.com/docs/full-body-tracking
https://docs.vrchat.com/docs/osc-avatar-parameters
https://docs.vrchat.com/docs/osc-resources
https://docs.vrchat.com/docs/osc-debugging

Rokoko + Move.ai

Potential hacks for improvement:

• explore video vs image
• crop to the frame of the most confident body
• workerize a buffer of frames that can be assigned to threaded workers
• explore offscreen-canvas as an alternate execution space for more multithreading
• see what happens if we disable video rendering, and only show the skeleton (fps improvement?)
• see if PWA or fullscreen makes any difference (increased browser priority?)

[HexaField]

Boneworks / Bonelab

• Jumping mechanism is great
  Pop One
• Gliding w/ arms
  Walkabout
• Flying by pointing up and teleport button
  Echo Arena
• Zero G flying
  VRChat

[HexaField]

Animation Sequencer #7505