PuzzleAvatar:
Assembly of Avatar from Unconstrained Photo Collections

Yuliang Xiu · Yufei Ye · Zhen Liu · Dimitris Tzionas · Michael J. Black

SIGGRAPH Asia 2024 (Journal Track)

PuzzleAvatar reconstructs a textured 3D clothed human from unconstrained photo collections.

Works On	Without Requiring or Being Limited by
✅ Any number of photos	❌ Human body pose (3D HPS, 2D Keypoints, etc.)
✅ In any poses	❌ Camera pose
✅ From any views	❌ Geometric cues (depth, normal, etc.)
✅ With any cropping or occlusion	❌ Re-projection losses
✅ Any A-posed body mesh template	❌ SMPL-(X/H), GHUM, Frank, Adam, SCAPE, etc

News 🚩

• [2024/11/09] PuzzleIOI dataset and benchmark code get released.
• [2024/09/10] PuzzleAvatar code gets released.
• [2024/05/23] PuzzleAvatar paper gets released.

Getting Started

1. Set up the paths in scripts/env.sh.
2. Please follow the Installation Instruction to setup all the required packages.
3. Run PuzzleAvatar (Grounded-SAM $\rightarrow$ PuzzleBooth $\rightarrow$ SDS, takes about 4 hours)

# For custom data
bash scripts/run.sh data/human/yuliang results/human/yuliang human_yuliang

# For PuzzleIOI
bash scripts/run.sh data/PuzzleIOI/puzzle_capture/03632/outfit13 results/PuzzleIOI/puzzle_capture/03632/outfit13 03632_outfit13

The results will be saved in the experiment folder results/human/yuliang, and results/PuzzleIOI/puzzle_capture/03632/outfit13.

Dataset and Benchmark

1. Register at puzzleavatar.is.tue.mpg.de
2. Download datasets (194GB) with registered username and password

bash scripts/fetch_data.sh

3. For evaluation / benchmark

# render the reconstruction results (4 views)
# If the rendering process is stuck, please refer to the changes for PRT computation:
# https://github.com/YuliangXiu/ICON/pull/237/files

python -m render.render_batch_result -headless -out_dir ./results/ -split test

# calculate both 3D metrics (Chamfer, P2S, NC) and 2D metrics (PSNR, SSIM, LPIPS)
python -m multi_concepts.benchmark -split test

Acknowledgment

This implementation is built based on TeCH, Break-A-Scene, Grounded SAM, GPT-4V(ision), Stable Diffusion 2-1-base, BOFT-DreamBooth, Stable Dreamfusion, ECON.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 (CLIPE Project)

Kudos to all of our amazing contributors! PuzzleAvatar thrives through open-source. In that spirit, we welcome all kinds of contributions from the community.

Contributor avatars are randomly shuffled.

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

Disclosure

MJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon and Meshcapade GmbH. While MJB is a co-founder and Chief Scientist at Meshcapade, his research in this project was performed solely at, and funded solely by, the Max Planck Society.

Contact

For technical questions, please contact [email protected]

For commercial licensing, please contact [email protected]

Citation

@article{xiu2024puzzleavatar,
  title={PuzzleAvatar: Assembling 3D Avatars from Personal Albums},
  author={Xiu, Yuliang and Ye, Yufei and Liu, Zhen and Tzionas, Dimitrios and Black, Michael J},
  journal={ACM Transactions on Graphics (TOG)},
  year={2024},
  publisher={ACM New York, NY, USA}
}

PuzzleAvatar is mainly built upon TeCH, please also kindly cite it

@inproceedings{huang2024tech,
  title={{TeCH: Text-guided Reconstruction of Lifelike Clothed Humans}},
  author={Huang, Yangyi and Yi, Hongwei and Xiu, Yuliang and Liao, Tingting and Tang, Jiaxiang and Cai, Deng and Thies, Justus},
  booktitle={International Conference on 3D Vision (3DV)},
  year={2024}
}