[Project Page] [Paper]
Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the POSA data, model and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.
This repository contains the training, random sampling, and scene population code used for the experiments in POSA.
To install the necessary dependencies run the following command:
pip install -r requirements.txtThe code has been tested with Python 3.7, CUDA 10.0, CuDNN 7.5 and PyTorch 1.7 on Ubuntu 20.04.
To be able to use the code you need to get the POSA_dir.zip. After unzipping, you should have a directory with the following structure:
POSA_dir
├── cam2world
├── data
├── mesh_ds
├── scenes
├── sdf
└── trained_modelsThe content of each folder is explained below:
trained_modelscontains two trained models. One is trained on thecontactonly and the other one is trained oncontactandsemantics.datacontains the train and test data extracted from the PROX Dataset and PROX-E Dataset.scenescontains the 12 scenes from PROX Datasetsdfcontains the signed distance field for the scenes in the previous folder.mesh_dscontains mesh downsampling and upsampling related files similar to the ones in COMA.
You need to get the SMPLx Body Model. Please extract the folder and rename it
to smplx_models and place it in the POSA_dir above.
In addition, you need to get the POSA_rp_poses.zip file from AGORA Dataset and extract in the POSA_dir.
This file contrains a number of test poses to be used in the next steps. Note that you don't need the whole AGORA dataset.
Finally run the following command or add it to your ~/.bashrc
export POSA_dir=Path of Your POSA_dirYou can test POSA using the trained models provided. Below we provide examples of how to generate POSA features and how to pupulate a 3D scene.
To generate random features from a trained model, run the following command
python src/gen_rand_samples.py --config cfg_files/contact.yaml --checkpoint_path $POSA_dir/trained_models/contact.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --render 1 --viz 1 --num_rand_samples 3 Or
python src/gen_rand_samples.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --render 1 --viz 1 --num_rand_samples 3 This will open a window showing the generated features for the specified pkl file. It also render the features to the folder random_samples in POSA_dir.
The number of generated feature maps can be controlled by the flag num_rand_samples.
If you don't have a screen, you can turn off the visualization --viz 0.
If you don't have CUDA installed then you can add this flag --use_cuda 0. This applies to all commands in this repository.
You can also run the same command on the whole folder of test poses
python src/gen_rand_samples.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses --render 1 --viz 1 --num_rand_samples 3 Given a body mesh from the AGORA Dataset, POSA automatically places the body mesh in 3D scene.
python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --scene_name MPH16 --render 1 --viz 1 This will open a window showing the placed body in the scene. It also render the placements to the folder affordance in POSA_dir.
You can control the number of placements for the same body mesh in a scene using the flag num_rendered_samples, default value is 1.
The generated feature maps can be shown by setting adding --show_gen_sample 1
You can also run the same script on the whole folder of test poses
python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses --scene_name MPH16 --render 1 --viz 1 To place clothed body meshes, you need to first buy the Renderpeople assets, or get the free models.
Create a folder rp_clothed_meshes in POSA_dir and place all the clothed body .obj meshes in this folder. Then run this command:
python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --scene_name MPH16 --render 1 --viz 1 --use_clothed_mesh 1POSA has been tested on the AGORA dataset only. Nonetheless, you can try POSA with any SMPL-X poses you have. You just need a .pkl file
with the SMPLX body parameters and the gender. Your SMPL-X vertices must be brought to a canonical form similar to the POSA training data.
This means the vertices should be centered at the pelvis joint, the x axis pointing to the left, the y axis pointing backward, and the z axis pointing upwards.
As shown in the figure below. The x,y,z axes are denoted by the red, green, blue colors respectively.
See the function pkl_to_canonical in data_utils.py for an example of how to do this transformation.
To retrain POSA from scratch run the following command
python src/train_posa.py --config cfg_files/contact_semantics.yamlYou can also visualize the training data
python src/show_gt.py --config cfg_files/contact_semantics.yaml --train_data 1Or test data
python src/show_gt.py --config cfg_files/contact_semantics.yaml --train_data 0Note that the ground truth data has been downsampled to speed up training as explained in the paper. See training details in appendices.
If you find this Model & Software useful in your research we would kindly ask you to cite:
@inproceedings{Hassan:CVPR:2021,
title = {Populating {3D} Scenes by Learning Human-Scene Interaction},
author = {Hassan, Mohamed and Ghosh, Partha and Tesch, Joachim and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {Proceedings {IEEE/CVF} Conf.~on Computer Vision and Pattern Recognition ({CVPR})},
month = jun,
month_numeric = {6},
year = {2021}
}
If you use the extracted training data, scenes or sdf the please cite:
@inproceedings{PROX:2019,
title = {Resolving {3D} Human Pose Ambiguities with {3D} Scene Constraints},
author = {Hassan, Mohamed and Choutas, Vasileios and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {International Conference on Computer Vision},
month = oct,
year = {2019},
url = {https://prox.is.tue.mpg.de},
month_numeric = {10}
}
@inproceedings{PSI:2019,
title = {Generating 3D People in Scenes without People},
author = {Zhang, Yan and Hassan, Mohamed and Neumann, Heiko and Black, Michael J. and Tang, Siyu},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2020},
url = {https://arxiv.org/abs/1912.02923},
month_numeric = {6}
}
If you use the AGORA test poses, the please cite:
@inproceedings{Patel:CVPR:2021,
title = {{AGORA}: Avatars in Geography Optimized for Regression Analysis},
author = {Patel, Priyanka and Huang, Chun-Hao P. and Tesch, Joachim and Hoffmann, David T. and Tripathi, Shashank and Black, Michael J.},
booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2021},
month_numeric = {6}
}
For commercial licensing (and all related questions for business applications), please contact [email protected].