init commit

LICENSE

@@ -0,0 +1,59 @@
+License
+Software Copyright License for non-commercial scientific research purposes
+Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the POSA data, model and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License
+
+Ownership / Licensees
+The Software and the associated materials has been developed at the
+
+Max Planck Institute for Intelligent Systems (hereinafter "MPI").
+
+Any copyright or patent right is owned by and proprietary material of the
+
+Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter collectively “Max-Planck”)
+
+hereinafter the “Licensor”.
+
+License Grant
+Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:
+
+To install the Data & Software on computers owned, leased or otherwise controlled by you and/or your organization;
+To use the Data & Software for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects;
+Any other use, in particular any use for commercial, pornographic, military, or surveillance, purposes is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artefacts for commercial purposes. The software may not be used to create fake, libelous, misleading, or defamatory content of any kind excluding analyses in peer-reviewed scientific research. The Software may not be reproduced, modified and/or made available in any form to any third party without Max-Planck’s prior written permission.
+
+The Software may not be used for pornographic purposes or to generate pornographic material whether commercial or not. This license also prohibits the use of the Software to train methods/algorithms/neural networks/etc. for commercial, pornographic, military, surveillance, or defamatory use of any kind. By downloading the Software, you agree not to reverse engineer it.
+
+No Distribution
+The Data & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive purposes only.
+
+Disclaimer of Representations and Warranties
+You expressly acknowledge and agree that the Data & Software results from basic research, is provided “AS IS”, may contain errors, and that any use of the Data & Software is at your sole risk. LICENSOR MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE DATA & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of the Data & Software, (ii) that the use of the Data & Software will not infringe any patents, copyrights or other intellectual property rights of a third party, and (iii) that the use of the Data & Software will not cause any damage of any kind to you or a third party.
+
+Limitation of Liability
+Because this Data & Software License Agreement qualifies as a donation, according to Section 521 of the German Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only. If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee for the resulting damage.
+Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be liable in accordance with the German Product Liability Act in the event of product liability. The foregoing applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall be excluded.
+Patent claims generated through the usage of the Data & Software cannot be directed towards the copyright holders.
+The Data & Software is provided in the state of development the licensor defines. If modified or extended by Licensee, the Licensor makes no claims about the fitness of the Data & Software and is not responsible for any problems such modifications cause.
+
+No Maintenance Services
+You understand and agree that Licensor is under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Data & Software. Licensor nevertheless reserves the right to update, modify, or discontinue the Data & Software at any time.
+
+Defects of the Data & Software must be notified in writing to the Licensor with a comprehensible description of the error symptoms. The notification of the defect should enable the reproduction of the error. The Licensee is encouraged to communicate any use, results, modification or publication.
+
+Publications using the Data & Software
+You acknowledge that the Data & Software is a valuable scientific resource and agree to appropriately reference the following paper in any publication making use of the Data & Software.
+
+Citation:
+
+
+@inproceedings{Hassan:CVPR:2021,
+    title = {Populating {3D} Scenes by Learning Human-Scene Interaction},
+    author = {Hassan, Mohamed and Ghosh, Partha and Tesch, Joachim and Tzionas, Dimitrios and Black, Michael J.},
+    booktitle = {Proceedings {IEEE/CVF} Conf.~on Computer Vision and Pattern Recognition ({CVPR})},
+    month = jun,
+    month_numeric = {6},
+    year = {2021}
+}
+Commercial licensing opportunities
+For commercial uses of the Data & Software, please send email to [email protected]
+
+This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.

README.md

@@ -0,0 +1,187 @@
+## Populating 3D Scenes by Learning Human-Scene Interaction
+
+[[Project Page](https://posa.is.tue.mpg.de/)]
+[[Paper](https://arxiv.org/pdf/2012.11581.pdf)]
+
+![POSA Examples](./images/teaser.jpg)
+
+
+## License
+
+Software Copyright License for non-commercial scientific research purposes.
+Please read carefully the following [terms and conditions](LICENSE) and any accompanying
+documentation before you download and/or use the POSA data, model and
+software, (the "Data & Software"), including 3D meshes, images, videos,
+textures, software, scripts, and animations. By downloading and/or using the
+Data & Software (including downloading, cloning, installing, and any other use
+of the corresponding github repository), you acknowledge that you have read
+these [terms and conditions](LICENSE), understand them, and agree to be bound by them. If
+you do not agree with these [terms and conditions](LICENSE), you must not download and/or
+use the Data & Software. Any infringement of the terms of this agreement will
+automatically terminate your rights under this [License](LICENSE).
+
+## Description
+
+This repository contains the training, random sampling, and scene population code used for the experiments in [POSA](https://posa.is.tue.mpg.de/).
+
+## Installation 
+
+To install the necessary dependencies run the following command:
+```shell
+    pip install -r requirements.txt
+```
+The code has been tested with Python 3.7, CUDA 10.0, CuDNN 7.5 and PyTorch 1.7 on Ubuntu 20.04. 
+
+## Dependencies
+### POSA_dir
+To be able to use the code you need to get the [POSA_dir.zip](https://posa.is.tue.mpg.de/). After unzipping, you should have a directory with the following structure:
+```bash
+POSA_dir
+├── cam2world
+├── data
+├── mesh_ds
+├── scenes
+├── sdf
+└── trained_models
+```
+The content of each folder is explained below:
+- `trained_models` contains two trained models. One is trained on the `contact` only and the other one is trained on `contact` and `semantics`.
+- `data` contains the train and test data extracted from the [PROX Dataset](https://prox.is.tue.mpg.de/) and [PROX-E Dataset](https://github.com/yz-cnsdqz/PSI-release#datasets).
+- `scenes` contains the 12 scenes from [PROX Dataset](https://prox.is.tue.mpg.de/)
+- `sdf` contains the signed distance field for the scenes in the previous folder.
+- `mesh_ds` contains mesh downsampling and upsampling related files similar to the ones in [COMA](https://coma.is.tue.mpg.de/).
+
+### SMPL-X
+You need to get the [SMPLx Body Model](https://smpl-x.is.tue.mpg.de/). Please extract the folder and rename it 
+to `smplx_models` and place it in the `POSA_dir` above.
+
+### AGORA
+In addition, you need to get the `POSA_rp_poses.zip` file from [AGORA Dataset](https://agora.is.tue.mpg.de/) and extract in the `POSA_dir`. 
+This file contrains a number of test poses to be used in the next steps. Note that you don't need the whole AGORA dataset.
+
+Finally run the following command or add it to your `~/.bashrc`
+```Shell
+export POSA_dir=Path of Your POSA_dir
+```
+# Inference
+You can test `POSA` using the trained models provided. Below we provide examples of how to generate POSA features and how to pupulate a 3D scene.
+## Random Sampling
+To generate random features from a trained model, run the following command
+```Shell
+python src/gen_rand_samples.py --config cfg_files/contact.yaml --checkpoint_path $POSA_dir/trained_models/contact.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --render 1 --viz 1 --num_rand_samples 3 
+```
+Or
+```Shell
+python src/gen_rand_samples.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --render 1 --viz 1 --num_rand_samples 3 
+```
+
+This will open a window showing the generated features for the specified `pkl` file. It also render the features to the folder `random_samples` in `POSA_dir`.
+
+The number of generated feature maps can be controlled by the flag `num_rand_samples`.
+
+If you don't have a screen, you can turn off the visualization `--viz 0`.
+
+If you don't have CUDA installed then you can add this flag `--use_cuda 0`. This applies to all commands in this repository.
+
+You can also run the same command on the whole folder of test poses
+```Shell
+python src/gen_rand_samples.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses --render 1 --viz 1 --num_rand_samples 3 
+```
+## Scene Population
+Given a body mesh from the [AGORA Dataset](https://agora.is.tue.mpg.de/), `POSA` automatically places the body mesh in 3D scene. 
+```Shell
+python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --scene_name MPH16 --render 1 --viz 1 
+```
+This will open a window showing the placed body in the scene. It also render the placements to the folder `affordance` in `POSA_dir`.
+
+You can control the number of placements for the same body mesh in a scene using the flag `num_rendered_samples`, default value is `1`. 
+
+The generated feature maps can be shown by setting adding `--show_gen_sample 1`
+
+You can also run the same script on the whole folder of test poses
+```Shell
+python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses --scene_name MPH16 --render 1 --viz 1 
+```
+
+To place clothed body meshes, you need to first buy the [Renderpeople](https://renderpeople.com/) assets, or get the [free models](https://renderpeople.com/free-3d-people/). 
+Create a folder `rp_clothed_meshes` in `POSA_dir` and place all the clothed body `.obj` meshes in this folder. Then run this command:
+```Shell
+python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --scene_name MPH16 --render 1 --viz 1 --use_clothed_mesh 1
+```
+
+# Testing on Your Own Poses
+`POSA` has been tested on the `AGORA` dataset **only**. Nonetheless, you can try `POSA` with any `SMPL-X` poses you have. You just need a `.pkl` file
+with the `SMPLX` body parameters and the gender. Your `SMPL-X` vertices **must** be brought to a canonical form similar to the POSA training data.
+This means the vertices should be centered at the pelvis joint, the `x` axis pointing to the left, the `y` axis pointing backward, and the `z` axis pointing upwards.
+As shown in the figure below. The x,y,z axes are denoted by the red, green, blue colors respectively.
+
+![canonical_form](./images/canonical.png)
+
+See the function `pkl_to_canonical` in [data_utils.py](src/data_utils.py) for an example of how to do this transformation.
+
+# Training
+To retrain `POSA` from scratch run the following command
+```Shell
+python src/train_posa.py --config cfg_files/contact_semantics.yaml
+```
+
+# Visualize Ground Truth Data
+You can also visualize the training data
+```Shell
+python src/show_gt.py --config cfg_files/contact_semantics.yaml --train_data 1
+```
+Or test data
+```Shell
+python src/show_gt.py --config cfg_files/contact_semantics.yaml --train_data 0
+```
+Note that the ground truth data has been downsampled to speed up training as explained in the paper. [See training details in appendices](https://arxiv.org/pdf/2012.11581.pdf). 
+
+## Citation
+
+If you find this Model & Software useful in your research we would kindly ask you to cite:
+
+```
+@inproceedings{Hassan:CVPR:2021,
+    title = {Populating {3D} Scenes by Learning Human-Scene Interaction},
+    author = {Hassan, Mohamed and Ghosh, Partha and Tesch, Joachim and Tzionas, Dimitrios and Black, Michael J.},
+    booktitle = {Proceedings {IEEE/CVF} Conf.~on Computer Vision and Pattern Recognition ({CVPR})},
+    month = jun,
+    month_numeric = {6},
+    year = {2021}
+}
+```
+If you use the extracted training data, scenes or sdf the please cite:
+```
+@inproceedings{PROX:2019,
+  title = {Resolving {3D} Human Pose Ambiguities with {3D} Scene Constraints},
+  author = {Hassan, Mohamed and Choutas, Vasileios and Tzionas, Dimitrios and Black, Michael J.},
+  booktitle = {International Conference on Computer Vision},
+  month = oct,
+  year = {2019},
+  url = {https://prox.is.tue.mpg.de},
+  month_numeric = {10}
+}
+@inproceedings{PSI:2019,
+  title = {Generating 3D People in Scenes without People},
+  author = {Zhang, Yan and Hassan, Mohamed and Neumann, Heiko and Black, Michael J. and Tang, Siyu},
+  booktitle = {Computer Vision and Pattern Recognition (CVPR)},
+  month = jun,
+  year = {2020},
+  url = {https://arxiv.org/abs/1912.02923},
+  month_numeric = {6}
+}
+```
+If you use the AGORA test poses, the please cite:
+```
+@inproceedings{Patel:CVPR:2021,
+  title = {{AGORA}: Avatars in Geography Optimized for Regression Analysis},
+  author = {Patel, Priyanka and Huang, Chun-Hao P. and Tesch, Joachim and Hoffmann, David T. and Tripathi, Shashank and Black, Michael J.},
+  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
+  month = jun,
+  year = {2021},
+  month_numeric = {6}
+}
+```
+
+## Contact
+For commercial licensing (and all related questions for business applications), please contact [[email protected]](mailto:[email protected]).

cfg_files/contact.yaml

@@ -0,0 +1,37 @@
+output_dir: '$POSA_dir/trained_models/contact'
+
+#terms/weights
+contact_w: 1
+use_semantics: 0
+semantics_w: 0
+kl_w: 1.0
+
+trunc_val: 0.5
+log_interval: 5
+epochs: 15
+step: 1
+test: 1
+
+#arch/training common
+block_size: 1
+seq_length: 9
+channels: 64
+num_hidden_layers: 3
+z_dim: 256
+h_dim: 512
+drop_out: 0
+batch_size: 64
+num_groups: 8
+normalization_mode: 'group_norm'
+lr: 1e-3
+reduce_lr: 0
+num_workers: 0
+
+loss_type: 'bce'
+no_obj_classes: 42
+down_sample: 2
+up_sample: False
+
+#Paths
+data_dir: '$POSA_dir/data'
+ds_us_dir: '$POSA_dir/mesh_ds'

cfg_files/contact_semantics.yaml

@@ -0,0 +1,37 @@
+output_dir: '$POSA_dir/trained_models/contact_semantics'
+
+#terms/weights
+contact_w: 1
+use_semantics: 1
+semantics_w: 1
+kl_w: 0.1
+
+trunc_val: 0.5
+log_interval: 5
+epochs: 15
+step: 1
+test: 1
+
+#arch/training common
+block_size: 1
+seq_length: 9
+channels: 64
+num_hidden_layers: 3
+z_dim: 256
+h_dim: 512
+drop_out: 0
+batch_size: 64
+num_groups: 8
+normalization_mode: 'group_norm'
+lr: 1e-3
+reduce_lr: 0
+num_workers: 0
+
+loss_type: 'bce'
+no_obj_classes: 42
+down_sample: 2
+up_sample: False
+
+#Paths
+data_dir: '$POSA_dir/data'
+ds_us_dir: '$POSA_dir/mesh_ds'

mpcat40.tsv

@@ -0,0 +1,43 @@
+mpcat40index	mpcat40	hex	wnsynsetkey	nyu40	skip	labels
+0	void	#ffffff		void		remove,void
+1	wall	#aec7e8	wall.n.01,baseboard.n.01,paneling.n.01	wall		wallpaper
+2	floor	#708090	floor.n.01,rug.n.01,mat.n.01,bath_mat.n.01,landing.n.01	floor,floor mat		
+3	chair	#98df8a	chair.n.01,beanbag.n.01	chair		
+4	door	#c5b0d5	door.n.01,doorframe.n.01,doorway.n.01,doorknob.n.01,archway.n.01	door		garage door
+5	table	#ff7f0e	table.n.02,dressing.n.04	table,desk	counter.n.01	
+6	picture	#d62728	picture.n.01,photograph.n.01,picture_frame.n.01	picture		
+7	cabinet	#1f77b4	cabinet.n.01,cupboard.n.01	cabinet		
+8	cushion	#bcbd22	cushion.n.03	pillow		couch cushion
+9	window	#ff9896	window.n.01,windowsill.n.01,window_frame.n.01,windowpane.n.02,window_screen.n.01	window		
+10	sofa	#2ca02c	sofa.n.01	sofa		
+11	bed	#e377c2	bed.n.01,bedpost.n.01,bedstead.n.01,headboard.n.01,footboard.n.01,bedspread.n.01,mattress.n.01,sheet.n.03	bed		
+12	curtain	#de9ed6	curtain.n.01	curtain,shower curtain		curtain rod,shower curtain rod,shower rod
+13	chest_of_drawers	#9467bd	chest_of_drawers.n.01,drawer.n.01	dresser,night stand		
+14	plant	#8ca252	plant.n.02			
+15	sink	#843c39	sink.n.01	sink		
+16	stairs	#9edae5	step.n.04,stairway.n.01,stairwell.n.01			
+17	ceiling	#9c9ede	ceiling.n.01,roof.n.01	ceiling		
+18	toilet	#e7969c	toilet.n.01,bidet.n.01	toilet		
+19	stool	#637939	stool.n.01			
+20	towel	#8c564b	towel.n.01	towel		
+21	mirror	#dbdb8d	mirror.n.01	mirror		
+22	tv_monitor	#d6616b	display.n.06	television		
+23	shower	#cedb9c	shower.n.01,showerhead.n.01			
+24	column	#e7ba52	column.n.07,post.n.04			
+25	bathtub	#393b79	bathtub.n.01	bathtub		
+26	counter	#a55194	countertop.n.01,counter.n.01,kitchen_island.n.01	counter		
+27	fireplace	#ad494a	fireplace.n.01,mantel.n.01			
+28	lighting	#b5cf6b	lamp.n.02,lampshade.n.01,light.n.02,chandelier.n.01,spotlight.n.02	lamp		
+29	beam	#5254a3	beam.n.02			
+30	railing	#bd9e39	railing.n.01,bannister.n.02			
+31	shelving	#c49c94	bookshelf.n.01,shelf.n.01,rack.n.05	shelves		
+32	blinds	#f7b6d2	window_blind.n.01	blinds		
+33	gym_equipment	#6b6ecf	sports_equipment.n.01,treadmill.n.01,exercise_bike.n.01			
+34	seating	#ffbb78	bench.n.01,seat.n.03			
+35	board_panel	#c7c7c7	panel.n.01	whiteboard		board
+36	furniture	#8c6d31	furniture.n.01	otherfurniture		
+37	appliances	#e7cb94	home_appliance.n.01,stove.n.02,dryer.n.01	refridgerator		washing machine and dryer
+38	clothes	#ce6dbd	clothing.n.01	clothes		
+39	objects	#17becf	physical_object.n.01,material.n.01	books,paper,box,bag,otherprop	structure.n.01,way.n.06,vent.n.01,unknown.n.01,pool.n.01	
+40	misc	#7f7f7f		person,otherstructure	unknown.n.01	
+41	unlabeled	#000000	unknown.n.01			unknown

requirements.txt

@@ -0,0 +1,21 @@
+openmesh
+sklearn
+numpy>=1.16.2
+torch==1.7.0
+torchgeometry>=0.1.2
+PyYAML==5.1.1
+configargparse
+smplx
+pillow
+opencv-python
+tqdm
+pyrender==0.1.36
+trimesh>=2.37.6
+scipy
+open3d
+times
+DateTime
+tensorboard
+pandas
+ipdb
+