News

• 31/07/2022 We plan to release the THUman5.0 dataset for acadamic use. The dataset contains 10 dynamic human sequences which are captured by 32 RGB cameras with resolution of 4000x3096. Please see here for more detais.

DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras

Project Page | Paper | Data

DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras
Ruizhi Shao, Zerong Zheng, Hongwen Zhang, Jingxiang Sun, Yebin Liu ECCV 2022

We plan to release the training and testing code of DiffuStereo in this repository as soon as possible. Any discussions or questions would be welcome!

Installation

Please see INSTALL.md for manual installation.

Pretrained model

We will provide the pretrained diffusion models for stereo matching including 20 degree and 45 degree as soon as possible. You can download and put them into the checkpoints/ directory.

Download: pretrained model for 20 degree stereo matching

Run the code on the THUman2.0 dataset

Please see THUman2.0 to download DEMO samples and the dataset.

Structure of DEMO samples

thuman_demo/
├── depth_db
│   └── 0001
│       ├── xxx.npz  --- the depth of view xxx (reconstructed by DoubleField)
├── img
│   └── 0001
│       ├── xxx.png  --- the image of view xxx
├── mask
│   └── 0001
│       ├── xxx.png  --- the mask of view xxx
├── normal_db
│   └── 0001
│       ├── xxx.png  --- the normal of view xxx (reconstructed by DoubleField)
└── parameter        --- We use perspective camera model to render images
    └── 0001   
        ├── xxx_extrinsic.npy  --- the extrinsic of view xxx (3x4 world-to-camera matrix)
        ├── xxx_intrinsic.npy  --- the intrinsic of view xxx (3x3 matrix)

Inference on a single stereo matching pair

To inference on a single stereo matching pair of DEMO samples, please run the following script.

python -m app.inference --config configs/thuman_demo.yaml --dataroot [the directory of DEMO samples] --view_list 0 20

Visualization on a single stereo matching pair

The results will be saved in results/thuman_demo/. To visualize the results, use MeshLab to open results/thuman_demo/fusion000.ply and apply Possion Reconstruction with depth=11 or depth=10.

Inference on multiple stereo matching pairs

To inference on multiple stereo matching pairs of DEMO samples, please run the following script.

python -m app.inference --config configs/thuman_demo_multi.yaml --dataroot [the directory of DEMO samples] --view_list 0 20 90 110 180 200 270 290 --use_db_normal

For cases with 45 degree, please run the following script.

python -m app.inference --config configs/thuman_demo_multi_45.yaml --dataroot [the directory of DEMO samples] --view_list 0 45 90 135 180 225 270 315 --use_db_normal

Naive multi-view fusion and visualization on the results

The results will be saved in results/thuman_demo_multi/. To simply fuse multi-view depth point clouds, use MeshLab to open results/thuman_demo_multi/fusion000.ply and apply Possion Reconstruction with depth=11 or depth=10.

Run the code on the THUman5.0 dataset

Please see DATASET.md to download DEMO samples and the dataset.

Structure of DEMO samples

real_demo/
├── depth_db
│   └── 0001
│       ├── xxx.npz  --- the depth of view xxx (reconstructed by DoubleField)
├── img
│   └── 0001
│       ├── xxx.jpg  --- the image of view xxx (after undistortion)
├── mask
│   └── 0001
│       ├── xxx.jpg  --- the mask of view xxx (after undistortion)
├── normal_db
│   └── 0001
│       ├── xxx.png  --- the normal of view xxx (reconstructed by DoubleField)
└── parameter        --- parameters of real world perspective camera model (after undistortion)
    └── 0001   
        ├── xxx_extrinsic.npy  --- the extrinsic of view xxx (3x4 world-to-camera matrix)
        ├── xxx_intrinsic.npy  --- the intrinsic of view xxx (3x3 matrix)

Inference on one stereo matching pair

To inference on a single stereo matching pair of DEMO samples, please run the following script.

python -m app.inference --config configs/real_demo.yaml --dataroot [the directory of DEMO samples] --view_list 0 1

Visualization on one stereo matching pair

The results will be saved in results/real_demo/. To visualize the results, use MeshLab to open results/real_demo/fusion000.ply and apply Possion Reconstruction with depth=11 or depth=10.

Multi-view fusion

On account of calibration error and the complicated lighting environment in the real-world dataset, naive multi-view fusion based on Poisson Reconstruction will generate noise and artifacts. We recommend using traditional multi-view fusion algorithm or programming our proposed light-weight multi-view fusion to reproduce the final results.

Training on the 3D human scan dataset

Please see THUman2.0 to download DEMO samples and the dataset.

TODO

Run the code on the custom dataset

To run the code on the custom dataset, please see CUSTOM. We also clarify our camera model and provide some examples to render depth and normal maps.

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{shao2022diffustereo,
    author = {Shao, Ruizhi and Zheng, Zerong and Zhang, Hongwen and Sun, Jingxiang and Liu, Yebin},
    title = {DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras},
    booktitle = {ECCV},
    year = {2022}
}

Acknowledgments

Our project is benefit from these great resources:

• Diffusion Probabilistic Models for 3D Point Cloud Generation
• Diffusion Models Beat GANS on Image Synthesis
• RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching
• StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision
• POP: The Power of Points for Modeling Humans in Clothing

Thanks for their sharing code.