1) Environment requirements
- Python 3.x
- Pytorch 1.11
- CUDA 9.2 or higher
The following installation guild suppose python=3.7 pytorch=1.11 and cuda=10.2. You may change them according to your system.
Create a conda virtual environment and activate it.
conda create -n softgroup python=3.7
conda activate softgroup
2) Clone the following two repositories.
git clone https://github.com/thangvubk/SoftGroup.git
git clone https://github.com/dbolya/yolact.git
note: Please put the above two projects in the same directory.
3) Install the dependencies.
conda install pytorch cudatoolkit=10.2 -c pytorch
pip install spconv-cu102
pip install -r requirements.txt
pip install opencv-python
pip install pycocotools
pip install PyQt5
pip install opencv-contrib-python==4.5.2.52
pip install pybullet
pip install open3D==0.8.0
4) Install build requirement.
sudo apt-get install libsparsehash-dev
5) Setup
python setup.py build_ext develop
6) Replace files
The files that need to be replaced are as follows.
SoftGroup
├── configs
│ ├── softgroup_s3dis_backbone_fold5.yaml
│ ├── softgroup_s3dis_fold5.yaml
├── data
│ ├── coco.py
│ ├── config.py
├── dataset
│ ├── s3dis
│ ├── downsample.py
│ ├── prepare_data.sh
│ ├── prepare_data_inst.py
│ ├── prepare_data_inst_gttxt.py
├── train_softgroup.py
├── train.py
├── yolact.py
Prepare your own 3D model files (urdf format)
1) Generate dataset
Please creates folders as follows.
dataset
├── sense_data
│ ├── depth
│ ├── ints_img
│ ├── label
│ ├── label_img
│ ├── rgb_img
│ ├── Stanford3dDataset_v1.2
│ ├── Area_1
│ ├── Area_2
│ ├── Area_3
│ ├── Area_4
│ ├── Area_5
│ ├── Area_6
│ ├── data
│ ├── banana
│ ├── bowl
│ ├── ...
Then run:
conda activate softgroup
cd create_data
python create_val_dataset.py
note: The 3D model path needs to be modified in create_val_dataset.py
2) Prepare 2D instance segmentation label
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install
pip install git+git://github.com/waspinator/[email protected]
Put ./create_data/create_json.py to ./cocoapi/PythonAPI
python create_name_list.py
python create_json.py
After running the script, you should get two files instances_train2017.json and instances_val2017.json.
Put the above two files to ./SoftGroup/data/coco/annotations.
Put the images under the ./dataset/sense_data/rbg_img folder into ./SoftGroup/data/coco/images.
3) Prepare 3D instance segmentation label
Put the ./dataset/sense_data/Stanford3dDataset_v1.2 to ./SoftGroup/dataset/s3dis/ folder.
Preprocess data
cd SoftGroup/dataset/s3dis
bash prepare_data.sh
4) Prepare shape estimation dataset
The generated dataset is stored in the ./dataset/sense_data/data directory
cd create_data
python create_scale_obj_dataset.py
1) Train 3D instance segmentation model
python train_softgroup.py --config=softgroup_s3dis_backbone_fold5.yaml
python train_softgroup.py --config=softgroup_s3dis_fold5.yaml
2) Train 2D instance segmentation model
python train.py --config=yolact_resnet50_config
3) Train 2D+3D instance segmentation model
First, Replace the following files with the files in the replace_files directory
SoftGroup
├── softgroup
│ ├── data
│ ├── custom.py
│ ├── s3dis.py
│ ├── __init__.py
│ ├── model
│ ├── softgroup.py
│ ├── util
│ ├── optim.py
Then run
python train_2D+3D.py
4) Train shape estimation model
python train_test_cp.py
5) Eval 2D+3D instance segmentation model
python eval_test.py
6) Eval overall_pipline
python eval_test_scale.py
7) Eval only 2D+shape-estimation
python eval_ints_scale.py
8) Eval only 3D+shape-estimation
python test_scale.py