SIENet: Spatial Information Enhancement Network

This is the official implementation of SIENet (Pattern Recognition, 2022), a 2-stage detector leveraging spatial structure information for robust and accurate 3D detection from point clouds. For more details, please refer to:

Spatial information enhancement network for 3D object detection from point cloud

Ziyu Li, Yuncong Yao, Zhibin Quan, Jin Xie, Wankou Yang

[Paper]

Introduction

LiDAR-based 3D object detection pushes forward an immense influence on autonomous vehicles. Due to the limitation of the intrinsic properties of LiDAR, fewer points are collected at the objects farther away from ego-vehicle. This imbalanced density of point clouds degrades the detection accuracy. To address the challenge, we propose the SIENet (Spatial Information Enhancement Network) to recover the dense shapes and extract spatial information for robust and accurate 3D detection from point clouds. SIENet has two key components:

• Spatial Information Enhancement (SIE) module. We devise the SIE module to predict the spatial shapes of foreground points within proposals, and extract the structure information to learn representative features for further box refinement.
• Hybrid-Paradigm Region Proposal Network (HP-RPN). This sub-network includes multiple branches to learn discriminate features and generate accurate proposals for the SIE module.

Important Notes: The training process of SIENet requires 4 NVIDIA TITAN XP GPUs (total batch size 8), which means a minimum GPU memory of 48 (4*12) GB is needed for reproducing the results. If your GPU resource is less than 48 GB, you can decrease the batch size in the training command, but this may harm the final performance. The 3D/BEV AP fluctuates at different epochs, thus it is advised to evaluate all saved checkpoints.

*The code is mainly based on OpenPCDet.

Requirements

We have tested this project with the following environments:

• ubuntu 16.04/18.04
• PyTorch 1.2.0
• Spconv v1.0
• OpenPCDet v0.2.0

Installation

a. Clone this repository.

git clone [email protected]:Liz66666/SIENet.git

b. Install some dependent libraries as follows:

• Install the dependent libraries of python:

pip install -r requirements.txt

• Install the SparseConv library, we use the implementation from [spconv]. Please note that we recommend installing the spconv v1.0 with (commit 8da6f96) instead of the latest one.

c. Compile CUDA operators of pcdet by running the following command:

python setup.py develop

d. Prepare the KITTI dataset according to the [guidance doc] of OpenPCDet.

Model Zoo

We provide the checkpoint of SIENet, as well as the pre-trained spatial shape prediction network (reproduced version of PCN on car class).

Model	Easy@R11	Mod.@R11	Hard@R11	Easy@R40	Mod.@R40	Hard@R40	Download
SIENet	89.57	84.40	79.08	92.49	85.43	83.05	model-235M
PCN	-	-	-	-	-	-	model-78M

• As mentioned in the paper, the spatial shape prediction network needs a pre-trained PCN to initialize, thus please download the pre-trained PCN and organize it as follows:

SIENet
├── data
├── checkpoints
│	├── pcn_step170000.pth
├── pcdet
├── tools

Training

You could optionally add extra command line parameters to specify your preferred parameters. The performance of SIENet is quite unstable on KITTI. If you cannot reproduce the results, remember to run it multiple times.

• Train with a single GPU:

python train.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} 

• Train with multiple GPUs:

bash scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} 

Evaluation

• Test with a pretrained model:

python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT_PATH}

• Test with a series of models saved in a folder:

python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt_dir ${CKPTDIR_PATH} --eval_all

• To test with multiple GPUs:

bash scripts/dist_test.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT_PATH}

License

SIENet is released under the Apache 2.0 license.

Acknowledgment

Thanks for the works below, this project will not be possible without these excellent codebases.

• OpenPCDet

• SA-SSD

• PCN

• Deformable PV-RCNN

Citation

If you find this project useful in your research, please consider citing:

@article{li2022spatial,
  title={Spatial Information Enhancement Network for 3D Object Detection from Point Cloud},
  author={Li, Ziyu and Yao, Yuncong and Quan, Zhibin and Xie, Jin and Yang, Wankou},
  journal={Pattern Recognition},
  pages={108684},
  year={2022},
  publisher={Elsevier}
}