initial commit

weiyithu · Mar 28, 2022 · 76367c4 · 76367c4
1 parent dbcf27b
commit 76367c4
Show file tree

Hide file tree

Showing 173 changed files with 23,881 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -1,2 +1,94 @@
-# LiDAR-Distillation
-LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection
+# LiDAR Distillation
+### [Paper](https://arxiv.org/abs/2109.01129) | [Model](https://drive.google.com/drive/folders/1X_w57Q_MIFlI3lzhRt7Z8C5X9tNS8cg-?usp=sharing)
+<br/>
+
+> LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection  
+> [Yi Wei](https://weiyithu.github.io/), Zibu Wei, [Yongming Rao](https://raoyongming.github.io/), [Jiaxin Li](https://www.jiaxinli.me), [Jiwen Lu](http://ivg.au.tsinghua.edu.cn/Jiwen_Lu/), [Jie Zhou](https://scholar.google.com/citations?user=6a79aPwAAAAJ&hl=en&authuser=1)  
+
+<p align='center'>
+<img src="./docs/framework.png" width='80%'/>
+</p>
+
+
+## Introduction
+
+In this paper, we propose the LiDAR Distillation to bridge the domain gap induced by different LiDAR beams for 3D object detection. In many real-world applications, the LiDAR points used by mass-produced robots and vehicles usually have fewer beams than that in large-scale public datasets. Moreover, as the LiDARs are upgraded
+to other product models with different beam amount, it becomes challenging to utilize the labeled data captured by previous versions’ high-resolution sensors. Despite the recent progress on domain adaptive 3D
+detection, most methods struggle to eliminate the beam-induced domain gap. 
+
+## Model Zoo
+
+### Cross-dataset Adaptation
+
+| model     | method | AP_BEV | AP_3D |  
+|------------|----------:|:-------:|:-------:|
+| SECOND-IoU | Direct transfer | 32.91  | 17.24 | 
+| SECOND-IoU | ST3D | 35.92 | 20.19 | 
+| [SECOND-IoU](https://cloud.tsinghua.edu.cn/f/5a1f958ce2d84e81823b/?dl=1) | Ours | 40.66 | 22.86 | 
+| [SECOND-IoU](https://cloud.tsinghua.edu.cn/f/a3dd295ac1824d19abf8/?dl=1) | Ours (w / ST3D) | 42.04 | 24.50 |
+| PV-RCNN | Direct transfer | 34.50 | 21.47 | 
+| PV-RCNN | ST3D | 36.42 | 22.99 | 
+| [PV-RCNN](https://cloud.tsinghua.edu.cn/f/1cea8483c9494df5b46f/?dl=1) | Ours | 43.31 | 25.63 | 
+| [PV-RCNN](https://cloud.tsinghua.edu.cn/f/3fa7c39d269e4425b4e3/?dl=1) | Ours (w / ST3D) | 44.08 | 26.37 |
+| PointPillar | Direct transfer | 27.8 | 12.1 | 
+| PointPillar | ST3D | 30.6 | 15.6 | 
+| [PointPillar](https://cloud.tsinghua.edu.cn/f/d919a97bb66d467498d6/?dl=1) | Ours | 40.23 | 19.12| 
+| [PointPillar](https://cloud.tsinghua.edu.cn/f/8466efb0c91c4d4caeb0/?dl=1) | Ours (w / ST3D) | 40.83 | 20.97 |
+
+Results of cross-dataset adaptation from Waymo to nuScenes. The training Waymo data used in our work is version 1.0.
+
+### Single-dataset Adaptation
+
+|  beams  | method | AP_BEV | AP_3D |  
+|------------------|----------:|:-------:|:-------:|
+| 32 | Direct transfer | 79.81 | 65.91 | 
+| 32 | ST3D | 71.29 | 57.57 | 
+| [32](https://cloud.tsinghua.edu.cn/f/2664ab0e209348a1b5c2/?dl=1) | Ours | 82.22 | 70.15 | 
+| 32* | Direct transfer | 73.56  | 57.77 | 
+| 32* | ST3D | 67.08 | 53.30 | 
+| [32*](https://cloud.tsinghua.edu.cn/f/be087fb6b9634a3c98fe/?dl=1) | Ours | 79.47 | 66.96 | 
+| 16 | Direct transfer | 64.91  | 47.48 | 
+| 16 | ST3D | 57.58 | 42.40 | 
+| [16](https://cloud.tsinghua.edu.cn/f/d8888921ff2a48c9a381/?dl=1) | Ours |  74.32 |  59.87 | 
+| 16* | Direct transfer |  56.32  | 38.75 | 
+| 16* | ST3D | 55.63 | 37.02 | 
+| [16*](https://cloud.tsinghua.edu.cn/f/21d8708a9c464945be61/?dl=1) | Ours | 70.43 | 55.24 | 
+
+Results of single-dataset adaptation on KITTI dataset with PointPillars (moderate difficulty). For SECOND-IoU and PV-RCNN, we find that it is easy to raise cuda error on low-beam data, which is may caused by the bug in spconv. Thus, we do not provide the model but you can still run these experiments with the [yamls](tools/cfgs/kitti_models).
+
+
+## Installation
+
+Please refer to [INSTALL.md](docs/INSTALL.md).
+
+## Getting Started
+
+Please refer to [GETTING_STARTED.md](docs/GETTING_STARTED.md).
+
+## License
+
+Our code is released under the Apache 2.0 license.
+
+## Acknowledgement
+
+Our code is heavily based on [OpenPCDet v0.2](https://github.com/open-mmlab/OpenPCDet/tree/v0.2.0) and [ST3D](https://github.com/CVMI-Lab/ST3D). Thanks OpenPCDet Development Team for their awesome codebase.
+
+## Citation
+
+If you find this project useful in your research, please consider cite:
+```
+@article{wei2022lidar,
+    title={LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection},
+    author={Wei, Yi and Wei, Zibu and Rao, Yongming and Lu, Jiwen and Zhou, Jie},
+    journal={arXiv preprint arXiv:2022.6980},
+    year={2022}
+}
+```
+```
+@misc{openpcdet2020,
+    title={OpenPCDet: An Open-source Toolbox for 3D Object Detection from Point Clouds},
+    author={OpenPCDet Development Team},
+    howpublished = {\url{https://github.com/open-mmlab/OpenPCDet}},
+    year={2020}
+}
+```
diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md
@@ -0,0 +1,177 @@
+# Getting Started
+The dataset configs are located within [tools/cfgs/dataset_configs](../tools/cfgs/dataset_configs), 
+and the model configs are located within [tools/cfgs](../tools/cfgs) for different datasets. 
+
+### KITTI Dataset
+* Please download the official [KITTI 3D object detection](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) dataset and organize the downloaded files as follows (the road planes could be downloaded from [[road plane]](https://drive.google.com/file/d/1d5mq0RXRnvHPVeKx6Q612z0YRO1t2wAp/view?usp=sharing), which are optional for data augmentation in the training):
+* NOTE: if you already have the data infos from `pcdet v0.1`, you can choose to use the old infos and set the DATABASE_WITH_FAKELIDAR option in tools/cfgs/dataset_configs/kitti_dataset.yaml as True. The second choice is that you can create the infos and gt database again and leave the config unchanged.
+
+```
+OpenPCDet
+├── data
+│   ├── kitti
+│   │   │── ImageSets
+│   │   │── training
+│   │   │   ├──calib & velodyne & label_2 & image_2 & (optional: planes)
+|   |   |   ├──modes
+|   |   |      ├──64
+|   |   |      ├──32
+|   |   |      ├──32^
+|   |   |      ├──16
+|   |   |      ├──16^
+│   │   │── testing
+│   │   │   ├──calib & velodyne & image_2
+├── pcdet
+├── tools
+```
+
+* Generate the data infos and extract point cloud data in modes folder by running the following command: 
+```python 
+python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml
+```
+
+### NuScenes Dataset
+* Please download the official [NuScenes 3D object detection dataset](https://www.nuscenes.org/download) and 
+organize the downloaded files as follows: 
+```
+OpenPCDet
+├── data
+│   ├── nuscenes
+│   │   │── v1.0-trainval (or v1.0-mini if you use mini)
+│   │   │   │── samples
+│   │   │   │── sweeps
+│   │   │   │── maps
+│   │   │   │── v1.0-trainval  
+├── pcdet
+├── tools
+```
+
+* Install the `nuscenes-devkit` with version `1.0.5` by running the following command: 
+```shell script
+pip install nuscenes-devkit==1.0.5
+```
+
+* Generate the data infos by running the following command (it may take several hours): 
+```python 
+python -m pcdet.datasets.nuscenes.nuscenes_dataset --func create_nuscenes_infos \ 
+    --cfg_file tools/cfgs/dataset_configs/nuscenes_dataset.yaml \
+    --version v1.0-trainval
+```
+
+### Waymo Open Dataset
+* Please download the official [Waymo Open Dataset](https://waymo.com/open/download/), 
+including the training data `training_0000.tar~training_0031.tar` and the validation 
+data `validation_0000.tar~validation_0007.tar`.
+* Unzip all the above `xxxx.tar` files to the directory of `data/waymo/raw_data` as follows (You could get 798 *train* tfrecord and 202 *val* tfrecord ):  
+```
+OpenPCDet
+├── data
+│   ├── waymo
+│   │   │── ImageSets
+│   │   │── raw_data
+│   │   │   │── segment-xxxxxxxx.tfrecord
+|   |   |   |── ...
+|   |   |── modes
+│   │   │   │── 64
+│   │   │       │── segment-xxxxxxxx/
+|   |   |       |── ...
+│   │   │   │── 32
+│   │   │   │── 16^
+│   │   │── pcdet_gt_database_train_sampled_xx/
+│   │   │── pcdet_waymo_dbinfos_train_sampled_xx.pkl   
+├── pcdet
+├── tools
+```
+* Install the official `waymo-open-dataset` by running the following command: 
+```shell script
+pip3 install --upgrade pip
+# tf 2.0.0
+pip3 install waymo-open-dataset-tf-2-0-0==1.2.0 --user
+```
+
+* Extract point cloud data with different beams from tfrecord and generate data infos by running the following command (it takes several hours, 
+and you could refer to `data/waymo/modes/64` to see how many records that have been processed): 
+```python 
+python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
+    --cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml
+```
+
+Note that you do not need to install `waymo-open-dataset` if you have already processed the data before and do not need to evaluate with official Waymo Metrics. 
+
+## Training & Testing
+
+
+### Test and evaluate the pretrained models
+* Test with a pretrained model: 
+```shell script
+python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}
+```
+
+* To test all the saved checkpoints of a specific training setting and draw the performance curve on the Tensorboard, add the `--eval_all` argument: 
+```shell script
+python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --eval_all
+```
+
+* To test with multiple GPUs:
+```shell script
+sh scripts/dist_test.sh ${NUM_GPUS} \
+    --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}
+
+# or
+
+sh scripts/slurm_test_mgpu.sh ${PARTITION} ${NUM_GPUS} \ 
+    --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}
+```
+
+
+### Train a model
+You could optionally add extra command line parameters `--batch_size ${BATCH_SIZE}` and `--epochs ${EPOCHS}` to specify your preferred parameters. 
+
+
+* Train with multiple GPUs or multiple machines
+```shell script
+sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}
+
+# or 
+
+sh scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} ${NUM_GPUS} --cfg_file ${CONFIG_FILE}
+```
+
+* Train with a single GPU:
+```shell script
+python train.py --cfg_file ${CONFIG_FILE}
+```
+
+### LiDAR Distillation 
+Take PointPillars on Waymo -> nuScenes as an example:
+1. Train 64-beam teacher model
+```shell script
+sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file cfgs/da-waymo-nus_models/pointpillar/pointpillar.yaml \
+    --batch_size ${BATCH_SIZE} --extra_tag 64  --exp_name ${EXP_NAME} 
+```
+2. Train 32-beam model whose teacher is 64-beam model
+```shell script
+sh scripts/dist_train_mimic.sh ${NUM_GPUS} --cfg_file cfgs/da-waymo-nus_models/pointpillar/pointpillar.yaml \
+    --batch_size ${BATCH_SIZE} --extra_tag 32 --teacher_tag 64 --pretrained_model ${64-BEAM MODEL} \
+    --pretrained_teacher_model ${64-BEAM MODEL} --mimic_weight 1 --mimic_mode roi --exp_name ${EXP_NAME} 
+```
+3. Train 16*-beam model whose teacher is 32-beam model
+```shell script
+sh scripts/dist_train_mimic.sh ${NUM_GPUS} --cfg_file cfgs/da-waymo-nus_models/pointpillar/pointpillar.yaml \
+    --batch_size ${BATCH_SIZE} --extra_tag 16^ --teacher_tag 32 --pretrained_model ${32-BEAM MODEL} \
+    --pretrained_teacher_model ${32-BEAM MODEL} --mimic_weight 1 --mimic_mode roi --exp_name ${EXP_NAME} 
+```
+Notice that you need to select the **best model** as the teacher model for the next stage. 
+
+Although our method is designed for beam-induced domain gap, we can easily
+combine our method with other general 3D UDA methods (e.g. ST3D) to solve general domain gaps.  
+```shell script
+sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file cfgs/da-waymo-nus_models/pointpillar_st3d/pointpillar_st3d.yaml \
+    --batch_size ${BATCH_SIZE} --extra_tag 16^ --pretrained_model ${16*-BEAM MODEL} --exp_name ${EXP_NAME} 
+```
+
+For evaluation, you also need to add `--extra_tag`:
+```shell script
+sh scripts/dist_test.sh ${NUM_GPUS} --extra_tag 16^ \
+    --cfg_file cfgs/da-waymo-nus_models/pointpillar_st3d/pointpillar_st3d.yaml --batch_size ${BATCH_SIZE} --ckpt ${CKPT}
+```
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
@@ -0,0 +1,30 @@
+# Installation
+
+### Requirements
+All the codes are tested in the following environment:
+* Linux (tested on Ubuntu 14.04/16.04)
+* Python 3.6+
+* PyTorch 1.1
+* CUDA 9.0
+* [`spconv v1.0 (commit 8da6f96)`](https://github.com/traveller59/spconv/tree/8da6f967fb9a054d8870c3515b1b44eca2103634) or [`spconv v1.2`](https://github.com/traveller59/spconv)
+
+
+### Install `pcdet v0.2`
+NOTE: Please re-install `pcdet v0.2` by running `python setup.py develop` even if you have already installed previous version.
+
+a. Clone this repository.
+```shell
+git clone https://github.com/weiyithu/LiDAR-Distillation.git
+```
+
+b. Install the dependent libraries as follows:
+
+* Install the dependent python libraries: 
+```
+pip install -r requirements.txt 
+```
+
+c. Install this `pcdet` library by running the following command:
+```shell
+python setup.py develop
+```
diff --git a/docs/framework.png b/docs/framework.png
diff --git a/pcdet/__init__.py b/pcdet/__init__.py
@@ -0,0 +1,24 @@
+import subprocess
+from pathlib import Path
+
+from .version import __version__
+
+__all__ = [
+    '__version__'
+]
+
+
+def get_git_commit_number():
+    if not (Path(__file__).parent / '../.git').exists():
+        return '0000000'
+
+    cmd_out = subprocess.run(['git', 'rev-parse', 'HEAD'], stdout=subprocess.PIPE)
+    git_commit_number = cmd_out.stdout.decode('utf-8')[:7]
+    return git_commit_number
+
+
+script_version = get_git_commit_number()
+
+
+if script_version not in __version__:
+    __version__ = __version__ + '+py%s' % script_version