Point-In-Context: Understanding Point Cloud via In-Context Learning

Arxiv, 2024
Mengyuan Liu · Zhongbin Fang📧 · Xia Li📧 . Joachim M. Buhmann
Deheng Ye . Xiangtai Li . Chen Change Loy

❗❗❗ Please click the button to view our new extended journal paper : Point-In-Context: Understanding Point Cloud via In-Context Learning, 2024. Based on the previous conference paper, we further propose Point-In-Context++ (PIC++). PIC++ possesses strong performance and generalization ability in segmentation tasks. It can generalize to unseen datasets and can also perform part-specific segmentation by customizing unique prompts.

Ps: the code of PIC++ will be available soon.

Explore In-Context Learning for 3D Point Cloud Understanding

NeurIPS (Spotlight), 2023
Zhongbin Fang · Xiangtai Li · Xia Li
Joachim M. Buhmann . Chen Change Loy . Mengyuan Liu📧

⭐ Our work is the first to explore in-context learning in 3D point clouds, including task definition, benchmark, and baseline models.

🙂News

• [2024.4.19] Our extended journal paper, Point-In-Context-Segmenter, is released on Arxiv. ⭐⭐⭐
• [2023.9.24] Training and testing code is released, and PIC-Sep and PIC-Cat is released
• [2023.9.22] Our Point-In-Context is accepted at NeurIPS 2023 as a spotlight! 🎉🎉🎉
• [2023.6.16] Our Point-In-Context is released and GitHub repo is created.

⚡Features

In-context learning for 3D understanding

• The first work to explore the application of in-context learning in the 3D domain.
• A new framework for tackling multiple tasks (four tasks), which are unified into the same input-output space.
• Can improve the performance of our Point-In-Context (Sep & Cat) by selecting higher-quality prompts.

New benchmark

• A new multi-task benchmark for evaluating the capability of processing multiple tasks, including reconstruction, denoising, registration, and part segmentation.

Strong performance

• Surpasses classical models (PointNet, DGCNN, PCT, PointMAE), which are equipped with multi-task heads.
• Surpasses even task-specific models (PointNet, DGCNN, PCT) on registration when given higher-quality prompts.

✋Run

1. Requirements

PyTorch >= 1.7.0 < 1.11.0; python >= 3.7; CUDA >= 9.0; GCC >= 4.9; torchvision;

pip install -r requirements.txt

Chamfer Distance & embedding

cd ./extensions/chamfer_dist
python setup.py install --user
cd ./extensions/emd
python setup.py install --user

Pytorch3d

git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
export CUB_HOME=/usr/local/cuda/include/
FORCE_CUDA=1 python setup.py install

2. Dataset Generation

You can preprocess the dataset yourself, see the data_processing.

Alternatively, we have provided the pre-processed_datasets (recommend). Please download it and unzip it in data/

3. Training Point-In-Context

To train Point-In-Context on our dataset, run the following command:

CUDA_VISIBLE_DEVICES=<GPU> python main.py --config cfgs/PIC_Sep.yaml --exp_name exp/training/PIC_Sep

4. Evaluation

To evaluate the performance on Part Segmentation task, run the following command:

CUDA_VISIBLE_DEVICES=0 python eval_seg.py --config cfgs/PIC_Sep.yaml --exp_name exp/evaluate/PIC_Sep --ckpts exp/training/PIC_Sep/ckpt-best.pth --data_path <path_to_data>

To evaluate the performance on Reconstruction, Denoising, Registration tasks, run the following command:

CUDA_VISIBLE_DEVICES=0 python eval_cd.py --config cfgs/PIC_Sep.yaml --exp_name exp/evaluate/PIC_Sep --ckpts exp/training/PIC_Sep/ckpt-best.pth --data_path <path_to_data>

📚Released Models

Name	Params	Rec. (CD↓)	Deno. (CD↓)	Reg. (CD↓)	Part Seg. (mIOU↑)
PIC-Sep	28.9M	4.4	7.5	8.6	78.60
PIC-Cat	29.0M	4.9	6.0	14.4	79.75

The above results are reimplemented and are basically consistent with the results reported in the paper.

😃Visualization

In-context inference demo (part segmentation, denoising, registration). Our Point-In-Context is designed to perform various tasks on a given query point cloud, adapting its operations based on different prompt pairs. Notably, the PIC has the ability to accurately predict the correct point cloud, even when provided with a clean input point cloud for the denoising task.

Visualization of predictions obtained by our PIC-Sep and their corresponding targets in different point cloud tasks.

License

MIT License

Citation

If you find our work useful in your research, please consider citing:

@article{liu2024pointincontext,
  title={Point-In-Context: Understanding Point Cloud via In-Context Learning}, 
  author={Liu, Mengyuan and Fang, Zhongbin and Li, Xia and Buhmann, Joachim M and Li, Xiangtai and Loy, Chen Change},
  journal={arXiv preprint arXiv:2401.08210},
  year={2024}
}
@article{fang2024explore,
  title={Explore in-context learning for 3d point cloud understanding},
  author={Fang, Zhongbin and Li, Xiangtai and Li, Xia and Buhmann, Joachim M and Loy, Chen Change and Liu, Mengyuan},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2024}
}

Acknowledgement

This work is built upon the Point-MAE, Visual Prompt, Painter.

Thanks to the following excellent works: PointNet, DGCNN, PCT, Point-BERT, ACT, I2P-MAE, ReCon;