Erik Sandström* · Yue Li* · Luc Van Gool · Martin R. Oswald
(* Equal Contribution)
Paper | Video
Point-SLAM produces accurate dense geometry and camera tracking on large-scale indoor scenes.
Due to the spatially adaptive anchoring of neural features, Point-SLAM can encode high-frequency details more effectively than NICE-SLAM which leads to superior performance in rendering, recon- struction and tracking accuracy while attaining competitive runtime and memory usage. The first row shows the feature anchor points. For NICE-SLAM we show the centers of non-empty voxels located on a regular grid, while the density of anchor points for Point-SLAM depends on depth and image gradients. The row below depicts resulting renderings showing substantial differences on areas with high- frequency textures like the vase, blinds, floor or blanket.
Point-SLAM Architecture.
Table of Contents
First you have to make sure that you have all dependencies in place. The simplest way to do so is to use anaconda.
If you are running Point-SLAM on a cluster GPU without a display, we recommend installing the headless version of Open3D which is required to evaluate the depth L1 metric of the reconstructed mesh. This requires compiling Open3D from scratch. The code is tested on Open3D 15.1 and 16.0. If compiling Open3D from scratch, remove the Open3D dependency from the env.yaml file.
You can create an anaconda environment called point-slam.
conda env create -f env.yaml
conda activate point-slamFor evaluating the F-score, download and install this library using pip.
git clone https://github.com/tfy14esa/evaluate_3d_reconstruction_lib.git
cd evaluate_3d_reconstruction_lib
pip install .To run the project inside a docker container, first create the container and then run it in interactive mode, using the available GPUs.
docker build -t point-slam:latest .
docker run -it --rm --gpus all point-slam:latest
For detailed information see the Docker documentation about the run arguments. After starting the container in a terminal window, continue with the instructions below.
Download the data as below and the data is saved into the ./datasets/Replica folder. Note that the Replica data is generated by the authors of iMAP (but hosted by the authors of NICE-SLAM). Please cite iMAP if you use the data.
bash scripts/download_replica.shTo be able to evaluate the reconstruction error, download the ground truth Replica meshes where unseen region have been culled.
bash scripts/download_cull_replica_mesh.shbash scripts/download_tum.shDATAROOT is ./datasets by default. Please change the input_folder path in the scene specific config files if stored somewhere else on your machine.
Please follow the data downloading procedure on the ScanNet website, and extract color/depth frames from the .sens file using this code.
[Directory structure of ScanNet (click to expand)]
DATAROOT is ./datasets by default. If a sequence (sceneXXXX_XX) is stored in other places, please change the input_folder path in the config file or in the command line.
DATAROOT
└── scannet
└── scene0000_00
└── frames
├── color
│ ├── 0.jpg
│ ├── 1.jpg
│ ├── ...
│ └── ...
├── depth
│ ├── 0.png
│ ├── 1.png
│ ├── ...
│ └── ...
├── intrinsic
└── pose
├── 0.txt
├── 1.txt
├── ...
└── ...
We use the following sequences:
scene0000_00
scene0025_02
scene0059_00
scene0062_00
scene0103_00
scene0106_00
scene0126_00
scene0169_00
scene0181_00
scene0207_00
For running Point-SLAM, we recommend using weights and biases for the logging. This can be turned on by setting the wandb flag to True in the configs/point_slam.yaml file. Also make sure to specify the path wandb_folder. If you don't have a wandb account, first create one. Each scene has a config folder, where the input_folder and output paths need to be specified. Below, we show some example run commands for one scene from each dataset. If you use a batch processing system (e.g SLURM), you might find our repro.sh script useful.
To run Point-SLAM on the room0 scene, run the following command.
python run.py configs/Replica/room0.yamlAfter reconstruction, the trajectory error will be evaluated and so will the mesh accuracy along with the rendering metrics.
To run Point-SLAM on the freiburg1_desk scene, run the following command.
python run.py configs/TUM_RGBD/freiburg1_desk.yamlAfter reconstruction, the trajectory error will be evaluated automatically.
To run Point-SLAM on the scene0000_00 scene, run the following command.
python run.py configs/ScanNet/scene0000_00.yamlAfter reconstruction, the trajectory error will be evaluated automatically.
If you want to develop your own system off of Point-SLAM, we provide a test function to make sure that any changes you do to the codebase produces expected results. The test_deterministic.py script runs the code for a limited set of frames and evaluates the map and trajectory against a reference to check whether they are the same. This can be useful when e.g. refactoring.
Our codebase is partially based on NICE-SLAM and we thank the authors for making this codebase publicly available. Our work would not have been possible without your great efforts!
There may be minor differences between the released codebase and the results reported in the paper. Further, we note that the GPU hardware has an influence, despite running the same seed and conda environment.
If you find our code or paper useful, please cite
@inproceedings{Sandström2023ICCV,
author = {Sandström, Erik and Li, Yue and Van Gool, Luc and R. Oswald, Martin},
title = {Point-SLAM: Dense Neural Point Cloud-based SLAM},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2023}
}Contact Erik Sandström or Yue Li for questions, comments and reporting bugs.