Loop detection freezing with SuperGlue, please fix it. Test on 2022-IlluminationInvariant. #896
Comments
|
Btw the superpoint with kdtree works with loop detection but freezes with superGlue. |
|
Currently working on a docker image to reproduce the results (working so far but minor issues to fix this week, maybe today if I have time). There is however a known issue with loading python scripts in rtabmap when there is more than one thread using Python at the same time. The results presented in the paper were generated using Related to introlab/rtabmap_ros#534 |
|
Updated README with a docker example: https://github.com/introlab/rtabmap/tree/master/archive/2022-IlluminationInvariant#docker |
|
@matlabbe, thanks |
|
Reproduced the problem on standalone, stuck on: It seems freezing when initializing superglue: To reproduce: export XAUTH=/tmp/.docker.xauth
touch $XAUTH
xauth nlist $DISPLAY | sed -e 's/^..../ffff/' | xauth -f $XAUTH nmerge -
docker run --gpus all -it --rm --ipc=host --runtime=nvidia \
--env="DISPLAY=$DISPLAY" \
--env="QT_X11_NO_MITSHM=1" \
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
--env="XAUTHORITY=$XAUTH" \
--volume="$XAUTH:$XAUTH" \
-v ~/Downloads/Illumination_invariant_databases:/workspace/databases \
rtabmap_frontiers \
rtabmap --SuperPoint/ModelPath /workspace/scripts/superpoint_v1.pt \
--SuperGlue/Path /workspace/scripts/SuperGluePretrainedNetwork/rtabmap_superglue.py \
--Kp/DetectorStrategy 11 \
--Mem/UseOdomFeatures false \
--Vis/CorNNType 6 Say "Yes" to all startup dialogs, then Open Preferences->Source, set Source->Database, scroll down and set database path to any databases in "/workspace/databases". Say "Yes" to use odometry data and "Yes" to process all data. Click ok, new database, then start. The difference between
I see two solutions:
|
|
I wonder why the latest release (https://github.com/introlab/rtabmap/releases/tag/0.20.16) can not reproduce this issue. |
|
Tested with 0.20.16 using the docker image (checking out 0.20.16 inside and rebuild it) and the same problem happens. Digging more into the issue, I tried to replicate a minimal example on how python is used inside rtabmap across threads, based on this example: // runs in a new thread
void f(PyInterpreterState* interp, const char* tname)
{
std::string code = R"PY(
from __future__ import print_function
import sys
print("TNAME: sys.xxx={}".format(getattr(sys, 'xxx', 'attribute not set')))
)PY";
code.replace(code.find("TNAME"), 5, tname);
PyThreadState* threadState = PyThreadState_New(interp);
PyEval_RestoreThread(threadState);
//sub_interpreter::thread_scope scope(interp);
PyRun_SimpleString(code.c_str());
PyThreadState_Clear(threadState);
PyThreadState_DeleteCurrent();
}
int main()
{
initialize init;
PyThreadState* mainState;
mainState = PyEval_SaveThread();
PyEval_RestoreThread(mainState);
PyRun_SimpleString(R"PY(
# set sys.xxx, it will only be reflected in t4, which runs in the context of the main interpreter
from __future__ import print_function
import sys
sys.xxx = ['abc']
print('main: setting sys.xxx={}'.format(sys.xxx))
)PY");
mainState = PyEval_SaveThread();
// Simulating here a thread using the main python interpreter
std::thread t4{f, mainState->interp, "t4(main)"};
t4.join();
PyEval_RestoreThread(mainState);
return 0;
}This works as expected. I then checked where exactly the code is freezing on superglue side, and it seems it happens when it calls Maybe related issue: huggingface/transformers#8649
|
|
At least on ROS it works. I tested by adding ros noetic in the rtabmap_frontiers docker image. Launch the docker image: export XAUTH=/tmp/.docker.xauth
touch $XAUTH
xauth nlist $DISPLAY | sed -e 's/^..../ffff/' | xauth -f $XAUTH nmerge -
docker run --gpus all -it --rm --ipc=host --runtime=nvidia \
--env="DISPLAY=$DISPLAY" \
--env="QT_X11_NO_MITSHM=1" \
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
--env="XAUTHORITY=$XAUTH" \
--volume="$XAUTH:$XAUTH" \
--network host \
--privileged \
rtabmap_frontiers Install ros noetic and build rtabmap_ros in the container, then after launching realsense D435i like in this tutorial, from inside the container: roslaunch rtabmap_ros rtabmap.launch args:="-d \
--SuperPoint/ModelPath /workspace/scripts/superpoint_v1.pt \
--SuperGlue/Path /workspace/scripts/SuperGluePretrainedNetwork/rtabmap_superglue.py \
--Reg/RepeatOnce false \
--Vis/CorGuessWinSize 0 \
--Kp/DetectorStrategy 11 \
--Vis/FeatureType 11 \
--Mem/UseOdomFeatures false \
--Vis/CorNNType 6" \
depth_topic:=/camera/aligned_depth_to_color/image_raw \
rgb_topic:=/camera/color/image_raw \
camera_info_topic:=/camera/color/camera_info \
approx_sync:=false \
wait_imu_to_init:=true \
imu_topic:=/rtabmap/imuTo make sure there is no second matching done after superglue, set |
|
Hello, |
|
|
I am trying to use Superglue via that dockerfile. I installed ROS Noetic using the standard method and everything works okay until I use catkin_make, which brings an error where it cannot find empy. I have found there is an issue when multiple interpreters are installed, but everything seems to be pointing to the missing dependency. The problem is that the environment uses conda, and it cannot install empy due to a conflict with other packages installed in the image. How did you manage to install ROS and rtabmap_ros? |
|
I am doing that inside docker, but there are two versions of python now. In the last-mentioned issue, you do not seem to have an issue with that, ¿ Are you using catkin_make or catkin build to build in noetic? If I try to use pip instead of conda or the pytorch image, the problem is that installing pytorch with pip does not include c++11 abi, so rtabmap is unable to build with the "undefined reference to" error. If you install libtorch with c++11 abi, then pytorch is not installed, and if you install both, they clash and python segfaults. Building using conda results in the error I mentioned in my previous comment. To clarify, I am using the commands in Step 1 to 3 inside docker. |
|
I'm testing SuperPoint/SuperGlue on freiburg2_pioneer_slam3 dataset. I can see detections, but no matches. This is how I test it: |
|
Superglue should work for loop closure detection, not odometry (unless you choose F2F odometry). Here a small comparison between different approaches.
For that dataset, it seems there is 3 sec missing while the robot was rotating around 1671st frame. I fixed the code to make Looking at the results, the lack of loop closures for GFTT is more related to binary descriptors, not that it is not Superglue. Here is a difference of matching superpoint features with and without superglue respectively: |
|
@GVMCOTESA what is your base image? Is it the one from nvidia like in the frontiers dockerfile? |
|
I did it with native installed libraries. For docker, you may use frontiers dockerfile. If you want to go ROS, I also recently created an image for rtabmap_ros. |
|
Yes, It is the frontiers one. I will try the new image, thank you. |
With the matching tool, is the rtabmap-databaseviewer intended? @matlabbe Are you seeing this too? |
|
The matching tool is not |
|
It worked, I was not aware that rtabmap_ros must be in its own container separated from the rest, now I have a separate container for simulating the robot. I have to say, there must be something misconfigured on my part, the map rotates wildly each time a loop closure is detected, and it doesn't seem to be converging. Is there a set of calibration parameters that can help reduce this? |
|
Can you share the database? |
|
Regarding the app freezing on superglue initialization (#896 (comment)). Here is a gdb log when it happens: I think the problem is that:
The difference between standalone and ros is that for the later the python interpreter is running in same thread than the one pytorch is running onto. Questions:
|
|
Fixed in f1cd819 |
I am glad we could skip the docker for now cause I found it has some performance issues related to the docker itself. |
|
@cdb0y511 I too note that building pytorch from source avoids the ""undefined reference" errors. However, I also note a severe (>>10x) peformance loss with this pytorch compiled from sources. Question: Did you also recording a loss in inference speed when building pytorch from source? Did you overcome it in some way? |
…mSize opencv asserts in debug build (876)
|
sorry to bother everyone, Superglue can run rtabslam without docker ? as follow step is correct?
i get this : QAQ |
|
See #1221 (comment) |
|
i ran this as follow : up and down just different with add/ not add --Vis/CorNNType 6 roslaunch rtabmap_launch rtabmap.launch args:="-d and i get [ERROR] (2024-02-06 00:05:32.077) PyMatcher.cpp:63::PyMatcher() Module "demo_superglue" could not be imported! (File="/home/lun/rtabmap/archive/2022-IlluminationInvariant/scripts/SuperGluePretrainedNetwork/demo_superglue.py") File "/home/lun/rtabmap/archive/2022-IlluminationInvariant/scripts/SuperGluePretrainedNetwork/demo_superglue.py", line 51, in File "/home/lun/.local/lib/python3.8/site-packages/torch/init.py", line 237, in ImportError: /home/lun/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZNK5torch3jit5Graph8toStringEb error but i ran roslaunch rtabmap_launch rtabmap.launch args:="-d i get Parameters.cpp:1149::parseArguments() Parameter migration from "SuperGlue/Path" to "PyMatcher/Path" (value=/home/lun/rtabmap/archive/2022-IlluminationInvariant/scripts/SuperGluePretrainedNetwork/demo_superglue.py). is it correct ? |
|
If you don't use So you get this error when using superglue: Has pytorch been built from source? Uninstall the one installed with pip if you rebuilt pytorch from source. |
|
i follow the website: this is not build pytorch, right ? |
These are the already compiled pytorch libraries. But the "Download here (cxx11 ABI)" worked for me without having to compile pytorch from the sources. I unpacked them into my home directory "/home/he/projects/libtorch". I added the library path by adding this line to the end of my ~/.bashrc: And build my ros2 humble workspace specifying the cmake directory of the library. |
|
@HelmutE89 thanks, i still have same problem QQ |
|
In practice, you can get already great performance with SuperPoint and standard KNN. However, SuperGlue would give more matches in general than KNN and could resolve very large point of view differences (great for loop closure detection). |
@matlabbe Thank you for your response. |
…some elemSize opencv asserts in debug build (876)" This reverts commit f7e4b38.
Hi, @matlabbe.
It happens with the latest git,
How to reproduce:
with the latest git, and I test with the
https://github.com/introlab/rtabmap/blob/71a28bb570e26f6bbcb78bd7a95ea75c24a4d4d8/archive/2022-IlluminationInvariant/README.md
first DB loc_190321-165128.db, source load from DB, use odometer in the DB, and Kp/DetectorStrategy, Vis/FeatureType should be set to 11 (SuperPoint), and Vis/CorNNType is set to 6 (SuperGlue). It freezes in a few detections like below
The odometer seems to continue, but loop detection stops. I can press stop, but can not close the DB, need to force kill it.
I return to the latest release(https://github.com/introlab/rtabmap/releases/tag/0.20.16)
Everything is OK, superGlue works.
Btw, the superGlue works on the odometer, but freezes with the loop detection.
Ubuntu:20.04, test on both Cuda 11.6 and 11.7. libtorch 1.8.2.
I think there are some issues with loop detection with the recent commit.
I hope you can fix it soon.
Thanks,@matlabbe
The text was updated successfully, but these errors were encountered: