Empty pointcloud data when accelerate_gpu_with_glsl = True #3206
Comments
|
Hi @edgarcamilocamacho In rare cases on some computers, a pointcloud will disappear when GLSL is enabled. There is no way to predict in advance which computers will be unable to use GLSL, unfortunately. If you are using a Jetson then the usual way to accelerate a pointcloud will be to enable the librealsense SDK's CUDA support. This can be done by installing the Jetson version of the librealsense packages on the installation_jetson.md instruction page or by building from source code with the flag -DBUILD_WITH_CUDA=ON included in the CMake build instruction. |
|
So if I install
I should get both of them running with GPU, right? There is a way to identify that it's actually using the GPU? |
|
Following that idea I this in my Jetson container (the sumarized steps from the links I sent before): # apt-key adv --keyserver keyserver.ubuntu.com --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key # adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE
# add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u
# apt-get install -y librealsense2-utils librealsense2-dev ros-humble-realsense2-*After that I get these versions installed: I got I could install |
|
Wait... seeing it closely, it installed
|
|
I am not involved in the compilation and publishing of packages, so do not have advice to provide about their contents. As a general rule, assume that if installation is not being performed with the Jetson packages on the https://github.com/JetsonHacksNano/installLibrealsense |
|
Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please Thanks! |
|
These are the steps I'm following in my dockerfile (in the Jetson). I took the steps from the scripts you told me. # Install librealsense
RUN cd /tmp && \
wget https://github.com/IntelRealSense/librealsense/archive/refs/tags/v2.56.1.tar.gz && \
tar -xzf v2.56.1.tar.gz && \
cd librealsense-2.56.1 && \
mkdir build && \
cd build && \
cmake .. \
-DBUILD_PYTHON_BINDINGS:bool=true \
-DPYTHON_EXECUTABLE=/usr/bin/python3 \
-DBUILD_EXAMPLES=true \
-DCMAKE_BUILD_TYPE=release \
-DFORCE_LIBUVC=true \
-DFORCE_RSUSB_BACKEND=false \
-DBUILD_WITH_CUDA:bool=true \
&& \
make -j 10 && \
make install && \
cd ../.. && \
rm -r v2.56.1.tar.gz librealsense-2.56.1/
# Install realsense-ros
RUN source /setup_env.bash && \
mkdir -p /robot/additional_ros_packages/src && \
cd /robot/additional_ros_packages/src && \
git clone -b 4.56.1 https://github.com/IntelRealSense/realsense-ros.git && \
cd /robot/additional_ros_packages && \
colcon build --merge-install --cmake-args '-DBUILD_ACCELERATE_GPU_WITH_GLSL=ON' && \
rm -rf src build logAnd I still have the same problem. I attach the complete dockerfile. In addition this is the docker-compose configuration: cameras:
# common
network_mode: host
pid: host
privileged: true
runtime: nvidia
restart: unless-stopped
# specific
image: gary.core.cameras.cameras:jetsonorin.4.3.beta
environment:
# display
- XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR}
- DISPLAY=:0
env_file:
# common
- ./config/env/env
volumes:
# common
- ./config:/robot/config
- ./generic_persistent_data:/robot/generic_persistent_data
- ./component_persistent_data/cameras:/robot/component_persistent_data
# devices
- /dev:/dev
- /run/udev:/run/udev:ro
# display
- ${XDG_RUNTIME_DIR}/gdm/Xauthority:/root/.Xauthority
- /tmp/.X11-unix:/tmp/.X11-unix:rw |
|
My knowledge of Docker is admittedly limited, so there is not much that I can add to the previous advice, unfortunately. I would recommend removing the -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON pointcloud acceleration parameter, since the pointcloud will already be being accelerated by -DBUILD_WITH_CUDA:bool=true when this Dockerfile is used on Jetson. Havibg said that, there have been a small number of past cases where problems disappear when not setting -DBUILD_WITH_CUDA to true. So you could try it either with -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON or with DBUILD_WITH_CUDA:bool=true but not have both set to true at the same time. |
|
Hi @edgarcamilocamacho Do you require further assistance with this case, please? Thanks! |
|
Hello @MartyG-RealSense , thanks for the fast response and sorry for the slow testing, we have several things on the table, but for sure next week we'll build the container again and test. |
|
It's no problem at all. I look forward to your next report. Good luck! |
|
Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please? Thanks! |
|
Hi @MartyG-RealSense,
I've the following findings (Using Setup 1 hardware):
Long story: Scenario1:
sudo apt-get install ros-humble-realsense2-camera ros-humble-realsense2-camera-msgs ros-humble-realsense2-description
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info
lsof -p $(pgrep -f /opt/ros/humble/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.so
realsense 76448 zzzzz mem REG 259,2 28264 807521 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so realsense 76448 zzzzz mem REG 259,2 27392 807519 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so realsense 76448 zzzzz mem REG 259,2 6581504 807412 /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 realsense 76448 zzzzz mem REG 259,2 1151296 813087 /opt/ros/humble/lib/librealsense2_camera.so realsense 76448 zzzzz mem REG 259,2 14344 807517 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so Scenario2:
sudo apt-get install librealsense2-utils librealsense2-dev librealsense2-gl-dev Note2: I've not patched the kernel
git clone https://github.com/IntelRealSense/realsense-ros.git cd realsense-ros colcon build --cmake-args -DBUILD_ACCELERATE_GPU_WITH_GLSL=1 #overlay this build on top of humble release source $(pwd)/install/setup.bash
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info
lsof -p $(pgrep -f $(pwd)/install/realsense2_camera/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.s Output: realsense 80059 zzzzz mem REG 259,2 79448 138474 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so realsense 80059 zzzzz mem REG 259,2 60784 138463 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so realsense 80059 zzzzz mem REG 259,2 6581504 807412 /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 realsense 80059 zzzzz mem REG 259,2 1450648 3030330 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 realsense 80059 zzzzz mem REG 259,2 2318512 138826 /home/zzzzz/realsense-ros/install/realsense2_camera/lib/librealsense2_camera.so realsense 80059 zzzzz mem REG 259,2 24904 138475 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so Note the following: /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 When I run the node with accelerate_gpu_with_glsl:=True point cloud is not generated. Otherwise it is generated without problems because ros version of librealsense2 is running as expected. Scenario 3:
sudo apt-get remove ros-humble-realsense2-camera ros-humble-realsense2-camera-msgs ros-humble-realsense2-description At this point the following should return empty results: ls /opt/ros/humble/lib/aarch64-linux-gnu/librealsense* ls /opt/ros/humble/lib/librealsense*
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info IMPORTANT: Now we don't get point cloud data in all cases. (So there must be a problem with the libraries built by librealsense, ros version is working, lets deep dive in Scenario4)
lsof -p $(pgrep -f $(pwd)/install/realsense2_camera/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.s Output: realsense 80962 zzzzz mem REG 259,2 79448 138474 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so realsense 80962 zzzzz mem REG 259,2 60784 138463 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so realsense 80962 zzzzz mem REG 259,2 7855232 3015185 /usr/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 realsense 80962 zzzzz mem REG 259,2 1450648 3030330 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 realsense 80962 zzzzz mem REG 259,2 2318512 138826 /home/zzzzz/realsense-ros/install/realsense2_camera/lib/librealsense2_camera.so realsense 80962 zzzzz mem REG 259,2 24904 138475 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so As expected, this time, both libraries are linked correctly and against the ones built by librealsense /usr/lib/aarch64-linux-gnu/librealsense2.so.2.55.1 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1 Scenario4:
sudo apt-get remove librealsense2-dev librealsense2-gl-dev librealsense2 librealsense2-gl librealsense2-utils librealsense2-udev-rules #Dependencies are already installed, skipping... cd ~ git clone https://github.com/IntelRealSense/librealsense.git cd librealsense #Switch to v2.55.1 tag git checkout v2.55.1 #Install USB udev rules sudo ./scripts/setup_udev_rules.sh #Skipping kernel patch we'll be using FORCE_RSUSB_BACKEND mkdir build cd build cmake ../ -DBUILD_EXAMPLES=false -DBUILD_GRAPHICAL_EXAMPLES=false -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release sudo make uninstall make clean make sudo make install
cd ~/realsense-ros colcon clean workspace -y colcon build #overlay this build on top of humble release source $(pwd)/install/setup.bash
ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info Point cloud generation works, however on the ros logs I see the following warning: (ros/realsense builds do not print this warning) WARNING (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0 [INFO] [1728932468.838272790] [camera_d435_front]: RealSense Node Is Up! WARNING (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0 |
|
Update: cmake ../ -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=true We're back to square 1. Point cloud can now be generated with CPU but not with GPU. |
|
control_transfer returned warnings can be safely disregarded if the pointcloud appears to be performing normally. |
…t provided. See: IntelRealSense/realsense-ros#3206 When called through realsense-ros package, depth mapping texture is not provided, thus point cloud is not generated. A new shader is generated to handle the case.
|
I've found the reason and implemented a solution, I'll test and let you know about the results. The problem occurs in "librealsense" library if point cloud generation is opengl accelerated and there's no depth_texture provided. depth_to_points function returns empty point data as this operation is deferred to get_texture_map function. However, since there's no depth texture provided, get_texture_map is never called and we get empty point cloud. I've created another shader which just does point cloud extraction but not depth texture mapping and executed it within depth_to_points function if texture map is not available. (See also resdocta/librealsense@81b78a1 for details) |
|
Thanks so much for sharing the details of your solution. I look forward to hearing about your test results. :) |
Hello
Issue Description
When I set
accelerate_gpu_with_glsl = True(at the begining of the execution, not dinamically), I get empty data pointcloud like:But with
accelerate_gpu_with_glsl = Falseit works perfectly.Maybe I'm setting something wrong, I attach the list of ros params:
rs-params.txt
The text was updated successfully, but these errors were encountered: