FPS drop and stream latency on Jetson ROS

[nkorenicFR]

---

Required Info
Camera Model	D435i
Firmware Version	5.12.07.100
Operating System & Version	L4T-32.4.4
Kernel Version (Linux Only)	4.9.140-tegra
Platform	Jetson Xavier AGX
SDK Version	2.49.0
Language	ROS melodic + realsense ROS wrapper 2.3.1
Segment	Robot

Issue Description

Stream FPS drop and latencies (hundreds of microseconds up to a few seconds) occur in ROS Jetson when pointcloud or align depth is enabled.
The issue can be reproduced by running roslaunch realsense2_camera rs_camera.launch. Below are given some cases which reproduce the issue. FPS drop was measured with rostopic hz <topic> and latencies are observed by displaying stream with rosurun rviz rviz and rosrun image_view image_view image:=<image topic>.

---

enable pointcloud	align depth	filters	RGB Hz	depth Hz	Comment
false	false	-	30	30	-
true	false	-	10	24	issue present
true	false	colorizer	9	22	issue present
true	false	decimation	30	30	-
true	false	disparity	17	30	issue present
true	true	-	20	40	issue present
true	true	colorizer	16	27	issue present
true	true	decimation	12	34	issue present
true	true	disparity	11	45	issue present
false	true	-	30	30	-
false	true	colorizer	24	36	issue present
false	true	decimation	30	30	-
false	true	disparity	17	40	issue present

LibRealSense

Realsense is built from source with CUDA support cmake .. -DBUILD_EXAMPLES=true -DCMAKE_BUILD_TYPE=release -DFORCE_RSUSB_BACKEND=false -DBUILD_WITH_CUDA=true. Native kernel drivers are used and the L4T kernel is patched with ./scripts/patch-realsense-ubuntu-L4T.sh.

ROS launch

Lunch enables two camera streams: RGB (1280x720, RGB8, 30FPS) and, depth (848x480, Z16, 30FPS)
The launch is built and run with librealsense v2.49.0.

CPU load

On the issue, the system has increased CPU consumption (sometimes one core is fully loaded). Our Jetson AGX system is configured to full power mode (30W) with all cores enabled.

CPU load when pointcloud and align depth used

CPU load when pointcloud and align depth not used

Reproducing in RealSense Viewer

In the viewer, similar could be reproduced when GLSL processing and rendering were disabled. Is it possible that ROS uses LibRealSense without GLSL support?

Similar issues

A similar issue has already been reported #9519, but it is closed without an adequate solution.

[MartyG-RealSense]

Hi @nkorenicFR The RealSense ROS wrapper version used should ideally be matched as closely as possible with the librealsense SDK version listed for that particular wrapper in its release notes. The librealsense version to match with wrapper 2.31 is 2.48.0. There is not a wrapper for matching with 2.49.0, and the next wrapper version should be for upcoming librealsense version 2.50.0.

In regard to GLSL, it is not enabled in RealSense applications by default and has to be deliberately incorporated into programs, like it is in the RealSense Viewer.

[nkorenicFR]

Hi @MartyG-RealSense ,

Thank you for your fast response. I will try with librealsense 2.48.0.
Regarding GLSL in ROS wrapper, i guess that realsense2-gl module (#3654) is not used, as i could not locate rs2::gl::pointcloud or rs2::gl::colorize. Could we expect better ROS performances (FPS, stream latency) by adding GLSL to the RealSense ROS wrapper?

[MartyG-RealSense]

I am not personally aware of anyone who has tried adjusting the rs2:: instructions in the ROS wrapper to convert them to rs2::gl:: instructions. The key lines in the wrapper's base_realsense_node.cpp to look at for each type of rs2:: operation are likely to be the ones below (I say 'likely' as I am not involved in the wrapper's development).

Pointcloud
https://github.com/IntelRealSense/realsense-ros/blob/development/realsense2_camera/src/base_realsense_node.cpp#L1273

Align
https://github.com/IntelRealSense/realsense-ros/blob/development/realsense2_camera/src/base_realsense_node.cpp#L1253

Colorizer
https://github.com/IntelRealSense/realsense-ros/blob/development/realsense2_camera/src/base_realsense_node.cpp#L1258

[nkorenicFR]

Same behavior with RealSense SDK 2.48.0, the issue is still present. Any idea what else could we try?

Does Intel have any benchmark results on ROS jetson that can be used for comparison? Is this expected behavior on the Jetson platform?

[MartyG-RealSense]

Hi @nkorenicFR Dips in performance when enabling pointcloud or align in the RealSense ROS wrapper with Jetson boards is an issue that has previously been reported by a small number of Jetson users, as mentioned in #9519 (comment)

There are not ROS benchmark results available from Intel that compare pointcloud and align performance on Jetson with non-Jetson systems. Intel did publish a CPU performance table of alignment with CUDA versus non-CUDA when CUDA support was first introduced in the librealsense SDK though. That table can be found at the link below.

#2670