rgb-depth alignment is too slow even built with CUDA

[ProtossDragoon]

---

Required Info
Camera Model	D435i, D435
Firmware Version	05.12.14
Operating System & Version	Linux 18 Jetson OS
Kernel Version (Linux Only)	4.9
Platform	NVIDIA Jetson AGX Xavier
SDK Version	2.48.0
Language	python3 & realsense2_camera (ROS)
Segment	Robot

Issue Description

Hi,
I'm using two realsense d435 devices in my project and I have noticed that the execution time of the rgb-depth alignment on Jetson Xavier AGX has very long latency (about 1.5~2 seconds latency exists between alignment processed result visualization and real-world happening), and pumping CPU a lot. While on the same machine, without rgb-d alignment code, it runs much faster, no latency.
Given #2670, maybe it's not optimal performance on my Jetson Xavier AGX.

+I already used "build with cuda" cmake flag.

Thanks!

[MartyG-RealSense]

Hi @ProtossDragoon May I first confirm whether your alignment lag is occurring in the librealsense SDK or in ROS, as you have RViz in the background of your image. Thanks!

[ProtossDragoon]

Hi @ProtossDragoon May I first confirm whether your alignment lag is occurring in the librealsense SDK or in ROS, as you have RViz in the background of your image. Thanks!

Thank you for your reply, @MartyG-RealSense

• case 1 : SDK, realsense-viewer, camera1, no latency, rgb-d calibrated 3D visualization
• case 2 : SDK, realsense-viewer, camera2, no latency, rgb-d calibrated 3D visualization
• ros case 1 : librealsense2-ros, rviz, camera1 + camera2, no latency, no rgb-d calibration
• ros case 2 : librealsense2-ros, rviz, camera1 + camera2, latency & frame drop, rgb-d calibrated

additional information : my d435 launchfile (ros case 2)

<launch>
    <arg name="serial_no"           default=""/>
    <arg name="usb_port_id"         default=""/>
    <arg name="device_type"         default=""/>
    <arg name="json_file_path"      default=""/>
    <arg name="camera"              default="d435_bottom"/>
    <arg name="tf_prefix"           default="$(arg camera)"/>

    <arg name="external_manager"    default="false"/>
    <arg name="manager"             default="realsense_manager"/>
    <arg name="output"              default="screen"/>

    <!-- sensor : fisheye camera -->
    <arg name="fisheye_width"       default="960"/>
    <arg name="fisheye_height"      default="540"/>
    <arg name="enable_fisheye"      default="false"/>
    <arg name="fisheye_fps"         default="15"/>

    <!-- sensor : depth camera -->
    <arg name="depth_width"         default="640"/>
    <arg name="depth_height"        default="480"/>
    <arg name="enable_depth"        default="true"/>
    <arg name="depth_fps"           default="15"/>

    <!-- sensor : infrared camera -->
    <arg name="infra_width"         default="640"/>
    <arg name="infra_height"        default="480"/>
    <arg name="enable_infra"        default="true"/>
    <arg name="enable_infra1"       default="false"/>
    <arg name="enable_infra2"       default="false"/>
    <arg name="infra_rgb"           default="false"/>
    <arg name="infra_fps"           default="30"/>

    <!-- sensor : rgb camera -->
    <arg name="color_width"         default="640"/>
    <arg name="color_height"        default="480"/>
    <arg name="enable_color"        default="true"/>
    <arg name="color_fps"           default="15"/>

    <!-- sensor : inertial and gyro -->  
    <arg name="gyro_fps"            default="400"/>
    <arg name="accel_fps"           default="250"/>
    <arg name="enable_gyro"         default="false"/>
    <arg name="enable_accel"        default="false"/>

    <!-- advanced -->
    <arg name="enable_pointcloud"         default="false"/> <!-- deprecated argument, do not use. -->
    <arg name="enable_sync"               default="true"/> <!-- gathers closest frames of different sensors, infra red, color and depth, to be sent with the same timetag. This happens automatically when such filters as pointcloud are enabled. -->
    <arg name="align_depth"               default="true"/> <!-- will publish additional topics with the all the images aligned to the depth image. -->
    <arg name="publish_tf"                default="true"/>
    <arg name="tf_publish_rate"           default="0"/>

    <arg name="filters"                   default="pointcloud or none"/> <!-- pointcloud: will add a pointcloud topic /camera/depth/color/points. -->
    <arg name="clip_distance"             default="6"/> <!-- remove from the depth image all values above a given value (meters). Disable by giving negative value (default) -->
    <arg name="pointcloud_texture_stream" default="RS2_STREAM_COLOR"/>
    <arg name="pointcloud_texture_index"  default="0"/>
    <arg name="allow_no_texture_points"   default="false"/>

    <arg name="linear_accel_cov"          default="0.01"/>
    <arg name="initial_reset"             default="false"/>
    <arg name="unite_imu_method"          default=""/>
    <arg name="topic_odom_in"             default="odom_in"/>
    <arg name="calib_odom_file"           default=""/>
    <arg name="publish_odom_tf"           default="true"/>

    <group ns="$(arg camera)">
        <include file="$(find realsense2_camera)/launch/includes/nodelet.launch.xml">
            <arg name="tf_prefix"                value="$(arg tf_prefix)"/>
            <arg name="external_manager"         value="$(arg external_manager)"/>
            <arg name="manager"                  value="$(arg manager)"/>
            <arg name="output"                   value="$(arg output)"/>
            <arg name="serial_no"                value="$(arg serial_no)"/>
            <arg name="usb_port_id"              value="$(arg usb_port_id)"/>
            <arg name="device_type"              value="$(arg device_type)"/>
            <arg name="json_file_path"           value="$(arg json_file_path)"/>

            <arg name="enable_pointcloud"        value="$(arg enable_pointcloud)"/>
            <arg name="pointcloud_texture_stream" value="$(arg pointcloud_texture_stream)"/>
            <arg name="pointcloud_texture_index"  value="$(arg pointcloud_texture_index)"/>
            <arg name="enable_sync"              value="$(arg enable_sync)"/>
            <arg name="align_depth"              value="$(arg align_depth)"/>

            <arg name="fisheye_width"            value="$(arg fisheye_width)"/>
            <arg name="fisheye_height"           value="$(arg fisheye_height)"/>
            <arg name="enable_fisheye"           value="$(arg enable_fisheye)"/>

            <arg name="depth_width"              value="$(arg depth_width)"/>
            <arg name="depth_height"             value="$(arg depth_height)"/>
            <arg name="enable_depth"             value="$(arg enable_depth)"/>

            <arg name="color_width"              value="$(arg color_width)"/>
            <arg name="color_height"             value="$(arg color_height)"/>
            <arg name="enable_color"             value="$(arg enable_color)"/>

            <arg name="infra_width"              value="$(arg infra_width)"/>
            <arg name="infra_height"             value="$(arg infra_height)"/>
            <arg name="enable_infra"             value="$(arg enable_infra)"/>
            <arg name="enable_infra1"            value="$(arg enable_infra1)"/>
            <arg name="enable_infra2"            value="$(arg enable_infra2)"/>
            <arg name="infra_rgb"                value="$(arg infra_rgb)"/>

            <arg name="fisheye_fps"              value="$(arg fisheye_fps)"/>
            <arg name="depth_fps"                value="$(arg depth_fps)"/>
            <arg name="infra_fps"                value="$(arg infra_fps)"/>
            <arg name="color_fps"                value="$(arg color_fps)"/>
            <arg name="gyro_fps"                 value="$(arg gyro_fps)"/>
            <arg name="accel_fps"                value="$(arg accel_fps)"/>
            <arg name="enable_gyro"              value="$(arg enable_gyro)"/>
            <arg name="enable_accel"             value="$(arg enable_accel)"/>

            <arg name="publish_tf"               value="$(arg publish_tf)"/>
            <arg name="tf_publish_rate"          value="$(arg tf_publish_rate)"/>

            <arg name="filters"                  value="$(arg filters)"/>
            <arg name="clip_distance"            value="$(arg clip_distance)"/>
            <arg name="linear_accel_cov"         value="$(arg linear_accel_cov)"/>
            <arg name="initial_reset"            value="$(arg initial_reset)"/>
            <arg name="unite_imu_method"         value="$(arg unite_imu_method)"/>
            <arg name="topic_odom_in"            value="$(arg topic_odom_in)"/>
            <arg name="calib_odom_file"          value="$(arg calib_odom_file)"/>
            <arg name="publish_odom_tf"          value="$(arg publish_odom_tf)"/>
            <arg name="allow_no_texture_points"  value="$(arg allow_no_texture_points)"/>
        </include>
    </group>

</launch>

[MartyG-RealSense]

Thanks very much for the information. There has been a ROS / Jetson case to your one a week ago where, if pointcloud was enabled OR align_depth was enabled then it performed fine. But if both pointcloud AND align_depth were enabled in the same launch then it strongly lagged.

IntelRealSense/realsense-ros#1995

That RealSense user confirmed the behavior on Jetson TX2, NX and AGX but found that the problem did not occur on an Intel NUC computer.

Can you confirm whether you are using a point cloud in ROS or not, please?

I note your mention of CMake. Did you install the SDK on your Jetson using the native backend method described in Intel's Jetson installation instructions?

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md#building-from-source-using-native-backend

[ProtossDragoon]

• original case (no pointcloud, no align depth)
• pointcloud only
• align depth only
• pointcloud AND align depth

original case (no pointcloud, no align depth)

pointcloud only

align depth only

pointcloud AND align depth

CPU usage, latency and FPS drop was very high in all trials, except the first one.

Yes, maybe I installed native backend (./scripts/patch-realsense-ubuntu-L4T.sh, -DFORCE_RSUSB_BACKEND=false), not RSUSB backend.

Full install command for Realsense SDK in my case :

# Register the server's public key
sudo apt-key adv --keyserver keys.gnupg.net --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE

# Add the server to the list of repositories (ubuntu 18)
sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo bionic main" -u

# install SDK
sudo apt-get install librealsense2-utils
sudo apt-get install librealsense2-dev

cd ~/dev
git clone https://github.com/IntelRealSense/librealsense.git
cd librealsense
git checkout master

sudo apt-get install git libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev -y
./scripts/patch-realsense-ubuntu-L4T.sh

mkdir build && cd build

cmake .. -DBUILD_EXAMPLES=true \
-DCMAKE_BUILD_TYPE=release \
-DFORCE_RSUSB_BACKEND=false \
-DBUILD_WITH_CUDA=true

make -j$(($(nproc)-1))

sudo make install

I've been using mains-powered USB hub (one with an independent power supply) too.

On my Jetson env
jsetson sdk 4.5.1 -> install ros -> install realsense sdk with cuda -> remove opencv4 -> install opencv4 with cuda -> install realsesne-ros (realsense2_camera) and now here.
Do you recommend remove all of environment and reinstall everything?

[MartyG-RealSense]

Sometimes a complete wipe and reinstall of the whole system (Ubuntu, ROS, RealSense ROS wrapper) can fix an installation where you have tried everything else and it still isn't working, because the total reinstallation can remove a hidden problem that you would not otherwise have found. It is just frustrating for something to be fixed without knowing what the cause was, as you cannot take account of it in future installations.

[ProtossDragoon]

Okay, I understood that could be more progress in my case, not from the current limitation of realsense SDK or realsesne2-camera ros package. I'll close this issue and try to reinstall my all system to solve this problem. Thank you very much for support @MartyG-RealSense ! Could I ask more information if there's no advance after that?

[MartyG-RealSense]

Thanks very much @ProtossDragoon for the update - yes, you are welcome to post further questions. Good luck!