Jetson Orin with Jetpack 6.0

[Yaccobv2]

Required Info
Camera Model	D455
Firmware Version	5.15.1
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)	5.15
Platform	Jetson AGX Orin
SDK Version	2.54.2
Jetpack	6.0
Jetson L4T	36.2

Issue Description

Hi, I am trying to get the Realsense D455 camera up and running using newest Jetpack 6.0. I know that L4T 36.2 is not supported, so I can't rely on kernel modules using the patch-realsense-ubuntu-L4T.sh script. I tried building the SDK using the -DFORCE_RSUSB_BACKEND=true flag to bypass the kernel modules and I did it so successfully. Now I am able to run the realsense-viewer and display the data, but something is not working properly because every time I run the realsense-viewer everything gets laggy and after a few seconds Jetson literally freezes. When I unplug the camera, everything gets back to normal.

Dmesg logs:

[11230.388791] usb 2-3.1: USB disconnect, device number 23
[11232.809292] usb 2-3.1: new SuperSpeed USB device number 24 using tegra-xusb
[11232.842183] uvcvideo 2-3.1:1.1: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[11232.842650] usb 2-3.1: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455  (8086:0b5c)
[11232.860298] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/bus@0/3610000.usb/usb2/2-3/2-3.1/2-3.1:1.0/input/input1214
[11232.862540] uvcvideo 2-3.1:1.4: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[11232.862562] usb 2-3.1: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455  (8086:0b5c)
[11232.885375] hid-generic 0003:8086:0B5C.001E: device has no listeners, quitting

When I run the realsense-viewer app dmesg logs are spammed with:

[11450.904794] uvcvideo 2-3.1:1.1: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[11450.905285] usb 2-3.1: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 455  (8086:0b5c)
[11450.921954] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/bus@0/3610000.usb/usb2/2-3/2-3.1/2-3.1:1.0/input/input1318

and console displays warning:

0/01 14:40:11,739 WARNING [281473731156032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11

Question

Is it somehow possible to use the Realsense camera under Jetson L4T 36.2 using kernel modules or RUSB backend?

[martinerk0]

I have installed D455 on Jetpack 6 DP Orin nano with https://github.com/IntelRealSense/librealsense/blob/master/scripts/libuvc_installation.sh, although for some reason it only works with sudo realsense-viewer so maybe I have to fix permissions or udev rules somehow.

[MartyG-RealSense]

Thanks so much @martinerk0 for your advice to @Yaccobv2

Hi @Yaccobv2 If a new JetPack version is not yet supported by the L4T kernel patch script then the choices for getting librealsense working with an Nvidia Jetson device are usually to (a) do not apply the L4T patch script and use the SDK unpatched, or (b) build from source code with CMake with -DFORCE_RSUSB_BACKEND=TRUE.

The libuvc procedure referred to by @martinerk0 is the predecessor of RSUSB and has a similar kernel bypass effect to using that build flag.

If either RSUSB or libuvc are used then as well as bypassing the kernel, it should also be easier to use llibrealsense with Ubuntu 22.04 as RSUSB / libuvc is not dependent on a particular Linux version.

[martinerk0]

I just reinstalled SDK using
sudo ./libuvc_installation.sh -DBUILD_WITH_CUDA=true

but when I run realsense-viewer without sudo I get:

   10/01 16:24:35,692 INFO [281473192276000] (backend-v4l2.cpp:657) Enumerating UVC video0 realpath=/sys/devices/platform/bus@0/3610000.usb/usb2/2-1/2-1.2/2-1.2:1.0/video4linux/video0
   10/01 16:24:35,693 INFO [281473192276000] (backend-v4l2.cpp:657) Enumerating UVC video1 realpath=/sys/devices/platform/bus@0/3610000.usb/usb2/2-1/2-1.2/2-1.2:1.0/video4linux/video1
   10/01 16:24:35,693 INFO [281473192276000] (backend-v4l2.cpp:657) Enumerating UVC video2 realpath=/sys/devices/platform/bus@0/3610000.usb/usb2/2-1/2-1.2/2-1.2:1.0/video4linux/video2
   10/01 16:24:35,693 INFO [281473192276000] (backend-v4l2.cpp:657) Enumerating UVC video3 realpath=/sys/devices/platform/bus@0/3610000.usb/usb2/2-1/2-1.2/2-1.2:1.0/video4linux/video3
   10/01 16:24:35,694 INFO [281473192276000] (backend-v4l2.cpp:657) Enumerating UVC video4 realpath=/sys/devices/platform/bus@0/3610000.usb/usb2/2-1/2-1.2/2-1.2:1.3/video4linux/video4
   10/01 16:24:35,694 INFO [281473192276000] (backend-v4l2.cpp:657) Enumerating UVC video5 realpath=/sys/devices/platform/bus@0/3610000.usb/usb2/2-1/2-1.2/2-1.2:1.3/video4linux/video5
   10/01 16:24:35,705 WARNING [281473192276000] (d400-factory.cpp:1195) DS5 group_devices is empty.

with sudo it works fine. @MartyG-RealSense any ideas? There are few old issues with FW/SDK mismatch #12047

Same with rs-enumerate-devices, without sudo nothing, with sudo it works.

[martinerk0]

Okay I also checked this issue and when I checked installed packages there was also: ros-humble-librealsense2 installed through apt ☠️

So using:
sudo apt remove ros-humble-librealsense2
removed possible conflicts and now it works 😃

@Yaccobv2 did the libuvc_installation.sh work for you? I remember few years ago on Xavier NX trying to do kernel patching and I never got it working, but with libuvc script it worked fine.

[MartyG-RealSense]

@martinerk0 I'm pleased to hear that you achieved a non-sudo solution. :)

[Yaccobv2]

Thanks guys for the help, it worked. I've been tricked by a strange error that occurs when displaying only the rgb image in the realsense-viewer, while using the libuvc or rsusb flags when building the driver. As I mentioned earlier, the video becomes extremely laggy and my system freezes, but after enabling both the depth camera and rgb everything works perfectly. I'd also like to mention that this happened once when I was using the ros2 package for realsense, so I'm assuming it's something at a lower level.

[MartyG-RealSense]

Hi @Yaccobv2 I'm pleased to hear that it is working for you now. Thanks very much for the update!

[javiercordon]

Hi @MartyG-RealSense @Yaccobv2 !
You guys can also use Jetsonhacks install sh. It's been there for a couple of years and still working for on Jetson Orin
source: https://github.com/jetsonhacks/installRealSenseSDK
and run the buildLibrealsense.sh

It is a very complete solution and also noticed that it is used under isaac_ros_common, exactly the same build script

Anyhow, just trying to share an existing tool to install without much complication.

Side Note:
In the meantime, I've been struggling to get the RealSense D455 to work on the examples for Jetpack 5.1.2 in containers since it's still not supported for JetPack 6.0D (versions of libraries are different, then need to update the versions on dockerfiles)

[MartyG-RealSense]

Hi @javiercordon Thanks very much for your recommendation of what has worked well for you!

[yurii-drevnytskyi-dianomix]

Hi @Yaccobv2,
I have the same problem - high frame drop rate, while only RGB stream is enabled
And looks like IMU doesn't work either, which I'm trying to fix by using this tutorial
More details here: #13020
Do you have a solution or at least some helpful thoughts?

[Dygitz]

Hi @Yaccobv2, I have the same problem - high frame drop rate, while only RGB stream is enabled And looks like IMU doesn't work either, which I'm trying to fix by using this tutorial More details here: #13020 Do you have a solution or at least some helpful thoughts?

Same exact problem here

[jeezrick]

Hi @MartyG-RealSense @Yaccobv2 ! You guys can also use Jetsonhacks install sh. It's been there for a couple of years and still working for on Jetson Orin source: https://github.com/jetsonhacks/installRealSenseSDK and run the buildLibrealsense.sh

It is a very complete solution and also noticed that it is used under isaac_ros_common, exactly the same build script

Anyhow, just trying to share an existing tool to install without much complication.

Side Note: In the meantime, I've been struggling to get the RealSense D455 to work on the examples for Jetpack 5.1.2 in containers since it's still not supported for JetPack 6.0D (versions of libraries are different, then need to update the versions on dockerfiles)

What's not supported for JetPack 6.0? the hack install installRealSenseSDK or the whole realsense thing? I make installRealSenseSDK and jetpack 5.1.x worked. But I just updated it to jetpack 6.0, realsense-viewer cannot detect camera...

[jeezrick]

My problem is that after update to jetpack 6.0, realsense-viewer installed fine, but can't detect camera while lsusb showed realsense. I tried installRealSenseSDK method and reinstall with sudo ./libuvc_installation.sh -DBUILD_WITH_CUDA=true too. nothing works. Do you have any idea? @MartyG-RealSense

[MartyG-RealSense]

Hi @jeezrick Is your Viewer version 2.55.1, please? Official support for JetPack 6 was added in SDK 2.55.1.

If your SDK version is 2.55.1, have you updated your camera's firmware driver version to 5.16.0.1

[jeezrick]

Hi @jeezrick Is your Viewer version 2.55.1, please? Official support for JetPack 6 was added in SDK 2.55.1.

If your SDK version is 2.55.1, have you updated your camera's firmware driver version to 5.16.0.1

Hi, I can't detect the device by running rs-enumerate-devices, but I can see it in lsusb:

$ rs-enumerate-devices 
No device detected. Is it plugged in?
$ lsusb
Bus 002 Device 003: ID 8086:0b5c Intel Corp. Intel(R) RealSense(TM) Depth Camera 455 
Bus 002 Device 002: ID 0bda:0489 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0bda:c822 Realtek Semiconductor Corp. Bluetooth Radio 
Bus 001 Device 004: ID 1ea7:0066 SHARKOON Technologies GmbH [Mediatrack Edge Mini Keyboard]
Bus 001 Device 002: ID 0bda:5489 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I ran 'rs-enumerate-devices` on another non-jetson device, it's indeed 5.16.0.1, and Viewer is 2.55.1 too.

[jeezrick]

I changed line 47 in ./libuvc_installation.sh file to
cmake ../ -DFORCE_LIBUVC=true -DCMAKE_BUILD_TYPE=release -DFORCE_RSUSB_BACKEND=ON, and build it.
Now rs-enumerate-devices can detect camera fine, but in the Viewer the rgb and depth stream is highly unstable, can't hold at 30 fps. And cannot open motion module too. What should I do to fix it?

[MartyG-RealSense]

Hi @jeezrick In the RGB controls in the Viewer's options side-panel, please try finding and disabling an option called Auto-Exposure Priority. If Auto-Exposure is true and Auto-Exposure Priority is false then the Viewer will try to enforce a constant FPS for both the depth and RGB streams instead of permitting their FPS to vary.

It is a known issue that for some RealSense users, enabling three streams simultaneously that include the Motion Module will cause one of the streams to fail. The trigger for this is the enabling of the Motion Module, so changing the order that the streams are enabled in will make no difference. There is no fix in the Viewer for this issue, except to only enable two streams if you need to use the IMU (such as Depth and IMU, or RGB and IMU), or enable IMU only.

[jeezrick]

HI, thanks for the reply. Turns out my problem is exactly like his

Thanks guys for the help, it worked. I've been tricked by a strange error that occurs when displaying only the rgb image in the realsense-viewer, while using the libuvc or rsusb flags when building the driver. As I mentioned earlier, the video becomes extremely laggy and my system freezes, but after enabling both the depth camera and rgb everything works perfectly. I'd also like to mention that this happened once when I was using the ros2 package for realsense, so I'm assuming it's something at a lower level.

After this:

I changed line 47 in ./libuvc_installation.sh file to cmake ../ -DFORCE_LIBUVC=true -DCMAKE_BUILD_TYPE=release -DFORCE_RSUSB_BACKEND=ON, and build it. Now rs-enumerate-devices can detect camera fine, but in the Viewer the rgb and depth stream is highly unstable, can't hold at 30 fps. And cannot open motion module too. What should I do to fix it?

I downgrade firmware to 5.13 like some other issue mentioned. Then IMU works. But still has that problem that when only enable rgb, the stream is highly unstable. Enable both rgb and depth, everything seems fine.

So I still feels like I'm working with a broken realsense lib.

I wonder should I add the https://github.com/IntelRealSense/realsense_mipi_platform_driver/blob/dev/README_JP6.md#build-nvidia-kernel-drivers-dtb-and-d457-driver---cross-compile-x86-64 patch? since my imu is working. Will this fix the rgb stream problem?

[MartyG-RealSense]

Using the MIPI driver with a USB connection on a computer with JetPack 6 adds support for the IMU, which is a 'HID' device. It should not affect RGB.

# Kernel Image with SENSOR_HID support for RealSense USB cameras with IMU
scp -r images/6.0/rootfs/boot/Image [email protected]:~/