Strange Framerate behaviour when changing Exposure

[TimoK93]

Required Info
Camera Model	D435
Firmware Version	05.09.11.00
Operating System & Version	Linux Ubuntu 16.04
Kernel Version (Linux Only)	4.4.38
Platform	NVIDIA Jetson Tx2 (L4T 28.2, JetPack 3.2)
SDK Version	Running with LibRealSense v2.10.4
Segment	Robot
ROS	RealSense ROS v2.0.2

Issue Description

Hey devs,

im using my D435 cam on a Jetson TX2. I patched the kernel and installed librealsense with ROS extensions.

The setup seems to work, but there is a strange behaviour with the camera. I start a rs_camera nodelet configured with Depth and Color stream. Depth and Color streams are configured with 1280x720 and 30 fps. The color stream is just streaming with 15 fps. First i could not determine the cause but now i think its a firmware issue:
If i disable auto-exposure and set the exposure value to ~400 (realsense_viewer or rqt_reconfigure), the framerate increases up to ~25 fps. If i switch the exposure value to ~600 or ~200 the fps is decreasing less than 15 fps.

What is this?
And how can i create a stable stream to 30 fps in HD and FHD?

Thanks in advance!
Note: This behaviour is just existing with HD or FHD resolution!

[ev-mp]

Hello @TimoK93 ,
According to the description you're trying to configure RGB sensor for constant FPS rate.

The sensor FPS is by bound by exposure values by default.
Therefore, in case the exposure value exceed the FPS period ( 1000ms/30fps=33.333msec), then FPS drops accordingly.
This is consistent with the use-case-
Setting exposure 400 (=40msec in 100usec units ) can produce up to (1000msec/40msec=) 25 FPS.

To achieve the goal -

1. Disable "Auto-Exposure" and set it manually, verifying it does not exceed the upper bound.
2. Leave "Auto-Exposure switched on, but disable "Auto-Exposure Priority" control to enforce constant FPS.

Note that disabling auto-exposure can affect depth performance, especially in low light conditions/ w.o projector illumination. In that case modifying "Gain" can improve it (but also introduce more noise).

[TimoK93]

Thanks for reply!

I tried your suggestions:

1. Set Exposure manually to 400 -> Stable FPS ~25. Set Exposure to 333 -> FPS is increasing to 15 FPS with this Harware Notification every second frame: "Incomplete Frame detected! SIze 65784 out of 1843455 bytes (3%)"
2. Disabled Auto Exposure Priority and Enabled Auto Exposure: Same result as setting exposure manually to 333!

Could this be a USB3 Port limitation? I never heard about this correlated to D435 and JetsonTX2...

[ev-mp]

Hello @TimoK93 , The error type

"Incomplete Frame detected! ..

is an indication of a host USB controller having throughput issues, cannot keep the pace with the HW and looses frames. It is often the case of Virtual Machines and unfortunately Jetson TX2.

We've experimented with different setups including self-powered USB hubs but could not mitigate it. Also tried extension (PCI-E) USB board instead of onboard USB3 controllers with equally disappointing results.
Imho, the root cause is not necessary in the board's HW, but rather kernel/drivers/ BSP suite.

[TimoK93]

Thanks!

I will work on with 25 FPS. It is okay for my application!

[inders]

I am facing this issue of frame rate not being stable with Jetson tx2 and d415. How do I keep auto exposure ON, the library on Jetson does not compile if I do so.

[greipfrut]

I have found that if you use YUYV color stream in the realsense-viewer (or enable_stream(format=rs.format.yuyv, ...) if you're using pyrealsense2), you can hit a stable 30 fps if the appropriate exposure settings are used.

When looking at the datasheets (https://www.intelrealsense.com/wp-content/uploads/2020/06/Intel-RealSense-D400-Series-Datasheet-June-2020.pdf), the tables actually list the fps specs for YUV colorspace rather than RGB, so I suppose it kind of makes sense if stable 30 fps is only assured in YUV.

I then found this comment extremely helpful for getting it back into RGB: #4646 (comment)