Manual exposure control results in frames being dropped

[bulveops]

Required Info
Camera Model	D415
Firmware Version	05.12.08.200
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.8.0-48-generic
Platform	PC
SDK Version	2.43.0.3018
Language	python
Segment	Robot

Issue Description

I'd like to control exposure of an infrared sensor in an more sophisticated way, because auto-expose with static a region of interest results in bad depthmaps. The idea is to set exposure option on every frame, attempting to make constant the mean value of pixels inside tracked roi. I tried to change exposure time on each frame, but unfortunately there is a significant number of dropped frames.

import time
import pyrealsense2 as rs

def change_exposure_test():
    pipeline = rs.pipeline()
    config = rs.config()
    config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
    config.enable_stream(rs.stream.infrared, 1, 640, 480, rs.format.y8, 30)

    TEST_DURATION = 5.0

    print("*** No manual exposure control ***")
    total_frame_processed = 0
    profile = pipeline.start(config)
    first_frame_index = pipeline.wait_for_frames().get_depth_frame().get_frame_number()
    start = time.time()
    while time.time() - start < TEST_DURATION:
        frames = pipeline.wait_for_frames()
        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()
        cur_frame_index = depth_frame.get_frame_number()

        total_frame_processed += 1
    pipeline.stop()
    total_dropped_frames = cur_frame_index - first_frame_index - total_frame_processed + 1
    print(f"Processed {total_frame_processed} frames, dropped {total_dropped_frames} frames in {TEST_DURATION} seconds")


    print("*** Manual exposure control ***")
    total_frame_processed = 0
    profile = pipeline.start(config)
    first_frame_index = pipeline.wait_for_frames().get_depth_frame().get_frame_number()
    start = time.time()
    while time.time() - start < TEST_DURATION:
        frames = pipeline.wait_for_frames()
        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()
        cur_frame_index = depth_frame.get_frame_number()
        try:
            profile.get_device().first_depth_sensor().set_option(rs.option.exposure, 10000)
        except RuntimeError as err:
            print("Failed to set exposure: ", err)
        total_frame_processed += 1
    pipeline.stop()
    total_dropped_frames = cur_frame_index - first_frame_index - total_frame_processed + 1
    print(f"Processed {total_frame_processed} frames, dropped {total_dropped_frames} frames in {TEST_DURATION} seconds")

if __name__ == "__main__":
    change_exposure_test()

With this sample of code, I get the next output:

*** No manual exposure control ***
Processed 152 frames, dropped 0 frames in 5.0 seconds
*** Manual exposure control ***
Processed 51 frames, dropped 99 frames in 5.0 seconds

Is it a normal behaviour? How to control exposure without loosing frames?

[MartyG-RealSense]

Hi @bulveops Setting exposure manually can affect FPS if the value is set within a certain range, as described in the link below:

#1957 (comment)

If you wish to control exposure on a per-frame basis then the RealSense SDK's High Dynamic Range (HDR) mode may be suitable. The principles are described in Intel's HDR white-paper document.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

This section of the white paper describes how HDR Enabled mode can be tested in the RealSense Viewer program.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras#section-3-1-hdr-with-intel-real-sense-viewer

A RealSense user shared their Python script for setting the hdr_enabled option.

#7874 (comment)

There is also an example script for doing so in the paper.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras#section-3-2-controlling-hdr-programmatically

[bulveops]

Thank you for fast reply, @MartyG-RealSense
Suppose, HDR mode isn't a proper choice for a dynamic scenes, when moving objects are present. Another drawback is additional CPU utilization. That's why I turned to manual control of exposure parameter.
As regarding #1957 the exposure period of 10 ms (10000 as in code sample) is suitable for 30 fps sampling. Moreover, frame are dropping only if I set a new exposure value for every frame. With changed exposure parameter (value of 10000) I can retrieve frames with 30 fps.

[MartyG-RealSense]

My concern is that using a per-frame exposure technique that dates from before the HDR feature may be unsuited to the slow rolling shutter on the D415 camera model, compared to the fast global shutter on left and right sensors of the D435 / i and on both RGB and left-right sensors on the D455.

#3844 (comment)

[bulveops]

@MartyG-RealSense ,
unfortunately I have also frames being dropped with D455 model, which has the fast global shutter. Suppose, there are hardware delays or some limitation in the firmware that doesn't allow to use a per-frame exposure technique.

[MartyG-RealSense]

If the goal is to enforce the intensity value of the pixels inside the ROI at a constant then I am not sure that the exposure will directly influence that. When auto-exposure is enabled, the value that the ROI tries to average the pixel intensity at is determined by a value called the setpoint, described in point 3 of the section of Intel's camera tuning paper linked to below.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-make-sure-image-is-properly-exposed

[bulveops]

The auto-exposure algorithms operates with a static ROI. I want to enforce the intensity value of pixels inside the dynamic ROI to be constant. Alternatively to adjust exposure parameters for pixels belonging to the foreground.
BTW what are units for those setpoint parameter? By default it is set to the value of 400 units - it's more than upper value of 8bit images and very low for 16bit images.

[MartyG-RealSense]

My research of your question about the setpoint value indicates that it does not have a unit description associated with it.

I ran tests with the setpoint in the RealSense Viewer whilst the depth stream was enabled and editing the setpoint value altered the depth image in real-time.

My understanding though is that the Mean Intensity Set Point is part of the Advanced Mode's STAEControl struct (see the link below) but is normally custom-configured through a json file at program start-up rather than accessible dynamically during runtime.

https://intelrealsense.github.io/librealsense/doxygen/struct_s_t_a_e_control.html

[MartyG-RealSense]

There have been a couple of recent cases involving exposure difficulties that were resolved by the SDK's new feature for setting auto-exposure and gain maximum limits for IR sensors that was added in SDK version 2.42.0 and firmware version 5.12.11.0.

The header at the top of this case states that your currently installed firmware is 5.12.8.200, so updating to 5.12.11.0 or newer may provide an immediate benefit.

[MartyG-RealSense]

A RealSense user has posted Python code that they used to set the intensity setpoint.

#8698

[MartyG-RealSense]

Hi @bulveops Do you require further assistance with this case, please? Thanks!

[bulveops]

Thank you for the assistance! I found out that changing exposure parameters with interval more than 2-3 frames resolves the problem.

[MartyG-RealSense]

Excellent to hear that you found a solution @bulveops - thanks for the update!