Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D435 frame rate changes with respect to ambient lighting #5788

Closed
anug7 opened this issue Feb 5, 2020 · 13 comments
Closed

D435 frame rate changes with respect to ambient lighting #5788

anug7 opened this issue Feb 5, 2020 · 13 comments

Comments

@anug7
Copy link

anug7 commented Feb 5, 2020


Required Info
Camera Model D435
Firmware Version 05.11.15.00
Operating System & Version Ubuntu 16.04
Kernel Version (Linux Only) 4.15
Platform Lattepanda Alpha
SDK Version 2.22.0
Language C++
Segment Robot

Issue Description

The frame rate of D435 stream (both color & depth) are not constant when the external lighting varies. We have a external strobe LED to illuminate the scene and when the intensity is high, we see that frame rate goes high (6 fps) compared to 3.5 fps when illumination is very low. See the attachment for more details. For our requirement, we need constant fps on both the streams. Any pointers related to this issue would be really helpful.

Thank you in advance!

D435 settings:

  1. Color: 1920x1080 @ 6 fps
  2. Depth: 1280x720 @ 6 fps
  3. Using recorder to record the stream in bag file.
  4. I have tried disabling both auto exposure and auto white balancing. But no improvements.
  5. We record this streams when the system is under full load. e.g 400% cpu consumption. The recorded bag file also has this fps issue.

Code:

  1. Delay is calculating in the code by following piece of code, prev_time is 0.167 when the lighting intensity is high and it does to 0.367 - 0.500 when the intensity is low
if (boost::iequals(name, "Stereo Module")){
         sen.open(depth_stream);
         sen.start([&](rs2::frame f){ 
              printf("Time: %f\n", f.get_timestamp() /1000 - prev_time);
              prev_time = f.get_timestamp() / 1000;
          });
 }
  1. The auto exposure and white balancing are disabled by using the following code,
void set_controls(rs2::device dev, ros::NodeHandle n){
        int expo = 400, gain = 128, wb = 3700;
        bool  auto_expo = false, auto_wb = false;
        for (auto sen: dev.query_sensors()){
                auto name = sen.get_info(RS2_CAMERA_INFO_NAME);
                if(boost::iequals(name, "RGB Camera")){
                        sen.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, auto_expo);
                        sen.set_option(RS2_OPTION_EXPOSURE, expo);
                        sen.set_option(RS2_OPTION_GAIN, gain);
                        sen.set_option(RS2_OPTION_ENABLE_AUTO_WHITE_BALANCE, auto_wb);
                        sen.set_option(RS2_OPTION_WHITE_BALANCE, wb);
                }else if (boost::iequals(name, "Stereo Module")){
                        sen.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, false);
                        sen.set_option(RS2_OPTION_EXPOSURE, 5000);
                }
        }
}

Attachment details:

  1. First two images corresponds to high intensity (as per setup) scenario of color & depth images.
  2. Second two images corresponds to low intensity.

high_color
high_depth
low_color
low_depth

@MartyG-RealSense
Copy link
Collaborator

You can enforce a constant FPS speed by leaving auto-exposure enabled but disabling the 'Auto-Exposure Priority' option.

#5290

@anug7
Copy link
Author

anug7 commented Feb 5, 2020

@MartyG-RealSense ,
Thank you for the suggestion.

Will try the option and let you know. Just curious, Wont disabling the auto-exposure give constant fps?

Also for our requirement, we need the exposure to be manual.

@MartyG-RealSense
Copy link
Collaborator

You could disable auto-exposure and set your own values manually, though it may affect depth performance for the reasons discussed in the link below,

#1957 (comment)

@anug7
Copy link
Author

anug7 commented Feb 5, 2020

I tried setting it for depth stream. But it fails.

try{
    sen.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, false);
}
catch(...){
    std::cout << "Setting failed"<< "\n";
}

And with manual exposure, the problem still persists.

@anug7
Copy link
Author

anug7 commented Feb 5, 2020

@MartyG-RealSense

Also I tried for color stream for which I was able to set this value using the same code. But the problem still persists. I used auto exposure and disabled RS2_OPTION_AUTO_EXPOSURE_PRIORITY.

@MartyG-RealSense
Copy link
Collaborator

Generally, RS2_OPTION settings are set to false or true with a 0 or 1, where 0 = false and 1 = true. For example, to disable:

sen.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0.f);

@anug7
Copy link
Author

anug7 commented Feb 6, 2020

@MartyG-RealSense , I have tried your suggestion. Still the problem persists.
And also tried updating librealsense to 2.31.0 version.

@MartyG-RealSense
Copy link
Collaborator

If you have turned off auto-exposure, did you set a manual exposure value that is appropriate for the FPS speed that you are using, as described in the advice linked to earlier please?

#1957 (comment)

@anug7
Copy link
Author

anug7 commented Feb 6, 2020

Yes. I set the value of exposure to 400 (~25fps).

@MartyG-RealSense
Copy link
Collaborator

I wonder if the sudden changes in light conditions caused by the LED might be saturating the IR imager, in the same way that a stream may degrade or freeze if it captures a burst of sunlight (by looking up at the sky or experiencing a bright ray reflected off a surface such as a glass door).

@anug7
Copy link
Author

anug7 commented Feb 6, 2020

Is there a way we could handle this? such that we can get consistent fps.

@MartyG-RealSense
Copy link
Collaborator

You could apply a physical IR filter to the camera. This could take the form of a bulky external filter or a micro-thin filter film placed over the IR lens on the front of the camera.

Another approach may be to program the camera to reset automatically if an event that degrades performance occurs. Assuming that the LED will be strobing between low and high constantly though, prevention (filtering) is probably better in this particular case, otherwise you might have to reset the camera repeatedly.

The subject of physical filtering with cover materials is discussed in detail in the link below.

https://forums.intel.com/s/question/0D50P00004O1gOtSAJ/where-can-we-obtain-cover-material-and-gaskets-for-custom-mounting-of-the-d415-stereo-depth-module

@anug7
Copy link
Author

anug7 commented Feb 6, 2020

@MartyG-RealSense , Thank you for your suggestion. Will try!

@anug7 anug7 closed this as completed Feb 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants