Incoherent frameset: frame number mismatch frames #7807

randomstuff · 2020-11-18T09:40:52Z

Required Info
Camera Model	D435
Firmware Version	05.11.01.100
Operating System & Version	Debian testing
Kernel Version (Linux Only)	5.9.0
Platform	PC
SDK Version	2.39.0, development branch (> 2.39.0)
Language	C++
Segment	others

Issue Description

When enabling RS2_RS400_VISUAL_PRESET_DEFAULT, the frames are not coherent within a single rs2::frameset: the frame numbers (infrared and depth frames) do not match.

Minimal example (derived from issue 2224):

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <iostream>
#include <unistd.h>

int main(int argc, char * argv[])
{
    try
    {
        const int N = 100;

        rs2::config config;
        // 848x480
        // 1280x720
        config.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 30);
        config.enable_stream(RS2_STREAM_INFRARED, 848, 480, RS2_FORMAT_Y8, 30);
        rs2::pipeline pipe;
        {
            auto selection = pipe.start(config);
            rs2::device selected_device = selection.get_device();
            auto depth_sensor = selected_device.first<rs2::depth_sensor>();
            depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET,RS2_RS400_VISUAL_PRESET_DEFAULT); // (1)
        }

        double preTimestampDepth = 0;
        double preTimestampInfrared = 0;
        for (int i = 0; i < N; i++)
        {
            auto fs = pipe.wait_for_frames();
            auto depth_frame = fs.get_depth_frame();
            auto infrared_frame = fs.get_infrared_frame();
            double timestampDepth = depth_frame.get_timestamp();
            double timestampInfrared = infrared_frame.get_timestamp();
            if (i == 0)
            {
                printf("frame [%02d]. depth : frame# %lld, timestamp diff [%7.3f]. infrared : frame# %lld, timestamp diff[%7.3f].\n", i, 
                    depth_frame.get_frame_number(), 0.0f,
                    infrared_frame.get_frame_number(), 0.0f);
            }
            else
            {
                printf("frame [%02d]. depth : frame# %lld, timestamp diff [%7.3f]. infrared : frame# %lld, timestamp diff[%7.3f].\n", i, 
                    depth_frame.get_frame_number(), timestampDepth - preTimestampDepth,
                    infrared_frame.get_frame_number(), timestampInfrared - preTimestampInfrared);
            }
            preTimestampDepth = timestampDepth;
            preTimestampInfrared = timestampInfrared;
        }

        return EXIT_SUCCESS;
    }
    catch (const rs2::error & e)
    {
        std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
        return EXIT_FAILURE;
    }
    catch (const std::exception& e)
    {
        std::cerr << e.what() << std::endl;
        return EXIT_FAILURE;
    }
}

Output:

$ g++ test_realsense.cpp -lrealsense2 && ./a.out 
frame [00]. depth : frame# 42, timestamp diff [  0.000]. infrared : frame# 44, timestamp diff[  0.000].
frame [01]. depth : frame# 42, timestamp diff [  0.000]. infrared : frame# 46, timestamp diff[ 67.107].
frame [02]. depth : frame# 42, timestamp diff [  0.000]. infrared : frame# 47, timestamp diff[146.760].
frame [03]. depth : frame# 43, timestamp diff [213.867]. infrared : frame# 47, timestamp diff[  0.000].
frame [04]. depth : frame# 43, timestamp diff [  0.000]. infrared : frame# 48, timestamp diff[ 33.337].
frame [05]. depth : frame# 44, timestamp diff [ 33.337]. infrared : frame# 48, timestamp diff[  0.000].
frame [06]. depth : frame# 45, timestamp diff [ 33.314]. infrared : frame# 49, timestamp diff[ 33.314].
frame [07]. depth : frame# 46, timestamp diff [ 33.293]. infrared : frame# 50, timestamp diff[ 33.293].
frame [08]. depth : frame# 47, timestamp diff [ 33.489]. infrared : frame# 51, timestamp diff[ 33.489].
frame [09]. depth : frame# 48, timestamp diff [ 33.476]. infrared : frame# 52, timestamp diff[ 33.476].
frame [10]. depth : frame# 49, timestamp diff [ 33.462]. infrared : frame# 53, timestamp diff[ 33.462].
frame [11]. depth : frame# 50, timestamp diff [ 33.451]. infrared : frame# 54, timestamp diff[ 33.451].
frame [12]. depth : frame# 51, timestamp diff [ 33.438]. infrared : frame# 55, timestamp diff[ 33.438].
frame [13]. depth : frame# 52, timestamp diff [ 33.364]. infrared : frame# 56, timestamp diff[ 33.364].
frame [14]. depth : frame# 53, timestamp diff [ 33.347]. infrared : frame# 57, timestamp diff[ 33.347].
frame [15]. depth : frame# 54, timestamp diff [ 33.331]. infrared : frame# 58, timestamp diff[ 33.331].
frame [16]. depth : frame# 55, timestamp diff [ 33.047]. infrared : frame# 59, timestamp diff[ 33.047].
frame [17]. depth : frame# 56, timestamp diff [ 33.034]. infrared : frame# 60, timestamp diff[ 33.034].
frame [18]. depth : frame# 57, timestamp diff [ 33.020]. infrared : frame# 61, timestamp diff[ 33.020].

This is fixed by removing line (1).

The text was updated successfully, but these errors were encountered:

randomstuff · 2020-11-18T09:44:42Z

Same thing with depth and color:

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <iostream>
#include <unistd.h>

int main(int argc, char * argv[])
{
    try
    {
        const int N = 100;

        rs2::config config;
        config.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 30);
        config.enable_stream(RS2_STREAM_COLOR, 848, 480, RS2_FORMAT_RGB8, 30);
        rs2::pipeline pipe;
        {
            auto selection = pipe.start(config);
            rs2::device selected_device = selection.get_device();
            auto depth_sensor = selected_device.first<rs2::depth_sensor>();
            depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET,RS2_RS400_VISUAL_PRESET_DEFAULT);
        }

        double preTimestampDepth = 0;
        double preTimestampColor = 0;
        for (int i = 0; i < N; i++)
        {
            auto fs = pipe.wait_for_frames();
            auto depth_frame = fs.get_depth_frame();
            auto color_frame = fs.get_color_frame();
            double timestampDepth = depth_frame.get_timestamp();
            double timestampColor = color_frame.get_timestamp();
            if (i == 0)
            {
                printf("frame [%02d]. depth : frame# %lld, timestamp diff [%7.3f]. color : frame# %lld, timestamp diff[%7.3f].\n", i, 
                    depth_frame.get_frame_number(), 0.0f,
                    color_frame.get_frame_number(), 0.0f);
            }
            else
            {
                printf("frame [%02d]. depth : frame# %lld, timestamp diff [%7.3f]. color : frame# %lld, timestamp diff[%7.3f].\n", i, 
                    depth_frame.get_frame_number(), timestampDepth - preTimestampDepth,
                    color_frame.get_frame_number(), timestampColor - preTimestampColor );
            }
            preTimestampDepth = timestampDepth;
            preTimestampColor = timestampColor;
        }

        return EXIT_SUCCESS;
    }
    catch (const rs2::error & e)
    {
        std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
        return EXIT_FAILURE;
    }
    catch (const std::exception& e)
    {
        std::cerr << e.what() << std::endl;
        return EXIT_FAILURE;
    }
}

Output:

frame [00]. depth : frame# 53, timestamp diff [  0.000]. color : frame# 59, timestamp diff[  0.000].
frame [01]. depth : frame# 54, timestamp diff [ 33.985]. color : frame# 60, timestamp diff[ 33.999].
frame [02]. depth : frame# 55, timestamp diff [ 34.009]. color : frame# 61, timestamp diff[ 34.023].
frame [03]. depth : frame# 56, timestamp diff [ 34.034]. color : frame# 62, timestamp diff[ 34.177].
frame [04]. depth : frame# 57, timestamp diff [ 34.272]. color : frame# 63, timestamp diff[ 34.210].
frame [05]. depth : frame# 58, timestamp diff [ 34.225]. color : frame# 64, timestamp diff[ 34.243].
frame [06]. depth : frame# 59, timestamp diff [ 34.258]. color : frame# 65, timestamp diff[ 34.275].
frame [07]. depth : frame# 60, timestamp diff [ 34.291]. color : frame# 66, timestamp diff[ 34.249].
frame [08]. depth : frame# 61, timestamp diff [ 34.227]. color : frame# 67, timestamp diff[ 34.281].
frame [09]. depth : frame# 62, timestamp diff [ 34.295]. color : frame# 68, timestamp diff[ 34.312].
frame [10]. depth : frame# 63, timestamp diff [ 34.326]. color : frame# 69, timestamp diff[ 34.343].
frame [11]. depth : frame# 64, timestamp diff [ 34.357]. color : frame# 70, timestamp diff[ 34.374].
frame [12]. depth : frame# 65, timestamp diff [ 34.389]. color : frame# 71, timestamp diff[ 34.405].
frame [13]. depth : frame# 66, timestamp diff [ 34.420]. color : frame# 72, timestamp diff[ 34.277].
frame [14]. depth : frame# 67, timestamp diff [ 34.189]. color : frame# 73, timestamp diff[ 34.303].
frame [15]. depth : frame# 68, timestamp diff [ 34.315]. color : frame# 74, timestamp diff[ 34.330].
frame [16]. depth : frame# 69, timestamp diff [ 34.341]. color : frame# 75, timestamp diff[ 34.594].
frame [17]. depth : frame# 70, timestamp diff [ 34.366]. color : frame# 76, timestamp diff[ 33.999].
frame [18]. depth : frame# 71, timestamp diff [ 34.002]. color : frame# 77, timestamp diff[ 34.012].
frame [19]. depth : frame# 72, timestamp diff [ 34.015]. color : frame# 78, timestamp diff[ 34.161].
frame [20]. depth : frame# 73, timestamp diff [ 34.028]. color : frame# 79, timestamp diff[ 33.818].
frame [21]. depth : frame# 74, timestamp diff [ 33.816]. color : frame# 80, timestamp diff[ 33.823].
frame [22]. depth : frame# 75, timestamp diff [ 33.822]. color : frame# 81, timestamp diff[ 33.828].

MartyG-RealSense · 2020-11-18T17:27:21Z

Hi @randomstuff If you are not using a point cloud and do not need the reduced point cloud spraying that the Default visual preset provides, then it should be fine to leave the visual preset instruction out and let the default configuration settings be used.

The majority of people use a preset to achieve a biased weighting of settings that favor certain characteristics but have a trade-off in other characteristics (e.g the High Accuracy preset for higher accuracy at the cost of reduced fill rate). If you want a balanced spread of characteristics then the camera default settings should be sufficient.

randomstuff · 2020-11-19T10:44:16Z

I'm not sure I'm following you. The settings (such as RS2_RS400_VISUAL_PRESET_DEFAULT) are persistent from one execution to the next (apparently a power reset of the RealSense is needed to reset the settings): it is better to explicitly set a given profile in order to have reproducible results (otherwise, the behavior of my program might depend on whatever was executed before).

Note: I checked with a visual inspection of the frames and I think the frames are actually synchronized even if the frame numbers are not consistent.

MartyG-RealSense · 2020-11-19T19:48:22Z

Thanks very much for the update @randomstuff - given your visual inspection of the frames, do you believe that there is still a problem that needs resolving, please? Thanks!

randomstuff · 2020-11-20T08:45:56Z

The frames seems in sync but the frames have a different frame number. Is this expected (especially for depth/infrared) or are we expected to be able to use the frame numbers for frames synchronization?

MartyG-RealSense · 2020-11-20T15:47:16Z

I reviewed your case again from the beginning again to look for new insights. Basically, the situation as I understand it is:

Applying the Default visual preset causes two different streams (depth and color, or depth and infrared) to have different frame numbers, though they appear to be in sync.
The issue does not occur if the line marked in your code with the reference (1) - not the actual line 1 of the script - is removed. That referred-to line is:

depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET,RS2_RS400_VISUAL_PRESET_DEFAULT);

I researched similar cases of setting a built-in visual preset with C++ using this method, and observed that the line above seems to be missing some code compared to the cases I researched.

Could you therefore try adjusting your visual preset line to use the code below, please?

depth_sensor.set_option(rs2_option::RS2_OPTION_VISUAL_PRESET, rs2_rs400_visual_preset::RS2_RS400_VISUAL_PRESET_DEFAULT);

randomstuff · 2020-11-25T13:55:04Z

To summarize, when using the following lines after pipe.start(config):

depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET,RS2_RS400_VISUAL_PRESET_DEFAULT);

The frames in a rs2::frameset returned by a pipe.wait_for_frames() call do not have the same frame_number. The difference between the frame numbers inside these rs2::frameset is:

constant within a single execution;
but variable across different executions.

However, the timestamp is the same inside the same rs2::frameset:

frame [00]. depth : frame# 33, timestamp [1606311696103.691]. infrared : frame# 35, timestamp [1606311696103.691].
frame [01]. depth : frame# 34, timestamp [1606311696137.537]. infrared : frame# 36, timestamp [1606311696137.537].
frame [02]. depth : frame# 35, timestamp [1606311696171.317]. infrared : frame# 37, timestamp [1606311696171.317].
frame [03]. depth : frame# 36, timestamp [1606311696205.030]. infrared : frame# 38, timestamp [1606311696205.030].
frame [04]. depth : frame# 37, timestamp [1606311696238.677]. infrared : frame# 39, timestamp [1606311696238.677].
frame [05]. depth : frame# 38, timestamp [1606311696272.641]. infrared : frame# 40, timestamp [1606311696272.641].

As per bug 2224, I thought this difference of frame number was a sign that the frames were not synchronized correctly by librealsense inside the rs2::frameset (i.e. two frames in a same rs2::frameset were not actually mapping to the same instant) and that this was considered a bug.

However, visual inspection of the frames did not allow me to conclude there was a mismatch within a rs2::frameset.

In this use-case, for a single rs2::frameset:

should I expect the frames to map to the same moment in the real world?
should I expect the frames to have the same timestamp value?
should I expect the frames to have the same frame number?

How should I ensure that two frames actually map to the same moment in the real world?

because they are returned by the same pipe.wait_for_frames() code?
based on their timestamp?
based on their frame number?

randomstuff · 2020-11-25T13:55:10Z

You suggest I should try using the following line:

depth_sensor.set_option(rs2_option::RS2_OPTION_VISUAL_PRESET,
  rs2_rs400_visual_preset::RS2_RS400_VISUAL_PRESET_DEFAULT);

This line is however equivalent to the one I'm using:

depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET,RS2_RS400_VISUAL_PRESET_DEFAULT);

as:

typedef enum rs2_option
{
  // ....
  RS2_OPTION_VISUAL_PRESET,
  // ...
};
// ...
typedef enum rs2_rs400_visual_preset
{
  // ...
  RS2_RS400_VISUAL_PRESET_DEFAULT,
  // ...
} rs2_rs400_visual_preset;

MartyG-RealSense · 2020-11-25T19:08:14Z

I reviewed your case again from the start and noted that you list in the specification box at the top of the case that you are using kernel 5.9. SDK 2.39.0 is known to be able to work unofficially with kernel 5.4-generic (official kernel 5.4 support was added in SDK 2.40.0). There is no previous case I know of where a newer Linux Kernel than 5.4 has been used with librealsense though.

It may therefore be worth trying to build librealsense from source code with the RSUSB method that bypasses the kernel. RSUSB - activated by including the term -DFORCE_RSUSB_BACKEND=true in a CMake buid instruction - requires an internet connection but is not dependent on Linux versions or kernel versions and does not require patching.

#6368 (comment)

I akso note that the camera firmware version listed - 05.11.01.100 - is very old, and the recommended firmware version for SDK 2.39.0 is 5.12.8.200.

MartyG-RealSense · 2020-12-03T13:47:56Z

Hi @randomstuff Do you require further assistance with this case, please? Thanks!

randomstuff · 2020-12-07T13:55:16Z

Sorry for the delay.

Compiling the same code with FORCE_RSUSB_BACKEND=true, indeed fixes the issue.

Note: I need to run as root in this configuration but I guess I should be able to fix that using a suitable configuration.

I'll try to update the firmware as well.

MartyG-RealSense · 2020-12-07T13:57:24Z

Thanks very much for the update @randomstuff

randomstuff · 2020-12-07T14:17:32Z

I guess this was expected but updating the firmware does not change that.

randomstuff · 2020-12-07T14:42:48Z

For reference, the RUSB backend requires access to /dev/bus/usb/{bus_number}/{device_number} (as given by lsusb).

MartyG-RealSense · 2020-12-07T18:58:25Z

Hi @randomstuff Do you need further advice from me about anything please? Thanks!

randomstuff · 2020-12-09T09:28:25Z

I assume the bug will be fixed because this kernel version is not supported.

The current state is fine for me: AFAIU, the bug is more about the metadata which is incorrectly reported than an actual desync of the frames.

MartyG-RealSense · 2020-12-09T09:38:36Z

If you are referring to #2224, it looks as though timestamp differences are regarded as a consequence of the rules of depth-RGB sync rather than a bug.

#2224 (comment)

MartyG-RealSense · 2020-12-16T07:29:43Z

Hi @randomstuff Do you require further assistance with this case, please? Thanks!

randomstuff · 2020-12-16T07:54:23Z

No, it should be OK for me.

MartyG-RealSense · 2020-12-16T08:46:40Z

Thanks very much for the update @randomstuff - I will close this case now if you are satisified. Thanks again!

randomstuff changed the title ~~Incoherent frameset: frame number mismatch between IR and depth frames~~ Incoherent frameset: frame number mismatch frames Nov 18, 2020

MartyG-RealSense added D400 Series Linux labels Nov 18, 2020

MartyG-RealSense closed this as completed Dec 16, 2020

MartyG-RealSense mentioned this issue Mar 17, 2023

Depth and IR frame counters are not consistent #11579

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incoherent frameset: frame number mismatch frames #7807

Incoherent frameset: frame number mismatch frames #7807

randomstuff commented Nov 18, 2020 •

edited

Loading

randomstuff commented Nov 18, 2020 •

edited

Loading

MartyG-RealSense commented Nov 18, 2020

randomstuff commented Nov 19, 2020

MartyG-RealSense commented Nov 19, 2020

randomstuff commented Nov 20, 2020

MartyG-RealSense commented Nov 20, 2020 •

edited

Loading

randomstuff commented Nov 25, 2020

randomstuff commented Nov 25, 2020 •

edited

Loading

MartyG-RealSense commented Nov 25, 2020

MartyG-RealSense commented Dec 3, 2020

randomstuff commented Dec 7, 2020 •

edited

Loading

MartyG-RealSense commented Dec 7, 2020

randomstuff commented Dec 7, 2020

randomstuff commented Dec 7, 2020

MartyG-RealSense commented Dec 7, 2020

randomstuff commented Dec 9, 2020

MartyG-RealSense commented Dec 9, 2020 •

edited

Loading

MartyG-RealSense commented Dec 16, 2020

randomstuff commented Dec 16, 2020

MartyG-RealSense commented Dec 16, 2020

Incoherent frameset: frame number mismatch frames #7807

Incoherent frameset: frame number mismatch frames #7807

Comments

randomstuff commented Nov 18, 2020 • edited Loading

Issue Description

randomstuff commented Nov 18, 2020 • edited Loading

MartyG-RealSense commented Nov 18, 2020

randomstuff commented Nov 19, 2020

MartyG-RealSense commented Nov 19, 2020

randomstuff commented Nov 20, 2020

MartyG-RealSense commented Nov 20, 2020 • edited Loading

randomstuff commented Nov 25, 2020

randomstuff commented Nov 25, 2020 • edited Loading

MartyG-RealSense commented Nov 25, 2020

MartyG-RealSense commented Dec 3, 2020

randomstuff commented Dec 7, 2020 • edited Loading

MartyG-RealSense commented Dec 7, 2020

randomstuff commented Dec 7, 2020

randomstuff commented Dec 7, 2020

MartyG-RealSense commented Dec 7, 2020

randomstuff commented Dec 9, 2020

MartyG-RealSense commented Dec 9, 2020 • edited Loading

MartyG-RealSense commented Dec 16, 2020

randomstuff commented Dec 16, 2020

MartyG-RealSense commented Dec 16, 2020

randomstuff commented Nov 18, 2020 •

edited

Loading

randomstuff commented Nov 18, 2020 •

edited

Loading

MartyG-RealSense commented Nov 20, 2020 •

edited

Loading

randomstuff commented Nov 25, 2020 •

edited

Loading

randomstuff commented Dec 7, 2020 •

edited

Loading

MartyG-RealSense commented Dec 9, 2020 •

edited

Loading