Timestamp baisis in callback mode

[ztorok]

---

Required Info
Camera Model	{ D435i }
Firmware Version	(05.11.010.100)
Operating System & Version	{Win (10)
Platform	PC
SDK Version	{ master as of 09. May 2019 }
Language	{C++ }
Segment	{others }

Issue Description

I'm trying to align IR frames with IMU measurements, but I can't figure out how to read out the frames and IMU with the same time stamp basis (hardware clock, instead of system time).

I have followed this issue and did everything as described, but it didn't solve my issue: #3205

I modified the rs-save-to-disk.cpp example as follows:
`int main(int argc, char * argv[]) try
{
// Declare depth colorizer for pretty visualization of depth data
rs2::colorizer color_map;

rs2::config realSenseConfig;

realSenseConfig.enable_stream(RS2_STREAM_INFRARED, 640, 480, RS2_FORMAT_Y8, 30);
realSenseConfig.enable_stream(RS2_STREAM_ACCEL, RS2_FORMAT_MOTION_XYZ32F, 250);
realSenseConfig.enable_stream(RS2_STREAM_GYRO, RS2_FORMAT_MOTION_XYZ32F, 400);

std::atomic<bool> frameArrived = false;

rs2::pipeline realsensePipe;
auto pipelineProfile = realsensePipe.start(realSenseConfig, [&](rs2::frame frame)
{
    // We can only save video frames as pngs, so we skip the rest
    if (auto frameset = frame.as<rs2::frameset>())
    {
        for (const auto &fr : frameset)
        {
            if (auto vf = fr.as<rs2::video_frame>())
            {
                std::cout << "Video frame arrived" << std::endl;

                auto stream = frame.get_profile().stream_type();
                // Use the colorizer to get an rgb image for the depth stream
                if (vf.is<rs2::depth_frame>()) vf = color_map.process(frame);

                // Write images to disk
                std::stringstream png_file;
                png_file << "rs-save-to-disk-output-" << vf.get_profile().stream_name() << ".png";
                stbi_write_png(png_file.str().c_str(), vf.get_width(), vf.get_height(),
                    vf.get_bytes_per_pixel(), vf.get_data(), vf.get_stride_in_bytes());
                std::cout << "Saved " << png_file.str() << std::endl;

                // Record per-frame metadata for UVC streams
                std::stringstream csv_file;
                csv_file << "rs-save-to-disk-output-" << vf.get_profile().stream_name()
                    << "-metadata_1.csv";
                metadata_to_csv(vf, csv_file.str());

                frameArrived = true;
            }
        }
    }
});

while (!frameArrived)
{
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
    std::cout << "Waiting for depth frame" << std::endl;
}

realsensePipe.stop();

// Declare RealSense pipeline, encapsulating the actual device and sensors
rs2::pipeline pipe;
// Start streaming with default recommended configuration
pipe.start();

// Capture 30 frames to give autoexposure, etc. a chance to settle
for (auto i = 0; i < 30; ++i) pipe.wait_for_frames();

// Wait for the next set of frames from the camera. Now that autoexposure, etc.
// has settled, we will write these to disk
for (auto&& frame : pipe.wait_for_frames())
{
    // We can only save video frames as pngs, so we skip the rest
    if (auto vf = frame.as<rs2::video_frame>())
    {
        auto stream = frame.get_profile().stream_type();
        // Use the colorizer to get an rgb image for the depth stream
        if (vf.is<rs2::depth_frame>()) vf = color_map.process(frame);

        // Write images to disk
        std::stringstream png_file;
        png_file << "rs-save-to-disk-output-" << vf.get_profile().stream_name() << ".png";
        stbi_write_png(png_file.str().c_str(), vf.get_width(), vf.get_height(),
                       vf.get_bytes_per_pixel(), vf.get_data(), vf.get_stride_in_bytes());
        std::cout << "Saved " << png_file.str() << std::endl;

        // Record per-frame metadata for UVC streams
        std::stringstream csv_file;
        csv_file << "rs-save-to-disk-output-" << vf.get_profile().stream_name()
                 << "-metadata_2.csv";
        metadata_to_csv(vf, csv_file.str());
    }
}

return EXIT_SUCCESS;

}
catch(const rs2::error & e)
{
std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;
return EXIT_FAILURE;
}
catch(const std::exception & e)
{
std::cerr << e.what() << std::endl;
return EXIT_FAILURE;
}`

The two generated files have the following content:
No 1.:
Stream,Infrared
Metadata Attribute,Value
Time Of Arrival,1557408861706
Backend Timestamp,1557408861682
Actual Fps,30

No 2.:
Stream,Depth
Metadata Attribute,Value
Frame Counter,31
Frame Timestamp,2011290813
Sensor Timestamp,2011289961
Actual Exposure,1704
Gain Level,16
Auto Exposure,1
Time Of Arrival,1557408863824
Backend Timestamp,1557408863794
Actual Fps,30
Frame Laser Power,150
Frame Laser Power Mode,1
Exposure Priority,1
Exposure Roi Left,160
Exposure Roi Right,1120
Exposure Roi Top,90
Exposure Roi Bottom,630

Basically, the issue is that I don't get the "Sensor Timestamp" metadata when I start the RealSense in callback mode.

Any help is much appreciated.

[ev-mp]

@ztorok , according to the log the metadata is properly configured.

But there are certain things in the code that need to be addressed/clarified:

• The main loop is executed twice with two different configurations:
  The first time IR_Accel_Gyro are configured and the frames are pulled asynchronously:

realSenseConfig.enable_stream(RS2_STREAM_INFRARED, 640, 480, RS2_FORMAT_Y8, 30);
realSenseConfig.enable_stream(RS2_STREAM_ACCEL, RS2_FORMAT_MOTION_XYZ32F, 250);
realSenseConfig.enable_stream(RS2_STREAM_GYRO, RS2_FORMAT_MOTION_XYZ32F, 400);

The second time you run with Librealsense-recommended config which selects Depth+IR+Accel+Gyro . This time you work with synchronous pipe.wait_for_frames:

// Declare RealSense pipeline, encapsulating the actual device and sensors
rs2::pipeline pipe;
// Start streaming with default recommended configuration
pipe.start();
...

Is this on purpose, and if so - what is it?

1. In both segments there is UVC frames filter that discards all the IMU samples:

     for (const auto &fr : frameset)
        {
            if (auto vf = fr.as<rs2::video_frame>())
            {

    for (auto&& frame : pipe.wait_for_frames())
    {
        // We can only save video frames as pngs, so we skip the rest
        if (auto vf = frame.as<rs2::video_frame>())
        {

2. In both cases the output is written to the same files, meaning that the results produced by the first loops are overwritten.
3. The output files confirm that the metadata is properly set, so using the standard frame.get_timestamp() API should be sufficient to provide all timestamps in the same (HW) domain.
   You can get an additional confirmation by running realsense-viewer and checking the timestamp domain online, but based on the logs provided, imho, it is not necessary:
   
4. The attached code does not use the frame.get_timestamp() API, and if it did - most certainly you'd see that all timestamps are assigned to the same time domain. So it is not clear how you've determined the opposite.

There is a good chance that after taking a second look into the code you'll be able to resolve the issue.
And in case that it doesn't - the most productive way is to consolidate this fragmented demo into a single workable example that substantiates the claim and submit for review.

[ztorok]

Thanks @ev-mp.
Probably I wasn't specific enough. Actually, what I'm trying to do is to use the IR camera and the IMU of the RealSense D435i for visual inertial SLAM. Eventually I'll enable depth information to get a 3D reconstruction.
For this, I need synchronised image and IMU data with common timestamp bases and high frequency IMU data. This is why I want to use the asynchronous mode.

The problem is that the frame.get_timestamp() function returns inconsistent timestamps for motion and video frames. See following log:
Stream ID = 6 name: Gyro timestamp = 23001.871 Domain: Hardware Clock ms arrival: 1557726554221 ms
Stream ID = 5 name: Accel timestamp = 23043.71 Domain: Hardware Clock ms arrival: 1557726554222 ms
Stream ID = 6 name: Gyro timestamp = 23004.366 Domain: Hardware Clock ms arrival: 1557726554225 ms
Stream ID = 2 name: Infrared 2 timestamp = 1557726554224.93 Domain: System Time ms arrival: 1557726554224 ms
Stream ID = 5 name: Accel timestamp = 23044.201 Domain: Hardware Clock ms arrival: 1557726554226 ms
Stream ID = 6 name: Gyro timestamp = 23006.86 Domain: Hardware Clock ms arrival: 1557726554227 ms
Stream ID = 5 name: Accel timestamp = 23044.691 Domain: Hardware Clock ms arrival: 1557726554230 ms

As you can see, the motion frame timestamp is generated from hardware clock, whereas the IR frame gets system time. I generated this log basically with the same code as my asynchronous example by printing out the frame information in the callback:

std::cout << "Stream ID = " << frame.get_profile().unique_id() << " name: " << frame.get_profile().stream_name() << " timestamp = " << std::setprecision(15) << timestamp.count() / 1000.0 << " Domain: " << frame.get_frame_timestamp_domain() << " ms arrival: " << timeOfFrameArrival / 1000.0 << " ms" << std::endl;

I was trying to use the rs-save-to-disk example to easily reproduce my issue and noticed that it can access the metadata I need for the video frame, which is the sensor timestamp. However, it seems to me that difference in the amount of metadata wasn't because of the synchronous/asynchronous modes, but because of the different streams I used: depth vs. IR.

[ev-mp]

@ztorok hi,
Can you take a snapshot of the realsense-viewer with Depth+IR streaming and the stream info is turned on?
It is possible that when following the metadata enablement in #3205 the process did not finish correctly. To verify it you can rerun the registry script:

.\realsense_metadata_win10.ps1

Inspecting the script's output will tell whether the initial installation went well by reporting that no changes were needed. Otherwise you'll see that new registry keys were generated.

After running the script follow the instructions, replug the camera and repeat the timestamps check with realsense-viewer

[ztorok]

@ev-mp Thanks.
I can't find the option to enable IR stream in the realsense-viewer. I can only enable depth and RGB streams.

Unfortunately, my PC doesn't allow the execution of the registry script, even though I started PowerShell as administrator:
PS D:> .\realsense_metadata_win10.ps1 -op install
.\realsense_metadata_win10.ps1 : File D:\realsense_metadata_win10.ps1 cannot be loaded because running scripts is disabled on this system. For more information,
see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1

• .\realsense_metadata_win10.ps1 -op install

•   + CategoryInfo          : SecurityError: (:) [], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess
  
  

So I followed the manual prodedure.

[ev-mp]

@ztorok ,
1 - There installation guide describes how to troubleshoot this error - follo the link.

PS > Set-ExecutionPolicy RemoteSigned

2 - To enable IR stream(s) in the viewer hover with the mouse and expand the Stereo Module entry:

[ztorok]

@ev-mp
Ah, thanks, the options are visible before starting the depth stream :)

So, I ran the script, but it didn't seem to change anything:

2 connected RealSense devices were found:
USB\VID_8086&PID_0B3A&MI_00\6&3ac29587&0&0000
USB\VID_8086&PID_0B3A&MI_03\6&3ac29587&0&0003

Processing Registry branch HKLM:\SYSTEM\CurrentControlSet\Control\DeviceClasses{e5323777-f976-4f5b-9b55-b94699c46e44}
There are 4 total devices listed
2 of them are Intel Realsense
Device USB\VID_8086&PID_0B3A&MI_00\6&3ac29587&0&0000: skiping - metadata key already exists
Device USB\VID_8086&PID_0B3A&MI_03\6&3ac29587&0&0003: skiping - metadata key already exists

Processing Registry branch HKLM:\SYSTEM\CurrentControlSet\Control\DeviceClasses{65E8773D-8F56-11D0-A3B9-00A0C9223196}
There are 8 total devices listed
2 of them are Intel Realsense
Device USB\VID_8086&PID_0B3A&MI_00\6&3ac29587&0&0000: skiping - metadata key already exists
Device USB\VID_8086&PID_0B3A&MI_03\6&3ac29587&0&0003: skiping - metadata key already exists

Task Completed

And the IR streams show system time, whereas the depth hardware clock:

[ev-mp]

The snapshot suggests that the metadata registry key for IR endpoint was not set properly, and since it was done manually - there are chances for typos/mismatches.
Please remove all currently-installed registry keys and make a fresh install using the automation script:

PS > .\realsense_metadata_win10.ps1 -op remove_all
PS > .\realsense_metadata_win10.ps1 -op install_all

Then once again - reconnect the camera and check the timestamps.

[ztorok]

Ok, I have removed all

MetadataBufferSizeInKB0

Entries which I added and ran the script with remove all and install all options. I verified that install_all added the registry entries. Then I disconnected and connected the RealSense, but the result is still this:

[ev-mp]

@ztorok, according to the report

Ok, I have removed all
MetadataBufferSizeInKB0

there are discrepancies - each registry entry two keys should be set: MetadataBufferSizeInKB0 MetadataBufferSizeInKB1 as described in the manual:

Add DWORD 32bit value named MetadataBufferSizeInKB0 with value 5.
Add an additional DWORD 32bit value named MetadataBufferSizeInKB1 with value 5 for RS400 device

It seem that after the manual addition/removal of registry entries there are still some leftovers/inconsistencies that prevent the automation script to run properly.
Please manually review all the affected keys in the registry, revert them to the original state by removing all MetadataBufferSizeInKB0, MetadataBufferSizeInKB1 values and then rerun the automation script.
Note that the keys are located in two separate registry branches (manual):

HKLM\SYSTEM\CurrentControlSet\Control\DeviceClasses{e5323777-f976-4f5b-9b55-b94699c46e44}
HKLM\SYSTEM\CurrentControlSet\Control\DeviceClasses{65E8773D-8F56-11D0-A3B9-00A0C9223196}

If all the above fails then I'd suggest to revert the registry to the state before the changes by using Windows recovery points and then again rerun the automation script.

[ztorok]

@ev-mp Thanks, I can confirm that by adding the missing MetadataBufferSizeInKB1 entries in the registry solves the wrong timestamp domain.
The only issue I have left is that I can't get the script do the work for me. I added the missing MetadataBufferSizeInKB1 entries everywhere manually, where I found the MetadataBufferSizeInKB0 entry.

[ev-mp]

@ztorok , thanks for the update.
The script should run seamlessly on an unmodified registry. You can try it on another PC for verification.
Let us know if you have further issues with this

[Jxpenggf]

I also encountered a similar problem. I used the rs-callback in the example to obtain IMU and depth data, and found that when printing their timestamps at the same time, the depth data is about 25-30ms earlier than the IMU data.why?How to make them aligned