Problems in rs-convert

[YangJiao1996]

Required Info
Camera Model	D435
Firmware Version	05.10.13.00
Operating System & Version	Windows 10
Platform	PC
SDK Version	2.18.0
Language	C++

Issue Description

Hi there,

I have used the RealSense Viewer to record the streams as videos and then tried to extract the frames into .csv and .png files with the rs-convert tool. However, the utility tool sometimes doesn't work for me.

In most situations, the tool drops frames. I cannot convert all the frames through this tool. Sometimes the tool even crashes or get stuck when executing.
There are some log files generated by the tool, and in the log files are the warning messages like this:

 24/01 17:58:59,736 WARNING [16332] (playback_device.cpp:198) Playback device does not provide a matcher
 24/01 17:58:59,751 WARNING [1756] (playback_device.cpp:198) Playback device does not provide a matcher
 24/01 17:58:59,755 WARNING [15868] (playback_device.cpp:198) Playback device does not provide a matcher

There might be something wrong with the playback_device.cpp.

But I looked through the section that generates the warning messages but cannot figure out what it means:

std::shared_ptr<matcher> playback_device::create_matcher(const frame_holder& frame) const
{
    //TOOD: Use future implementation of matcher factory with the device's name (or other unique identifier)
    LOG_WARNING("Playback device does not provide a matcher");
    auto s = frame.frame->get_stream();
    return std::make_shared<identity_matcher>(s->get_unique_id(), s->get_stream_type());
}

I also noticed that @marcovs posted a similar issue #1919 and that issue seemed to be solved. However, I believed that I still have the same issue as marcovs with the 2.18.0 version of librealsense.

I think this issue could be quite critical because using RealSense Viewer to take the videos and using rs-convert to extract frames could be the most convenient way to get the data directly from the camera. It could make things much harder for the users of the camera if this issue is such widespread.

Thank you,
Yang

[codesian]

Hi @YangJiao1996
@UnaNancyOwen has develop a great tool to convert streams from bag files to images, using opencv. Works great!

[YangJiao1996]

Hi @codesian
Thank you for the suggestions. I also tried his tool for converting the bag files, but it still has some issues in synchronizing frames.

My current solution is developing a tool to directly save the streams as images (also using OpenCV) instead of bag files. It works fine now (at least the frames from different streams are synchronized, according to the frame numbers and timestamps). I may post the tool to Github once I think it is ready to be released.

[UnaNancyOwen]

Hi @YangJiao1996
The discontinuity frame numbers is caused by your bug file not including continuity frames.
Perhaps, It was dropped when recording.
The number of frames included in your bag file can be confirmed with RealSense Rosbag Inspector (rs-rosbag-inspector.exe). Also, I recommend that record with a machine with sufficient performance.

In general, the frame numbers do not match in each streams because they are not synchronized.
My tool outputs the original data as it is. Probably rs-convert will be the same.
It is counts-up since each sensor started streaming. I think it is nonsense to figure whether synchronized by frame numbers.

[YangJiao1996]

Hi @UnaNancyOwen
Thank you for the explanations. I do agree that the frames could be dropped during recording, but I am a little bit confused about your "synchronized" part.

I checked the RealSense API for the get_frame_number() and get_timestamp() methods, and it is said that both the frame numbers and timestamps are "in milliseconds since the device was started". I tend to believe that the frame numbers or timestamps could be used to determine whether the streams are synchronized.

The issues with rs-convert could be related to the unsynchronized frames. I will do more recordings with different machines to examine what actually happens when the frames are dropped. Also, thank you again for your tools and explanations!

[UnaNancyOwen]

I think that the description of get_frame_number() in API reference is incorrect.
Probably, the following description is correct.

the frame number of the frame, in milliseconds since the device was started

[neumannjan]

@YangJiao1996 Please add support for saving as csv/binary. Thanks.

[RealSenseCustomerSupport]

---

Hi YangJiao1996,

Regarding to the API description, what UnaNancyOwen corrected is right.

For sync'ing up frames from a bag file (assuming there is no frame drop during recording to bag files), besides using backend timestamp to decide frames from sensors, another choice could be with HW sync enabled and record to bag file.

Thanks!

[frankfalse]

Hi @YangJiao1996.
When you use the tool rs-convert, does it terminate its execution or not?
In my Ubuntu platform it shows the number of png files extracted from bag file, but doesn't end its execution and I must close it by the combination of keys Ctrl+C.

I better explain my problem in this issue: #3297 (comment)

Thanks!

[YangJiao1996]

Hi @frankfalse ,
No, sometimes it didn't terminate by itself and I also had to close it with Ctrl+C.
For now, the best solution to convert the bag files into other formats is using @UnaNancyOwen's tool.

[YangJiao1996]

For the rs-convert tool, I did more recordings with my camera and tested them with rs-convert, and only one of them caused the crash. It could be related to the compression format (1z4) of rosbag, or the state of the camera when recording.

[frankfalse]

Thank you for your replay.
I'll send you some feedback!

[frankfalse]

Hi @YangJiao1996 rs_bag2image save depth frame in black and white png files, while rs-convert save colorized images!
rs-convert seems to be faster than rs-convert and doesn't remain open at the end of its execution, but the result of its processing is not enough for me.

If I have made a mistake please help me!

[YangJiao1996]

Hi @frankfalse ,
The depth data are actually stored in millimeters as described in the documents.

What rs-convert does is to "colorize" the depth data using the colorizer() method, and what rs_bag2image does is to map the depth data in a 255 gray-scale. So in both of the tools the raw depth data are converted into another format. You can utilize the OpenCV functions or built-in rs2 SDK methods for other output formats.

[frankfalse]

Hi @YangJiao1996 thank for your interesting reply.
I have studied the program rs_bag2image but I don't reach my goal.

Instead I have found a workaround for the problem of rs-convert.
If you want you can read the issue #3297 and give me some suggestions.

[RealSenseCustomerSupport]

---

Hi YangJiao1996,

Wonder if you have any update on this one? Do you still need help on it?
And as noted in RS2.19.1, github #3297 has been fixed.

Thanks!

[YangJiao1996]

Hi @RealSenseCustomerSupport ,
RS2.19.1 solved issue #3297, but does not work on my bag file. I also did some changes on the rs-convert (see #3564) that works. You can take a look on my push request. If that is approved I will say the issue is solved.

[RealSenseCustomerSupport]

---

Hi YangJiao1996,

Thanks for the update.
PR 3564 has been tagged for v2.20.0, and should be reviewed and decided soon.

Thanks!

[YangJiao1996]

Hi @RealSenseCustomerSupport,
Also I got a test file for the update. This bag file is actually the one which caused the early termination of rs-convert and made me post this issue.

[RealSenseCustomerSupport]

---

Hi YangJiao1996,

Thanks for the updates and sharing a test file.

Cheers!