How to save the .bag file into rgb video and depth video?

[erwangccc]

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ L515 }
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	{Win 10
Platform	PC
SDK Version	{ 2.50.0 }
Language	{C }

Issue Description

I have recorded and saved some aligned video into bag file, and i want to convert the bag file into rgb video, depth file and IR video, so how to do this?
Thanks in advance!

[MartyG-RealSense]

Hi @erwangccc At #9826 the saving of RGB video with C++ code or with screen-capture software is discussed.

Although the referenced scripting uses a live camera by default, scripts can be adapted to use a bag file as the data source instead by inserting an enable_device_from_file instruction directly before the pipe.start line. For example:

rs2::config cfg;
cfg.enable_device_from_file(<filename>);
pipe.start(cfg); 

A bag file's frames can be converted to other file formats (though not video) using the RealSense SDK's rs-convert tool.

https://github.com/IntelRealSense/librealsense/tree/master/tools/convert

If you have installed the full RealSense SDK on Windows using the Intel.RealSense.SDK-WIN installer program on the SDK 'Releases' page then a pre-built executable version will be already installed on your computer and can be found and run using instructions at the link below.

https://support.intelrealsense.com/hc/en-us/community/posts/360051132573/comments/360014080493

In regard to converting infrared to video, a small number of RealSense users have attempted this and found that whilst RGB and depth could be exported to video correctly, an infrared export typically resulted in an empty video file or a corrupted one that could not be played back. The subject has been discussed recently at #10645 (comment) and #10647

[erwangccc]

Hi, @MartyG-RealSense, thanks for your quick response!

If we save depth data to a video format, then we read it, will we loss the depth info?

To me, i need the depth data, RGB and IR videos which are aligned to depth data for training. So what's the best way to do this?

Thanks.

[MartyG-RealSense]

Yes, depth information would be lost. The same occurs when saving to the PNG image format. You can preserve depth information by exporting depth from a bag to raw image, ply point cloud data file, bin binary or csv text file with rs-convert.

If your goal is to align all three streams, I wonder whether it may be easier to do so within the RealSense SDK using the information at #5093 and then export the combined image to video.

[erwangccc]

OK, got it.
I have tried the rs-align samples and save images by stbi, but saving images by stbi is very slow, it even can not reach 5fps at my pc. In fact, i want save image for 20/30fps.

Simply, my goal is

1. Just align IR image and depth is ok, it may be aligned when the device is capturing.
2. 20/30fps.
3. Save depth info for real time.

Do you have any suggestions？Have you tried save images and depth by opencv?

Thanks.

[MartyG-RealSense]

Depth and IR do not really need to be aligned, as the left IR camera has the benefit of always being pixel-perfect aligned, calibrated, and overlapped with the depth map, and perfectly time-synchronized. This is because depth and left IR originate from the same sensor.

The slowness of stibi is discussed at the link below, where another RealSense user also achieved around 4 FPS with this method.

https://community.intel.com/t5/Items-with-no-label/Quickly-saving-an-image-sequence-from-D435/m-p/471329

Advice given in that link was to process the data on multiple threads. This approach has also been taken by other users who wanted to process camera data in real-time and simultaneously save video without slow-down. The subject is discussed at #10113 and #1286

You can certainly save color and depth images with OpenCV. If you need to stream and capture simultaneously though, multi-threading may be the best approach to take.

[erwangccc]

@MartyG-RealSense Thanks.

Maybe it will also be slow if i use opencv API to save images.

I recorded IR video and depth map and saved it into bag file. For now, i have convert to csv file by rs-convert, and how to convert IR images from bag file?

If it can be done, i can read IR images and then save to video, it's ok for me.

[MartyG-RealSense]

There is not a confirmed-working method of saving infrared to video with OpenCV's VideoWriter at the time of writing this as the file that is output is either empty or corrupted and non-playable (color and depth writing work fine) though.

In the link below, someone was able to export infrared to video and achieve a playable file if they used the OpenCV instruction cv2.VideoWriter_fourcc(*'mp4v')

https://stackoverflow.com/questions/71945367/how-to-properly-use-cv2-videowriter-to-write-depth-video-and-infrared-video-simi

[erwangccc]

OK, Thank you very much, i have write IR images to video by python API, I think mp4 video format is ok for IR data which is y8 data format.

And the last question. i want save depth image to "npy" or "csv" or other format via python code, i think directly saving the origin data which is not readable for human is not a good way

depth_frame = aligned_frame.get_depth_frame() depth_images = np.array(depth_frame.get_data())

and how to convert the origin depth data to the meter data. I realize that it's readable for the "csv" data which is converted by rs-convert.

Thanks

[MartyG-RealSense]

There is a Python script at #4934 (comment) for saving the color as a PNG and the depth as an array of scaled matrices in an .npy file.

The raw 16-bit depth values can be converted to real-world meters by multiplying the raw value by the camera's depth unit sclae. On L515 this scale value is 0.000250

[erwangccc]

@MartyG-RealSense Thanks, I have read bag file and written IR videos and depth map (as meter)into npy file according to #4934 (comment) and read_bag_example.

And i found record video via Viewer may loss some frames. Sometimes it will loss 1/3 frames, sometimes it will loss just few frames which is acceptable. What are the main factors affecting the recording frame rate？

[MartyG-RealSense]

If the storage drive of the computer has slow speed then it may create a processing 'bottleneck' where it struggles to keep up with writing new frames that are being generated.

Having recording compression enabled may cause problems if the compression algorithm conflicts with the CPU of a particular computer.

If the CPU of the computer is over-burdened with processing and is near or at 100% utilization then that could also cause slowdown. You can check this on Windows by opening the Task Manager interface with Ctrl-Alt-Del and left-clicking on the Performance tab. You can then observe the demand that the current activity, such as recording, is placing on the computer.

Increasing the frame queue capacity of the pipeline can reduce the risk of frames being dropped by introducing latency into the system. For two streams, Intel suggests setting the queue capacity to '2' instead of the default '1'. The SDK's Frame Buffering Management documentation at the link below provides information about this and an example C++ script for setting a custom capacity value.

https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0#latency-vs-performance

Some RealSense users have found that using a value of '50' provides best performance. This may be an appropriate value for you to try as you are using three streams (depth, infrared and RGB).

[MartyG-RealSense]

Hi @erwangccc Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.

[pfcouto]

Hi @MartyG-RealSense I would like to bring this topic back to life. I have a depth camera, right now a d435i but later on I might use a L515. My camera is going to be mounted to a drone and will be powered by a raspberry pi or something. What I am about to say is my ideia, but if you have a different solution please share it with me. In the raspberry pi I will have a script that captures de depth and rgb streams and probably sends them to a RTMP server, I will then access both streams and with a CNN (convolutional neural network) I will recognize objects and send the processed stream to a website so that the user can see it.

I haven't tested what I am about to say, but I believe that it is correct. I will have problems with the stream because it is .bag, right? Do you think that I should send to the RTMP server two streams (depth and rgb) seperatly and then put them together using the timestamps (if possible) or is there a way to send the original stream directly to a RTMP? Is there a way to make a unique RGB-D stream on the raspberry pi and send this one?

I can only see this three options:

1. Send both streams to the RTMP server and then put them together using timestamps
2. Stream the original stream to the RTMP ( probably I will have problems later on)
3. Create a RGB-D stream (haven´t researched for this one).

One more possibility would be like in the picture, send both streams in one video file, then I will have to cut the black bars and align the streams. The example in the pictures uses a termal and not a depth camera, but you get the point.

To summarize is there a recommended method of streaming data from the D435i or L515?

Thanks!

[MartyG-RealSense]

Hi @pfcouto If your capture script was a librealsense script then it should treat a bag file as though it is a live camera, as a bag is like a video recording of raw camera data. If you do not define the data source as being a bag file though then the camera would be used as the live data source by default.

If your data source was a bag file then a librealsense script could retrieve the two individual RGB and depth streams from the bag file and then perform depth-color alignment on them in real-time to produce a combined image, if alignment is what you mean by 'put them together'.

If you are planning on using alignment and wish to remove a black 'letterbox' then color to depth alignment may achieve this, as aligning color to depth instead of depth to color should result in the combined image being stretched out to fill the screen instead of having a black border.

---

There is a Raspberry Pi project at the link below for streaming a RealSense camera using RTMP. It does not list which types of stream are captured though.

https://github.com/rootstream/realsense-rtmp-twitch

---

You could also investigate whether your goals could be achieved using RealSense's existing RTSP based networked streaming system that is compatible with Raspberry Pi 4.

https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras

[pfcouto]

HI @MartyG-RealSense, I haven't had a chance take a good look at those projects yet. However, I would like to ask you something. With the stream I need to send some metadata, such as GPS location and other metrics acquired by using a drone (the camera will be mounted in a drone). Do you know a good approach to send this with the stream? Or to send it separately and then align in according to the time? Have you seen a similar project?

I believe that the best way to stream is using RTMP. However, I don't know if the metadata will be lost in that process, I believe it will.

Thanks!

[MartyG-RealSense]

In researching your question I came across a live drone streaming commercial app called Dronestream that can transmit GPS coordinates. It can also save streams synced with maps and communications.

https://www.dronestream.io/

Another commercial product that supports sending video metadata, including GPS, is UgCS.

https://www.ugcs.com/video-streaming-with-ugcs

---

If you prefer to develop your own system, an approach discussed at the link below was sending GPS data via a GIS server.

https://forum.dji.com/forum.php?mod=redirect&goto=findpost&ptid=60761&pid=1571355