pipeline Frame Buffering Management in RealSense SDK 2.0

[SowisLiuYiming]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D415
Firmware Version	5.12.7.100
Operating System & Version	Ubuntu 16.04
Platform	PC
SDK Version	2.38.1

Issue Description

<Describe your issue / question / feature request / etc..>
I am investigating frame buffer management in pipelines. I saw the Web 。
https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0

I know the pipeline is a synchronized platform, it contains a frame queue that buffers the received frames.
The pipeline does not expose an API for setting the frame queue capacity, if the user wish to set the capacity to a value that is more compatible for his needs it can be done by changing the source code: pipeline_processing_block::pipeline_processing_block()
But I can't find this method in the code（librealsense-2.38.1\src）.

problem：pipeline_processing_block::pipeline_processing_block() Which file is this method in? How is it called?

[MartyG-RealSense]

Hi @SowisLiuYiming The frame queue capacity can be set with SDK script code.

C++ - see under the 'librealsense2' heading
https://dev.intelrealsense.com/docs/api-how-to#do-processing-on-a-background-thread

Python
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/depth_auto_calibration_example.py#L225-L228

[SowisLiuYiming]

@MartyG-RealSense
Thank you for your reply. I tried this fucation.
When a hub is used to connect multiple sensors, frames will sometimes be lost when obtaining frames. So we want to solve this problem with queues. However, the results are not ideal, and there are occasional times out when executing the pipeline.poll_for_frames method.
For example, test fetching frames 1000 times.
The 950 frame fetching time is less than 0.5s. --OK
The 50 frame fetching time is more than 0.5s. --NG
How can I improve this problem？

Here is my code：
`
rs2::pipeline pipeline;
rs2::frameset frameset;
rs2::frame unFilter;
const auto CAPACITY = 5; // allow max latency of 5 frames
rs2::frame_queue queue(CAPACITY);

std::thread t(& {
while (true)
{
if (queue.poll_for_frame(&unFilter))
{
unFilter.get_data();
}
}
});
t.detach();
for (int i = 0; i < 10; i++) {
bool isRet = pipeline.poll_for_frames(&frameset);
queue.enqueue(frameset.get_depth_frame());
queue.enqueue(frameset.get_color_frame());

if (!isRet) {
	std::this_thread::sleep_for(dura50);
}
else {
	break;
}

}
`

[MartyG-RealSense]

Some RealSense users have had good results with a frame queue capacity value of '50' instead of '5'.

Instead of sleeping for '50', do you get better results if you use for (int i = 0; i < 50; i++) { to do nothing for 50 frames.

It might look something like this:

if (!isRet) {
for (int i = 0; i < 10; i++) {
}
}

else {
break;
}

}

[SowisLiuYiming]

Changing the frame queue capacity value to "50" does not solve the problem.
isRet = pipeline.poll_for_frames(&frameset);It takes longer and longer for ”isRet“ to become True.
I want to solve pipeline.poll_for_frames(&frameset) delay problem.
The queue code I wrote does not solve this delay problem. And for much longer.
Is there a problem with any part of my code?

rs2::pipeline pipeline;
rs2::frameset frameset;
rs2::frame unFilter;
const auto CAPACITY = 50; // allow max latency of 5 frames
rs2::frame_queue queue(CAPACITY);

std::thread t([&]() {
    while (true)
    {
        if (queue.poll_for_frame(&unFilter))
        {
            unFilter.get_data();
        }
    }
});
t.detach();

bool isRet;
for (int i = 0; i < 100; i++) {
    isRet = pipeline.poll_for_frames(&frameset);
    queue.enqueue(frameset.get_depth_frame());
    queue.enqueue(frameset.get_color_frame());
    if (!isRet) {
        std::this_thread::sleep_for(dura50);
    }
    else {
        break;
    }
}

[MartyG-RealSense]

Is the bool isRet made true by the instruction bool isRet; ?

Assuming that the script has to pass through the 'unFilter' section of code before it can reach the instruction that makes isRet true, does performance improve if you comment out the unFilter.get_data() instruction to see whether that instruction is causing a delay?

if (queue.poll_for_frame(&unFilter))
        {
         //   unFilter.get_data();
        }
    }

[SowisLiuYiming]

// unFilter.get_data();
Commenting out this code can improve performance a bit.But it doesn't solve the original problem.
I think whether the queue needs to be released. How to release?
Or queue doesn't work for pipeline.poll_for_frames(& Frameset) in my code.

Original Code（"pipeline.poll_for_frames(&frameset) return true" timeout probability：0.5%）：

rs2::pipeline pipeline;
rs2::frameset frameset;
rs2::frame unFilter;

bool isRet;
for (int i = 0; i < 100; i++) {
    isRet = pipeline.poll_for_frames(&frameset);
    if (!isRet) {
        std::this_thread::sleep_for(dura50);
    }
    else {
        break;
    }
}
if (isRet) {
    auto vf = frameset.get_color_frame().as<rs2::video_frame>()
    ...
}

Now Code add queue（"pipeline.poll_for_frames(&frameset) return true" timeout probability：30%）：

rs2::pipeline pipeline;
rs2::frameset frameset;
rs2::frame unFilter;
const auto CAPACITY = 50; // allow max latency of 5 frames
rs2::frame_queue queue(CAPACITY);

std::thread t([&]() {
    while (true)
    {
        if (queue.poll_for_frame(&unFilter))
        {
            unFilter.get_data();
        }
    }
});
t.detach();

bool isRet;
for (int i = 0; i < 100; i++) {
    isRet = pipeline.poll_for_frames(&frameset);
    queue.enqueue(frameset.get_depth_frame());
    queue.enqueue(frameset.get_color_frame());
    if (!isRet) {
        std::this_thread::sleep_for(dura50);
    }
    else {
        break;
    }
}
if (isRet) {
    auto vf = frameset.get_color_frame().as<rs2::video_frame>()
    ...
}

[MartyG-RealSense]

The SDK has a C++ instruction called rs2_release_frame()

https://intelrealsense.github.io/librealsense/doxygen/rs__frame_8h.html#a1bdbdd207cda9d5dd015229bd0a7dc16

https://github.com/IntelRealSense/librealsense/blob/master/src/rs.cpp#L1024-L1028

Alternatively, performing a pipeline.Stop() releases all the resources in the pipeline.

[SowisLiuYiming]

Thanks for sharing the SDK. I have used both SDKS.
However, the "pipeline.poll_for_frames(& Frameset)" timeout issue is not resolved. Now trying "rs2::frame_queue" also fails.
I wonder if there are other ways, or what factors might influence it.

[MartyG-RealSense]

You mentioned earlier in this discussion that "When a hub is used to connect multiple sensors, frames will sometimes be lost when obtaining frames".

Did you build the SDK from source code with RSUSB backend set to True, please? If you did, then the RSUSB backend installation method is not best suited to use with multiple cameras. For multicam applications, It is better to perform a 'native' source code build (RSUSB = false) where the kernel is patched.

For a comparison of the advantages and disadvantages of these two installation methods, you can visit #5212 (comment) and scroll down to the section headed What are the advantages and disadvantages of using libuvc vs patched kernel modules?

[MartyG-RealSense]

Hi @SowisLiuYiming Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.