D435i: How to generate PCL Point Cloud at faster rate?

[milan-r-shah]

Required Info
Camera Model	D435i
Firmware Version	05.12.07.00
Operating System & Version	Linux (Ubuntu 18.04)
Kernel Version (Linux Only)	5.3.0-28-generic
Platform	PC
SDK Version	2.0 (v2.36.0)
Language	C++
Segment	others

Issue Description

Hi,
I have been using PCL wrapper code: rs-pcl-color.cpp by RealSense to generate the point cloud. However, based on my analysis, I found that to create each new instance of PCL Point Cloud, the code takes on average 1200 ms which is a very slow rate. How can I get a faster rate?

I tried changing

cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_BGR8, 30);
cfg.enable_stream(RS2_STREAM_INFRARED, 1280, 720, RS2_FORMAT_Y8, 30);
cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 30);

to

cfg.enable_stream(RS2_STREAM_COLOR, 848, 480, RS2_FORMAT_BGR8, 30);
cfg.enable_stream(RS2_STREAM_INFRARED, 848, 480, RS2_FORMAT_Y8, 30);
cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 30);

i.e. decreased the width & height but the rate at which PCL Point Cloud gets generated is more or less the same!

[MartyG-RealSense]

Hi @milan-r-shah When using CMake, did you include the build flag -DCMAKE_BUILD_TYPE=release to build with compiler optimizations enabled, please?

[milan-r-shah]

Thanks @MartyG-RealSense for such a quick response.

I tried building my code using -DCMAKE_BUILD_TYPE=release flag but the performance improvement is not significant. Now, it takes around 1050 ms to generate the point cloud.

Are there any ways to leverage the GPU or use multithreading in CPU or any other ways to generate the PCL PointCloud at faster rate?

[MartyG-RealSense]

Do you need an infrared stream for your PCL point cloud, please? Typically, you would just have depth and color in a point cloud that has color mapped to the depth points. What effect does it have if you remove the cfg definition line for infrared?

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/pcl/pcl-color/rs-pcl-color.cpp#L169

In regard to GPU acceleration: if you are using a librealsense SDK script then you should be able to use the SDK's GLSL processing block feature to offload work from the CPU to the GPU. It may not produce a noticable improvement on low-end configuration computers though. Details about GLSL can be found in the link below:

#7415 (comment)

[milan-r-shah]

Thanks again Marty. I tried disabling the INFRARED stream by commenting cfg.enable_stream(RS2_STREAM_INFRARED, 1280, 720, RS2_FORMAT_Y8, 30); but the performance improvement is almost as same as before i.e. it's still taking around 1050 ms.

As I'm still trying to understand Intel RealSense Cross Platform API, could you please tell me changing the number of times the following loop runs, would make any difference?

        // Wait for frames from the camera to settle
        for (int i = 0; i < 30; i++) {
            auto frames = pipe.wait_for_frames(); //Drop several frames for auto-exposure
        }

Reference:

librealsense/wrappers/pcl/pcl-color/rs-pcl-color.cpp

Line 201 in d8f5d42

for (int i = 0; i < 30; i++) {

I mean how computationally extensive pipe.wait_for_frames() method is?

Yup, I have been using SDK's script only i.e. rs-pcl-color.cpp. So, thanks for letting me know about GLSL. I would take a look and try that out right away!

[MartyG-RealSense]

The section of code that you quoted about waiting for the auto-exposure to settle just skips the first several frames after the stream starts, so wouldn't have an effect on performance after that.

wait_for_frames() is generally the best method to use for single-camera applications. Its purpose is to block until a complete frame is available.

The link below explains the diference between wait_for_frames() and other types of '_for_frames' operations.

#2422 (comment)

I look forward to hearing about your GLSL test results. Good luck!

[milan-r-shah]

Thanks Marty. As I'm still trying to understand the API, could you please tell me whether that quoted code snippet is required for each instance of a point cloud or only in the beginning? I mean I'm continuously generating point cloud by putting some code from rs-pcl-color.cpp in the infinite while loop. Right now, that quoted code snippet is in my while loop but to get a faster rate, can I put that outside of the while loop?

Regarding GLSL, I went through rs-gl sample code but it looks like it's not generating PCL Point Cloud. On the other hand, I'm relying on rs-pcl-color.cpp because I need PCL point cloud. So, how can I do that?

In fact, rs-gl.cpp uses example.hpp (https://github.com/IntelRealSense/librealsense/blob/master/examples/gl/CMakeLists.txt#L22) file of the SDK so how do I have to modify CMakeLists.txt in my standalone project?

[MartyG-RealSense]

If you are concerned about the several-frame skip then you can leave it out as it is totally optional. It is simply useful for RealSense users who want to reduce the possibility of incorrect data in the first several frames.

I see that rs-pcl-color makes use of the point cloud processing block.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/pcl/pcl-color/rs-pcl-color.cpp#L157

You should therefore be careful about using rs2::pointcloud pc in a while loop if you are doing so, as using the pointcloud processing block in a loop has a known issue that can result in a memory leak (degrading performance over time as memory is consumed due to not being released).

#2860

As far as I know, GL can be applied to the functions in your script that start with rs2::. For example, rs2::pointcloud pc becomes rs2::gl::pointcloud pc (in other words, insert gl:: after rs2::

[MartyG-RealSense]

Hi @milan-r-shah Do you still require support for this case, please? Thanks!

[milan-r-shah]

Hi @MartyG-RealSense
Sorry for the late reply but I got a little bit more occupied with some other tasks. I didn't get a chance to try out your last comment. However, I'm planning to try that out in a couple of weeks. Would it be possible for you to keep this issue open until then?
Thank you so much in advance!

[MartyG-RealSense]

Yes, I can keep it open til then. Good luck!

[MartyG-RealSense]

Making a note to keep this case open for a further time period.

[MartyG-RealSense]

Making a note to keep this case open for a further time period.

[MartyG-RealSense]

Making a note to keep this case open for a further time period.

[MartyG-RealSense]

Hi @milan-r-shah Do you have an update to provide about this case please? Thanks!

[milan-r-shah]

Hi @MartyG-RealSense
Apologies for the late reply. Also, I have been working on some other tasks and haven't got a chance to dive deep into this issue/task yet. Sorry about that. Is there any way that you close this issue for now and once I start working on it, I reopen it?
Thanks in advance :)

[MartyG-RealSense]

Yes, I can close it for you until you are ready. You can also close cases yourself using the Close Issue button under the comment box. Good luck!