How can I back reproject RGBD data to point cloud after resizing the images?

[zc08]

Required Info
Camera Model	D455
Firmware Version	05.12.06.00
Operating System & Version	Ubuntu 18.04.3
Kernel Version (Linux Only)	5.4.0-48-generic
Platform	PC
SDK Version	2.36.0
Language	C/python
Segment	Robot

Issue Description

I'm implementing some obstacle detection algorithm for android devices.
For efficiency reasons, I intend to resize both the rgb image and depth image to 1/4 of their original sizes, and then back-reproject the RGBD data to obtain 3D point cloud.
In the rs-pointcloud example, the back-projection and rgb-depth alignment APIs are demonstrated, but I don't know excactly how to do this after resizing.

In theory, I just need to adopt the normal back-projection process, with camera intrinsics (specifically focal length & principal point) rescaled accordingly. I dived into the code a bit, and found that the function
const float3 * pointcloud::depth_to_points(rs2::points output, const rs2_intrinsics &depth_intrinsics, const rs2::depth_frame& depth_frame, float depth_scale)
was basically doing what I want. But examining and picking out all the code and manually doing the modification seems cumbersome and error-prone.

Is there a simple way to do the reprojection after resizing the images ? (E.g. by modifying the intrinsic parameters inside the depth frame?) Thanks in advance!

[MartyG-RealSense]

Hi @zc08 It sounds as though you want to go through the following steps:

1. Reduce the image scale using a post-processing decimation filter
2. Align depth to color
3. Deproject the aligned image to generate a point cloud

Is this correct, please? If so, and you are able to use C++ (as the reference to rs-pointcloud would suggest, the scripting in the C++ discussion in the link below seems to cover the implementation of the three steps above (decimation, align, deprojection).

#5403

Alternatively, the link below has Python scripting for generating a decimated and aligned point cloud without using deproject (using pc.calculate instead):

#6718

[zc08]

Thank you for the reply!

1. Reduce the image scale using a post-processing decimation filter
2. Align depth to color
3. Deproject the aligned image to generate a point cloud

Yes, that's basically what I want to do. With one thing different, in step 1, I would like to pick the closest depth value instead of median value as the rs-depth-filter sample does, rather than decimation filter. (But I can try decimation filter too.)

And another question, does decimation filter alter the intrinsics, so that the deproject process is correct? Thanks!

[zc08]

I noted that the documentation mentioned that.

After the resulted frame is produced, the frame intrinsic parameters are recalculated to compensate for the resolution changes.

Thanks!

[MartyG-RealSense]

Thanks very much for the quote about intrinsics confirming the answer to your decimation question.

A setting that may be of interest to you if you are concerned with the confidence values of pixels is secondpeakdelta, which can be set by loading a json camera configuration file. The setting is described in Intel's paper about improving depth results on drones.

https://dev.intelrealsense.com/docs/depth-map-improvements-for-stereo-based-depth-cameras-on-drones#section-b-depth-camera-settings

[MartyG-RealSense]

Hi @zc08 Do you require further assistance with this case, please? Thanks!

[zc08]

No, thank you !