[Question] How to use rs2_project_color_pixel_to_depth_pixel in python to convert 1 point from color coordinate into depth coordinate

[gachiemchiep]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ D435 }
Firmware Version	05.11.11.100
Operating System & Version	Ubuntu 16
Kernel Version (Linux Only)	4.4.0-170-generic
Platform	4.4.0-170-generic
SDK Version	{ 2.21.0 }
Language	{python }
Segment	{Robot }

Issue Description

Hello guys.

Does anyone know how to use the rs2_project_color_pixel_to_depth_pixel method in python?

My current situation is as followed:

1. I takes color and depth image from realsense camera
2. Run some object detection on color frame
3. Then by detected object bounding box, I estimate the distance from camera into object

It's similar to this script distance_to_object

Because of the aligment from depth frame into color frame took a long time, the total fps did drop down. I want project only the coordinate of bounding box ( x1, y1) from color frame into depth frame (x2, y2).

I found some c++ method at 2265 and the wrapper for that method at pyrsutil_8cpp_source. But don't know how to use that method.

Please help/

[kafan1986]

Take a look at this post for similar query.
#5440 (comment)

[gachiemchiep]

@kafan1986 Thanks I will try your idea soon

[gachiemchiep]

@kafan1986
How do you get the value for * float(depthMinMeters), (float)depthMaxMeters* .
I'm using D435 so will depthMinMeters=0.11 and depthMaxMeters=10 be correct? These values were taken from intel d435

[kafan1986]

Yes it will generally be good enough. This is real world dependent, how far are you trying to measure?

[gachiemchiep]

@kafan1986
About 30~50cm away.
Do i need to calculate using depth_frame.
The good news is in librealsense 2.23, i can use rs2_project_color_pixel_to_depth_pixel in python and it was executed successful.

[kafan1986]

@kafan1986
About 30~50cm away.
Do i need to calculate using depth_frame.
The good news is in librealsense 2.23, i can use rs2_project_color_pixel_to_depth_pixel in python and it was executed successful.

If all your values are supposed to be within that range then you can lower your depthMaxMeters
distance to something like 1.0

[gachiemchiep]

@kafan1986
Hey thank you for your advice.
Here's snip of my code. Maybe it'll be useful for someone someday

# There values are needed to calculate the mapping
depth_scale = profile.get_device().first_depth_sensor().get_depth_scale()
depth_min = 0.11 #meter
depth_max = 1.0 #meter

depth_intrin = profile.get_stream(rs.stream.depth).as_video_stream_profile().get_intrinsics()
color_intrin = profile.get_stream(rs.stream.color).as_video_stream_profile().get_intrinsics()

depth_to_color_extrin =  profile.get_stream(rs.stream.depth).as_video_stream_profile().get_extrinsics_to( profile.get_stream(rs.stream.color))
color_to_depth_extrin =  profile.get_stream(rs.stream.color).as_video_stream_profile().get_extrinsics_to( profile.get_stream(rs.stream.depth))

color_points = [
    [400.0, 150.0],
    [560.0, 150.0],
    [560.0, 260.0],
    [400.0, 260.0]
]
for color_point in color_points:
   depth_point_ = rs.rs2_project_color_pixel_to_depth_pixel(
                depth_frame.get_data(), depth_scale,
                depth_min, depth_max,
                depth_intrin, color_intrin, depth_to_color_extrin, color_to_depth_extrin, color_point)

Here's the output. It's quite correct.

If the depth value for bottom right point became invalid, then the nearby valid point will be selected. But i think this is the desired behavior of rs2_project_color_pixel_to_depth_pixel method.

[subbhyd]

rs2_project_color_pixel_to_depth_pixel

i am getting following error
AttributeError: module 'pyrealsense2' has no attribute 'rs2_project_color_pixel_to_depth_pixel'

[gachiemchiep]

@subbhyd You need the realsense version larger than 2.31.0 , so please build re-build your realsense library to match that version

See the source code at : pyrsutil.cpp

[kithib]

@gachiemchiep Your work is great, I would like to ask what is the significance of the first parameter , depth_frame.get_data().
I used this in C++ . depth_frame.get_data() return void* ,but rs2_project_color_pixel_to_depth_pixel() needs const uint16_t* data.

[gachiemchiep]

@kithib
Did you try static_cast to converse void type into uint16 type?
Something like this should work

rs2_project_color_pixel_to_depth_pixel(point, static_cast<uint16_t*>depth_frame.get_data(), ....)

The cpp api is like this :
https://intelrealsense.github.io/librealsense/doxygen/rsutil_8h.html#af04b7b1ba50b1d9e03089d1e5186d2f8

[kithib]

@gachiemchiep Thank you very much, with your help I have solved this problem

[westpilgrim63]

hi , i used rs2_project_color_pixel_to_depth_pixel to project a point(the red point on the right rgb image) on rgb image to depth image(the red point on the left depth image),but i found the 2d coordinate on depth image in inaccurate ,how can i solve this?

[westpilgrim63]

@gachiemchiep

[gachiemchiep]

@westpilgrim63 please check whether depth at that point is valid.
If the depth is not valid then nearby point will be selected.

[yijionglin]

@gachiemchiep @westpilgrim63

To those who are going to use the provided snip of code, make sure to swap the order of the input of color_to_depth_extrin and depth_to_color_extrin. (see the similar C++ API document)

The first image is from the original code and the second one is from the modified one: