deprojection pixels to 3d point

[ghost]

Required Info
Camera Model	SR300
Operating System & Version	Win 10
Platform	PC
SDK Version	2.10.1

I am trying to get 3d point of objects using aligned color and depth frames that i have in order to measure those objects (length, width, etc...)

I am using rs.rs2_deproject_pixel_to_point in order to get 3d point of from depth image, but i am not quite sure what is the meaning of the values that returned and how to connect them to the real value of depth which corresponds to the pixel.

I am using the following code in order to construct a 3d point of pixel from depth frame

depth_sensor = profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(color_frame.profile)
depth_pixel = [100, 100]
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)

and the result is: [-5.814936230308376e-05, -3.9755548641551286e-05, 0.00012498664727900177]

what is the meaning of those values, and how can i use them on specific frames i saved on my disk? my depth frame is already aligned to the color frame (depth frame`s pixel is mapped to the same pixel in the rgb frame).
my goal is to find 3d points of two pixels in an image and calculate the real distance between them in order to measure length of object.

thank you

[ev-mp]

Hello @adarvit , the depth_point is basically a 3D coordinate in metric units with the origin located at the sensor's base. See Librealsense reference for more details.

As for the code - it seem legit. I'm not sure I understand the question, but if you want to verify the data flow you may check the following:

depth_frame = frames.get_depth_frame()

If the input is a depth-aligned-to-color frame, then its intrinsic and the color's frame intrinsic must be identical.

You haven't specified what was the original depth value at [100,100] before deprojecting the pixel, nor the 'depth_scale'. And while the results you've got are legit for a particular case (the 3D point is located very close, yet in front of the camera) it is not possible to deduce more from this.

Try several distinct (and valid) depth coordinates - deproject them to see how they correspond to the scene. One quick assert is that de-projection must not modify the Z (depth) value, only multiply by depth_scale

[ghost]

I want to find 3d point of depth pixel which i have already aligned to rgb frames. all the frames are already saved saved in my disk. (in order to align them i used the code as it shown in python examples)
now that i have pairs of aligned depth and rgb i am trying to measure lengths of objects - i thought that finding 3d points will be the right way to do it because i will be able to use them in Pythagoras for calculations.

i used depth_frame = frames.get_depth_frame() only for getting the depth intrinsic for the depth sensor, since it always returns the same values of intrinsic, not matter which frame the camera see in live, those values are constants.
now that i have my camera parameters and the pairs of depth and rgb frames, i am trying to find the 3d points.

depth_pixel = [100, 100]
depth _scale = 0.000124986647279
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)
this code always returns the same result. The depth frame - frames.get_depth_frame() doesn't influence the depth points results, since i tried several frames which were streamed live, and the results stayed the same (the depth value in [100,100] was different in each frame).

how can i use the aligned frames in my disk to find those 3d points in order to do my calculations? if rs.rs2_deproject_pixel_to_point method only works only on live stream frames, i would like to know how to implement the method by myself and apply it on my frames.

thank you

[vladsterz]

Use this function. Intrinsics change when stream resolution changes.
*Edit: By use, i mean you can implement it.

[ghost]

@vladsterz
rs2_deproject_pixel_to_point(float point[3], const struct rs2_intrinsics * intrin, const float pixel[2], float depth)
depth argument is the depth scale or the depth value of the pixel?

[ev-mp]

This indeed seem to be the reason-

depth_pixel = [100, 100]
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel_coordinate,depth_scale depth_value_in_meter)

The missing part- you need to retrieve the raw depth data that corresponds to [100,100] coordinate, convert it to meter using the depth_scale, and then provide the resulted value as input to rs2_deproject_pixel_to_point

[ghost]

thank you, it is solved my problem, but i am not sure i am doing it right since my calculations are far from being accurate.

for example i have this object, the real distance between those two points (x,y) is approximately 5 cm
depth_intrin = aligned_depth_frame.profile.as_video_stream_profile().intrinsics
depth_pixel_x = [150, 420]
depth_value_x = 1551
depth_pixel_y = [172,268]
depth_value_y = 1571
depth_point_x = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel_x, depth_value_x*depth_scale)
depth_point_y= rs.rs2_deproject_pixel_to_point(depth_intrin,depth_pixel_y,depth_value_y*depth_scale)
dist = math.sqrt(math.pow(depth_point_x[0]-depth_point_y[0],2)+math.pow(depth_point_x[1]-depth_point_y[1],2)+math.pow(depth_point_x[2]-depth_point_y[2],2))

the result is that the distance is almost 7 cm.

what could be the reason for the Inaccuracy?

[ev-mp]

Hi @adarvit , you can use rs-measure demo that provides the exact functionality and compare the results

[ghost]

I used it and it seems that rs-measure provides more accurate result. I cant use it for my frames since rs-measure takes live stream frames.

edit: i changed the intrinsics to be of the color sensor and not of the depth sensor and not the distance seems to be a little bit more accurate, is it possible?

[ghost]

thanks you all for the help, problem solved :)

[davidcann]

@adarvit, what was your ultimate solution? Was there something better than using the color sensor's intrinsics?

[soarwing52]

@adarvit Hi, I'm working on a similar project which is measure object with D435, by first taking pictures on-site, and then measure back in the lab. #3549
I'm still facing difficulties first capture the data and how to get the measurements in bag file.
I would like to ask if you can also share your more of your project?

[soarwing52]

I think the issue is you sent the x,y from Color Frame but use depth intrisinc to measure. I used color intrisinct

    def calculate_distance(self, x, y):
        color_intrin = self.color_intrin
        x_ratio, y_ratio = self.x_ratio, self.y_ratio
        ix, iy = int(self.ix/x_ratio), int(self.iy/y_ratio)
        x, y = int(x/x_ratio), int(y/y_ratio)
        udist = self.depth_frame.get_distance(ix, iy)
        vdist = self.depth_frame.get_distance(x, y)
        if udist ==0.00 or vdist ==0.00:
            dist = 'NaN'
        else:
            point1 = rs.rs2_deproject_pixel_to_point(color_intrin, [ix, iy], udist)
            point2 = rs.rs2_deproject_pixel_to_point(color_intrin, [x, y], vdist)
            dist = math.sqrt(
                math.pow(point1[0] - point2[0], 2) + math.pow(point1[1] - point2[1], 2) + math.pow(
                    point1[2] - point2[2], 2))
        return dist

This is how I do it and within in 5 meters it’s quite accurate within 5 cm

[10wei1000]

@soarwing52 Hi, may I ask what are the x_ratio and y_ratio in your code snippet?

@"anyone who could help": I am now dealing with a very similar problem where I try to find 3d points from 2d rgb pixels with depth aligned to color stream:

coordinates_up_left_3d = rs.rs2_deproject_pixel_to_point(intr_color, [ymin, xmin], depth_image[ymin, xmin]*depth_scale)

coordinates_up_right_3d = rs.rs2_deproject_pixel_to_point(intr_color, [ymin, xmax], depth_image[ymin, xmax]*depth_scale)

coordinates_down_left_3d = rs.rs2_deproject_pixel_to_point(intr_color, [ymax, xmin], depth_image[ymax, xmin]*depth_scale)

coordinates_down_right_3d = rs.rs2_deproject_pixel_to_point(intr_color, [ymax, xmax], depth_image[ymax, xmax]*depth_scale)

Even with color intrisinct in rs.rs2_deproject_pixel_to_point(): my result is far away from correct. Any ideas would be appreciated. Thanks!

---

Problem solved! The pixel coordinates should first have x value and then y value....not as the convention of [height, width]...

[soarwing52]

so, because in my application I adjusted the window size, so the x,y coordination must be calculated by the ratio to get the correct distance data, and avoid no data.
@10wei1000 you are looking for the four points of a square?
maybe its about the depth scale you put.
https://github.com/soarwing52/RealsensePython/blob/master/separate%20functions/measure_new.py
I used depth_frame.get_distance(ix,iy) as the third parameter, and its accurate
maybe show more of your codes

[10wei1000]

@soarwing52 Hi, thank you for your response. The problem is already solved. I placed the pixel coordinates with wrong order. The correct one should be x-coordinate comes first and then y-coordinate.