Depth Cam Intrinsics/Extrinsics

[macrohuang1993]

---

Required Info
Camera Model	{ D415}
Firmware Version	()
Operating System & Version	{Win (8.1/10)
Kernel Version (Linux Only)	()
Platform	PC
SDK Version	{ 2 }
Language	{C/C#/labview/nodejs/opencv/pcl/python/unity }
Segment	{}

Issue Description

I would like to capture depth map using D415 and then align the depth map to my own RGB cam (not the RGB cam on D415). To do that, I need to know the relative translation T /rotation R between the "depth camera", which generates the depth map, and my own RGB cam. I will also need to know the intrinsic camera matrices for RGB camera, K_rgb and for "depth camera", K_depth.

What I am thinking about is first doing the instrinc camera calibration on left-ir imager/my own RGB cam to get the intrinsic camera matrices, K_ir and K_rgb. Then do stereo calibration between left-ir imager and my own RGB cam to get the relative translation T' /rotation R' between the left-ir imager and my own RGB cam.

My question is: is it safe to assume:
1. R', T' is same as R, T? I am not sure because maybe D415 did stereo matching on rectified stereo pairs? In that case, I think they can be different.
2 .K_ir is same as K_depth?

Any input would be appreciated!

[MartyG-RealSense]

Hi @macrohuang1993 If you need to align the depth of a D415 to the RGB of a non-RealSense external camera, my research identified a couple of possible approaches from past cases of aligning RealSense with non-RealSense.

1. Create a virtual device with software_device. This will involve obtaining the intrinsic matrix for the external sensor, and the extrinsic matrix between D415 and the external sensor.

#4202

2. Use the MATLAB Stereo Camera Calibrator Toolbox to get the intrinsics of the external camera, and also the extrinsics between D415 and the external camera.

#5743

In regard to your questions:

1. I'm not sure about this question to be honest, given that it involves the positional relationship between a RealSense camera and a non-RealSense one (possibly together on a mount?). The subject has been addressed for RealSense cameras in the past. A RealSense team member created a Python tool for obtaining the extrinsics between a RealSense D435 and a RealSense T265 Tracking Camera on the same mount.

https://github.com/schmidtp1/librealsense/blob/calibrate-extrinsics/wrappers/python/examples/calibrate_extrinsics.py

2. My understanding from your description is that K_rgb refers to the intrinsics of your non-RealSense RGB camera, and K_depth and K-ir refers to the intrinsics of the D415 camera's depth and infrared respectively.

The left infrared sensor is perfectly aligned to the depth map. So K_ir should be the same as K_depth. Intel's Tuning depth cameras for best performance guide provides the following information:

---

The "LEFT IR" camera has the benefit of always 1. Being pixel-perfect aligned, calibrated, and overlapped with the depth map, 2. Perfectly time-synchronized, 3. Requires no additional computational overhead to align color-to-depth, and 4. Gives no additional occlusion artifacts, because it is exactly co-aligned.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-use-the-left-color-camera

[macrohuang1993]

This is very helpful, thank you very much!