Seeking Guidance on Hand-Eye Calibration for Camera-Mounted Robot Arm (D405)

[qidihan]

Required Info	Seeking Guidance on Hand-Eye Calibration for Camera-Mounted Robot Arm
Camera Model	{ D405 }
Firmware Version	(The newest)
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)
Platform	PC
SDK Version	{ }
Language	{python }
Segment	{Robot }

Issue Description

Hello,

I have recently mounted a D405 camera onto a robotic arm for a project that involves precise positioning and object manipulation. However, I am currently facing the challenge of performing hand-eye calibration to accurately determine the position and orientation of the camera relative to the robot arm's end-effector or the base.

I am looking for resources or guidance on how to proceed with the hand-eye calibration process. I looked some example on D435, but still don't know how to do it.
Does anyone have experience with this or can point me to some detailed documentation, tutorials that can help me understand and implement hand-eye calibration?

Thanks,
Qidi

[MartyG-RealSense]

Hi @qidihan There are hand-eye calibration tools for robot arms that work with RealSense 400 Series cameras. Here are a couple of suggestions.

autoCalibration and documentation
#3569 (comment)

'Robotic Hand-eye Calibration Workspace' eye-in-hand for ROS
https://github.com/lixiny/Handeye-Calibration-ROS

The above system makes use of visp, which has been separately recommended by some for hand-eye calibration with RealSense.

visp and tutorial
http://wiki.ros.org/visp_hand2eye_calibration

https://visp-doc.inria.fr/doxygen/visp-daily/tutorial-calibration-extrinsic.html

[qidihan]

The first example appears to be a Hand-to-Eye calibration example. The second one utilizes ROS. I only have Ubuntu 22.04, which uses ROS 2. Do you have any examples that demonstrate the use of ROS 2 with the RealSense D405 camera? I found one with link below. But it use opencv, so the camera will open with cv2. Does D405 has ability to open with cv2.VideoCapture?https://github.com/chrwave/Hand_Eye_Calibration

[MartyG-RealSense]

There is a ROS2 hand-eye calibration tool at https://github.com/tku-iarc/hand-eye-calibration_ROS2 that is RealSense compatible. It has been tested with the D435i model rather than D405 but it will likely be okay for D405 too as it uses the RealSense ROS2 wrapper which is D405 compatible.

A Python script at #12185 demonstrates successful use of cv2.videocapture with D405.