Minimal python code for computing extrinsics in multi-camera setup

[chinmay5]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D435i
	Operating System & Version
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC
SDK Version	2.0
Language	python
Segment	other

Issue Description

I have multiple realsense cameras and I would like to know their extrinsics (rotation + translation). However, I could not find some python code to do the same. As of now, I have a script to get the color images from the different cameras and also to create the point clouds from them. If I can get the extrinsics, I can use it to rotate the point clouds and do something like ICP to get good point cloud stitching. Any help would be highly appreciated.

[MartyG-RealSense]

Hi @chinmay5 There is an example of Python code at #1231 (comment) for getting extrinsics from a camera with the SDK's get_extrinsics_to instruction.

[chinmay5]

Hi @MartyG-RealSense thank you for your quick response. I checked the code and it seems to be for two frames from the same device. It is trying to align depth and color maps from the same caera. However, I am trying to align two different cameras. I could not make this snippet work for two cameras.

[MartyG-RealSense]

The RealSense SDK Python example program box_dimensioner_multicam can automatically calibrate together the positions of multiple cameras relative to each other when the program is launched, using a checkerboard image placed on the ground. It is a multi-script project though and so is not a minimal code example.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python/examples/box_dimensioner_multicam

If you are able to use a commercial software tool instead of creating your own application then the RealSense-supporting RecFusion Pro software, which includes a built-in multiple camera calibration tool, may be of interest. It has versions for Windows 10 and 64-bit Ubuntu (18.04 and 20.04).

https://www.recfusion.net/products/

[chinmay5]

Hi @MartyG-RealSense thank you for the code block. I saw that the code returns numpy arrays. When I tried converting the numpy array to point cloud, I do not have the colour information. I am not sure how to include it in the existing code block.

Essentially, I want some modifications to this piece:-

# Calculate the pointcloud using the depth frames from all the devices
point_cloud = calculate_cumulative_pointcloud(frames_devices, calibration_info_devices)

[MartyG-RealSense]

As box_dimensioner_multicam.py is already enabling the color stream via rs.stream.color, if your goal is a 'textured pointcloud' that maps RGB onto depth then the SDK's Python depth-to-color example program align_depth2color.py may provide useful insights for code to integrate into your application.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py

[chinmay5]

Hi @MartyG-RealSense thank you for the code block. I was able to get things to work. However, I am trying to reconstruct a small piece of rock something like

However, in the multi-stream setup, what I could reconstruct is mostly a plain surface with very little width.

Do you have suggestions on what can be used here? As of now I have put 2 D435i cameras but I can put one or two more.

Please also confirm one thing if possible. Earlier I was planning to use an ICP for the task. However, as far as I could understand, this multi camera alignment should be able to do a similar task, maybe not very robust but still useful.

[MartyG-RealSense]

The pointcloud of the rock appears not to have color texture applied to it when the above image is viewed zoomed-in.

The rock has a detailed surface texture and so this should make it easier for the camera to analyze it for depth information than if it had a smooth, low-detail surface.

How close are the D435i cameras positioned to the rock, please? I would recommend placing them at least 15 cm from the rock to help to ensure that there is not missing detail on the depth image due to being too close to the rock.

Is there a greater amount of depth detail on the image if you enable the IR Emitter component in your Python script to project a dot pattern onto the rock's surface? This can be done by setting the SDK instruction rs.option.emitter_enabled to a value of '1'.

import pyrealsense2 as rs
pipeline = rs.pipeline()
config = rs.config()
pipeline_profile = pipeline.start(config)
device = pipeline_profile.get_device()
depth_sensor = device.query_sensors()[0]
if depth_sensor.supports(rs.option.emitter_enabled):
depth_sensor.set_option(rs.option.emitter_enabled, 1)

---

You can access multiple RealSense cameras in Python, like in the project at the link below that was created by a RealSense user.

https://github.com/ivomarvan/samples_and_experiments/tree/master/Multiple_realsense_cameras

The more difficult part is stitching together the individual views from the multiple cameras into a single combined image and orienting each view so that they have a common position and rotation in 3D space, which is an 'affine transform' like the SDK instruction rs2_transform_point_to_point. More information about this instruction in regard to Python - which I linked to an example of earlier at #1231 (comment) - can be found at #5583

[chinmay5]

Hi @MartyG-RealSense , I made a slight change in my method. I do an ICP on the obtained point cloud from the different views. The final result I get is:-

Please ignore the artifacts since the base on which the rock is kept is not rigid (I will buy something more rigid so that such artifacts do not appear). However, I noticed a few things:-

I have two D415i and one D405. I observed that while using the example you suggested, (https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python/examples/box_dimensioner_multicam), the depth values are on a different scale for the D405 (seems they are in cm and are all positive). Is that intentional or did I make some mistake in the setup?

[MartyG-RealSense]

You are correct. The D405 has a default Depth Unit Scale of 0.01, whilst all other 400 Series cameras have a default depth unit scale of 0.001.

[chinmay5]

Hi, @MartyG-RealSense thanks for the information. Also, any idea what can be the reason for the change in the sign of distance as well? (In the box_multi_dimensioner_multicam.py code)

[MartyG-RealSense]

The original box_dimensioner_multicam.py program is suited to the cameras being a certain "sweet spot" distance from the observed object, such as 0.5 to 1 meters. If a camera is closer or more distant than this range then there may be corruptions in the results, such as bounding boxes that do not fit to the contours of an observed box. This phenomenon is demonstrated at #10100

If a D405 was placed 0.5 meters or more from an observed object then it could result in inaccurate depth values, as the ideal depth sensing range of the D405 is 7 cm to 50 cm.

[chinmay5]

Thank you for the information. I will try to adjust the alignment. However, the only thing that somewhat concerns me is still whether the sign flip is an artifact or an intended effect. I can imagine inaccuracy in depth but the change in sign is somewhat surprising.

[MartyG-RealSense]

If you have the IR emitter enabled and are projecting a dot pattern from the D435i, the D405 will not benefit from that as it has an IR Cut filter equipped on its left and right sensors, meaning that infrared is excluded from the range of visible frequencies that the D405 is able to see and so it cannot see the dot pattern projected by the D435i and use it to aid depth analysis.

[chinmay5]

Hi, @MartyG-RealSense one more thing. I do not want to have the checkerboard appear in my final scan. What sort of surface should I use so that only the rock appears in the scan? I would place the checkerboard printout on one side of the surface and simply flip it while performing the final scan.

Also, when I tried using the L515 Lidar in the same code section, I get the error message:-
Not enough points have a valid depth for calculating the transformatio
Place the chessboard on the plane where the object needs to be detected

I checked with realsense-viewer that the checkerboard is indeed within the view of the camera. However, somehow it still does not work while the D435i and D515 are working fine.

[MartyG-RealSense]

If the camera is always going to be a fixed distance from the checkerboard then you could try implementing a post-processing threshold filter in your Python code and set a maximum depth distance so that the distance from the camera represented by the floor under the rock is excluded from the depth image.

#8170 (comment) has an example of Python code to define a threshold filter with a maximum depth distance (rs.option.max_distance).

Alternatively, if you use a plain black color on the reverse of the checkerboard then that should prevent depth from being read from it when the checkerboard is flipped over.

In regard to L515, you could try experimenting with different Visual Preset camera configurations to see whether detection improves. #8619 demonstrates Python code for defining a preset, such as Short Range, using the SDK instruction rs.option.visual_preset

[chinmay5]

Hi, @MartyG-RealSense thank you so much for the information. Things are working as expected. Perhaps the final thing I wanted to know before you can go ahead and close the issue is, how to get the scan of the base of an object. For instance, for the rocks here, I wanted to get their base as well, (not only the surface from the top but also from the sides and the base). Any ideas would be highly appreciated.

[MartyG-RealSense]

If the program is using the calibrated-together positions of the cameras based on their position around the checkerboard then moving the cameras downwards to the base of the rock from the initial elevated position that they were calibrated in would not be an option.

Positioning additional cameras at the base of the rock would not work as the cameras would not be able to see the checkerboard from that board-level height.

Perhaps you could place additional cameras at an elevated position for the checkerboard calibration and then lower them down to board level when the calibration is complete and you are asked to place an object on the checkerboard (so the cameras would still have the same distance and orientation angles relative to each other but just a different vertical height).

[chinmay5]

@MartyG-RealSense thank you for the answers. Closing the issue.

[MartyG-RealSense]

You are very welcome, @chinmay5 - thanks very much for the update!