How to reduce the deviation between ir images with color images when use align_to_color()?

[NingCY]

Required Info
Camera Model	D415
Firmware Version	05.12.12.100
Operating System & Version	Win10
Kernel Version (Linux Only)	None
Platform	PC
SDK Version	{ legacy / 2.. }
Language	{C/C#/labview/nodejs/opencv/pcl/python/unity }
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

Hi, I‘m using D415 and I want to align 4 channels images(RGB, Depth, Infrared, Infrared2). I use align_to_color() to try to get 4 channels aligned pixel by pixel, I found depth image has been aligned with color image, but 2 ir images are not aligned and have deviation with other 2 channel images. See below:

How to get ir images without deviation? Thanks in advance.

[MartyG-RealSense]

Hi @NingCY Alignment is performed between two stream types, such as depth and color. Infrared (left-side infrared) does not need to be aligned as it is already perfectly aligned with the depth map, so in a depth-color-infrared alignment the depth is aligned to color. There is a Python alignment script for depth-color-Infrared (left side) at #5093

There would not be a strong benefit to trying to add Infrared 2 to the aligned image as it will be almost the same as the Infrared left-side image except for a small horizontal offset.

[NingCY]

@MartyG-RealSense yeah, I understand Infrared(left-side infrared) is already perfecty aligned with the depth map, and I use align_to_color() to try to finish a depth-color-infrared alignment, but after I did that, I got Infrared(left-side infrared) image have deviation with depth image, just like green arrow in lower left part of my posted image. You can see depth and color image is good, but 2 ir images both have deviations even include left-side infrared one.

I check #5093, he has tried to align all inputs to depth but will exists artifacts in rgb and ir images, that's not what i expected.

As far as I understand, Infrared (left-side infrared) is already perfectly aligned with the depth map, and depth image is aligned with rgb image, so Infrared (left-side infrared) should also be aligned with rgb image, why is there a deviation between Infrared (left-side infrared) and depth?

[MartyG-RealSense]

When depth to color alignment is performed, the depth stream will be resized to the size of the color stream's FOV. This is a software resizing, as the size of the physical sensor component's FOV cannot be changed. On the D415 camera model, depth FOV is 65° × 40° and color FOV is 69° × 42°.

A mis-alignment between depth-color-infrared during alignment is usually not an issue that arises because it is extremely rare that a three-stream alignment of depth, color and infrared is attempted.

In an alignment with the RealSense SDK's align processing block (align_to), two stream types only can be aligned and one of those types must be depth. So you could have depth-color alignment (the most common type of alignment) or depth-infrared alignment (which is very rarely used). When depth is aligned to color and the depth stream FOV is resized, the infrared stream's FOV should remain at its original size because the align processing block can only change the depth stream's FOV size, not both the depth and left infrared streams' FOV.

If there is a misalignment between infrared and color then a solution may be to perform color to depth alignment so that the color FOV is resized to match the depth / infrared FOV size.

[MartyG-RealSense]

Hi @NingCY Do you require further assistance with this case, please? Thanks!

[NingCY]

@MartyG-RealSense No, thanks for your help!

[MartyG-RealSense]

You are very welcome, @NingCY - thanks very much for the update!