Align to Depth produces lossy color image

[delucr]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ D435 }
Firmware Version	05.10.13.00
Operating System & Version	{Win (10)
Platform	PC
SDK Version	{ 2.0.<?> }
Language	{matlab }
Segment	{others }

Issue Description

In following the examples available for using the align function to align the rgb and depth images, all of the examples demonstate aligning depth to color as in:

align_to = realsense.stream.color;
alignedFs = realsense.align(align_to);
.....
fs = pipe.wait_for_frames();
aligned_frames = alignedFs.process(fs);

However, if you align color to depth as in:

align_to = realsense.stream.depth;
alignedFs = realsense.align(align_to);
.....
fs = pipe.wait_for_frames();
aligned_frames = alignedFs.process(fs);

The result is a color image that contains holes in it at what appears to be the location of zero depth information. To me this is unexpected. My understanding of what the align function should do is scale, size, and shift the color image as if it was in the depth camera's coordinate system. It appears though that the depth image is being used to filter the color image and therefore drop data from the color image where the depth image has zeros. Using a hole filling filter on the depth before aligning improves the color image but it is sub-standard of what I think align should do.

Please let me know if this is intended functionality or something that can be fixed.

Best regards,

Rich

[ev-mp]

@delucr hello,
The term "alignment" w.r.t SDK denotes a synthetic image generation with the help of depth map. In the process a new image is created using triangulation method comprising of 2D=>3D=>2D transformations.
The transformation is akin to a ray tracing - each pixel is first "fired" from 2D to 3D space, and then projected into the 2D plane of another sensor.
So by definition the aligned data is generated from depth in conjunction with color/ir data, hence the areas with no depth data coverage result in "holes" in the generated image.

The above serves a specific purpose - allowing 1:1 mapping of depth to color and vice versa. So when a user selects RGB pixel of interest then the SDK will be able to provide the corresponding Depth deterministically with no false positives.
This is a critical feature in scenarios where the depth is essentially used as a validation filter for other sensors - collision avoidance, segmentation, object recognition.

While using sheer 2D image manipulations would similarly allow to recalculate an image from a different point of view, it would also introduce fundamental flaws - preserve the pixels for which there is no corresponding depth data, and also introduce artifacts due to occlusion and different perspective, which would undermine the declared purpose.

Eventually there are more than one way to "align" images, depending of the use-case requirements. And the above explanation highlights the specific advantage (and also the utility) of the method implemented by the SDK.

[ev-mp]

@delucr, do you have any follow ups?
We'll close it if no further questions.

[delucr]

Thank you for the reply. I understand why it was implemented the way it was now. Thanks. From: ev-mp <[email protected]> Sent: Tuesday, October 22, 2019 2:01 AM To: IntelRealSense/librealsense <[email protected]> Cc: DeLuca, Richard <[email protected]>; Mention <[email protected]> Subject: Re: [IntelRealSense/librealsense] Align to Depth produces lossy color image (#5030) EXTERNAL EMAIL @delucr<https://urldefense.com/v3/__https:/github.com/delucr__;!QrfbRyVwSKw!eb5hqudDtHZQPqQC-g4yqrPF2vKLhBu0TOKjL2Wt_n5tQ-g7gNZJcjMIyaW2z1HtUp8x$>, do you have any follow ups? We'll close it if no further questions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/IntelRealSense/librealsense/issues/5030?email_source=notifications&email_token=AG2GT4ROAQL6JZP62TNMPMDQP2JJRA5CNFSM4I7POKXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB4T44Q*issuecomment-544816754__;Iw!QrfbRyVwSKw!eb5hqudDtHZQPqQC-g4yqrPF2vKLhBu0TOKjL2Wt_n5tQ-g7gNZJcjMIyaW2z_PwfKrv$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AG2GT4QCSSFIC2CCKYRBTEDQP2JJRANCNFSM4I7POKXA__;!QrfbRyVwSKw!eb5hqudDtHZQPqQC-g4yqrPF2vKLhBu0TOKjL2Wt_n5tQ-g7gNZJcjMIyaW2z_ggkF1U$>. CAUTION: External email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please forward all suspicious emails to [email protected]<mailto:[email protected]>. Follow this link to read our Privacy Statement<https://www.stryker.com/content/stryker/gb/en/legal/global-policy-statement.html/>

[ev-mp]

It was a non-trivial question, you're welcome.