align IR sensor to RGB

[zhaozhongch]

I am able to align the RGB and the depth based on the SDK example. I use D435.
However, I found rs2::align class can only align between rgb and depth. Now I want to align IR sensor to RGB image. I want the same result as the above example, is there anyway to do it?
I don't want rgb align to ir sensor. I know depth is aligned with one of the ir sensor, once I align rgb with depth, rgb is aligned with ir sensor but I'll end up with the following

The field of view of RGB is smaller than ir/depth, the bottom left RGB data will have a large black area. The following image has Depth align to rgb. I want ir sensor's image can do the same thing as the depth.

Code like the following won't work

rs2::align align_to_color(RS2_STREAM_COLOR);
....
rs2::frameset data = pipe.wait_for_frames();
auto frameset = align_to_color.process(data);
auto depth_align_to_color = frameset2.get_depth_frame();//this work, depth is aligned to colot
auto ir0_align_to_color = frameset2.get_infrared_frame(1);//this doesn't. ir0_align_to_color is the same as the original ir0

Thanks for your help!!!

[MartyG-RealSense]

I hope that the link below will be helpful to you.

#1556 (comment)

[MartyG-RealSense]

This case will be closed after 7 days from now if there are no further questions. Thanks!

[dorodnic]

@MartyG-RealSense - expiration does not apply to enhancement tickets.
It is currently a limitation of the API, and I cannot guaranty when its going to be addressed. However, the ticket should remain open until it is either addressed or rejected by the product team.

[MartyG-RealSense]

@dorodnic Leaving open is my policy too with all labeled tickets. In this one case I did not see your already applied label. Thanks for highlighting the oversight!

[cconstantine]

Thanks for leaving this open. I would also like a way to align the left IR sensor to RGB.

[bekmake]

@dorodnic Is there any news about aligning IR to RGB by API?
If there is still no such function, could you please guide on how to do it ourselves?

[MartyG-RealSense]

@bekmake Other than the suggested method in #1556 (comment) the only other approach to the problem that I am aware of is the one in the link below:

#5093

[zhaozhongch]

I should post this earlier.
I did some tricks to align the IR sensor and color. May not have the perfect result as you want but this is the best I can do under the current API and hardware limitations.

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <opencv2/opencv.hpp>   // Include OpenCV API for visualization

int main(int argc, char * argv[])
{
    // Declare depth colorizer for pretty visualization of depth data
    rs2::colorizer color_map;

    // Declare RealSense pipeline, encapsulating the actual device and sensors
    rs2::pipeline pipe;
    
    //Create a configuration for configuring the pipeline with a non default profile
    rs2::config cfg;

    int w  = 640, h = 480;
    //int w2 = 1080, h2 = 1920;

    cfg.enable_stream(RS2_STREAM_DEPTH, w, h, RS2_FORMAT_Z16, 30);
    cfg.enable_stream(RS2_STREAM_INFRARED, 1, w, h, RS2_FORMAT_Y8, 30);//left infrared
    cfg.enable_stream(RS2_STREAM_INFRARED, 2, w, h, RS2_FORMAT_Y8, 30);//right infrared
    cfg.enable_stream(RS2_STREAM_COLOR, w, h, RS2_FORMAT_BGR8, 30);

    auto profile = pipe.start(cfg);

    //enable two aligns
    rs2::align align_to_depth(RS2_STREAM_DEPTH);
    rs2::align align_to_color(RS2_STREAM_COLOR);

    while (1)
    {
        
        rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
        auto frameset3 = align_to_depth.process(data);

        auto frameset2 = align_to_color.process(data);//first let depth align to color
        auto frameset = align_to_depth.process(frameset2); //then let ir0 align to depth agian. Thus we have ir0 align to rgb

        auto color_align_to_depth = frameset3.get_color_frame();
        auto ir0_align_to_depth = frameset.get_infrared_frame(1);
        auto depth_align_to_color = frameset2.get_depth_frame();
        auto depth_color = c.colorize(depth_align_to_color);

        rs2::video_frame infra0 = data.get_infrared_frame(1);//original infra0 
        rs2::video_frame infra1 = data.get_infrared_frame(2);//original infra1


        // Create OpenCV matrix of size (w,h) from the colorized depth data
        Mat color_image_align_to_depth(Size(w, h), CV_8UC3, (void*)color_align_to_depth.get_data(), Mat::AUTO_STEP);
        Mat depth_image_align_to_color(Size(w, h), CV_8UC3, (void*)depth_color.get_data(), Mat::AUTO_STEP);

        Mat ir0_image_align_to_depth(Size(w, h), CV_8UC1, (void*)ir0_align_to_depth.get_data(), Mat::AUTO_STEP);
        Mat ir0(Size(w, h), CV_8UC1, (void*)infra0.get_data(), Mat::AUTO_STEP);
        Mat ir1(Size(w, h), CV_8UC1, (void*)infra1.get_data(), Mat::AUTO_STEP);
        Mat ir, dc, cd, all;

        // Update the window with new data
        cv::hconcat(ir0_image_align_to_depth, ir1, ir);//cv::hconcat(ir0, ir1, ir);
        cv::hconcat(depth_image_align_to_color, color_image_align_to_depth, cd);
        imshow(window_name0, ir);
        imshow(window_name1, cd); 

        waitKey(1);
    }

    return EXIT_SUCCESS;
}

From the code you can see that the trick is
1: align depth to rgb
2: align ir0 to depth
Then ir0 is aligned with rgb.
The result looks like the following

The upper left is the IR0 aligned to depth and color, the upper right is the original IR1 image just showing for comparison.

You can see basically the IR0, depth, color is aligned and there is not a large black area like the first image I post here (rgb align to depth). However, this method's shortcoming is also apparent. The upper left IR image now has no data at the place where there is no depth. The rgb image still has some wired black area but at least better than directly aligning it to depth.

[MartyG-RealSense]

@zhaozhongch Thanks so much for posting your solution for the RealSense community to benefit from!

[bekmake]

@MartyG-RealSense thanks for suggestions!
@zhaozhongch Thanks a lot for such a great solution!

[MartyG-RealSense]

Hi @zhaozhongch After you developed your excellent solution to aligning IR sensor to RGB, is it okay to close this case now please, or do you need further advice? Thanks!