How to write aligned 16-bit-depth and color images for L515 camera in C++

[hduonggithub]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ L515 }
Firmware Version	(01.04.01.02)
Operating System & Version	{Win (8.1/10) }
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC
SDK Version	{ legacy / 2.. }
Language	{C }
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

Hi Support Team,

I am trying to write to images for both color and 16-bit depth from L515 camera. The color and depth image must be aligned and the same dimension. The depth should be in 16-bit format.

I have played with rs-capture and rs-save-to-disk, but my depth images did not look right. I did check the format which is shown as below:

Please advise, Thanks!

Here is my code which is modified from rs-capture and rs-save-to-disk

// License: Apache 2.0. See LICENSE file in root directory.
// Copyright(c) 2017 Intel Corporation. All Rights Reserved.

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include "example.hpp"          // Include short list of convenience functions for rendering
#include <fstream>              // File IO
#include <iostream>             // Terminal IO
#include <sstream>              // Stringstreams

// 3rd party header for writing png files
#define STB_IMAGE_WRITE_IMPLEMENTATION
#include "stb_image_write.h"


// Capture Example demonstrates how to
// capture depth and color video streams and render them to the screen
int main(int argc, char* argv[]) try
{
    rs2::log_to_console(RS2_LOG_SEVERITY_ERROR);
    // Create a simple OpenGL window for rendering:
    window app(1280, 720, "RealSense Capture Example");

    // Declare depth colorizer for pretty visualization of depth data
    rs2::colorizer color_map;
    // Declare rates printer for showing streaming rates of the enabled streams.
    rs2::rates_printer printer;

    // Declare RealSense pipeline, encapsulating the actual device and sensors
    rs2::pipeline pipe;

    // Start streaming with default recommended configuration
    // The default video configuration contains Depth and Color streams
    // If a device is capable to stream IMU data, both Gyro and Accelerometer are enabled by default

    pipe.start();

    uint32_t file_no = 0;
    //rs2::align align_to(RS2_STREAM_DEPTH); ////////////CASE 1
    rs2::align align_to(RS2_STREAM_COLOR); /////////////CASE 2

    while (app) // Application still alive?
    {

        rs2::frameset data = pipe.wait_for_frames();
        // The show method, when applied on frameset, break it to frames and upload each frame into a gl textures
        // Each texture is displayed on different viewport according to it's stream unique id
        app.show(data);

        // HD saving to disk
        rs2::frameset aligned_set = align_to.process(data);
        rs2::frame frame_depth = aligned_set.get_depth_frame();
        rs2::frame frame_color = aligned_set.get_color_frame();
        auto vf_depth = frame_depth.as<rs2::video_frame>();
        auto vf_color = frame_color.as<rs2::video_frame>();

        std::cout << "After" << std::endl;;
        std::cout << "depth width: " << vf_depth.get_width() << std::endl;;
        std::cout << "depth height: " << vf_depth.get_height() << std::endl;;
        std::cout << "color width: " << vf_depth.get_width() << std::endl;;
        std::cout << "color height: " << vf_depth.get_height() << std::endl;;

        // Write images to disk
        //depth
        std::stringstream png_depth_file;
        png_depth_file << "dataset/depth/" << "depth" << file_no << ".png";
        stbi_write_png(png_depth_file.str().c_str(), vf_depth.get_width(), vf_depth.get_height(),
            vf_depth.get_bytes_per_pixel(), (uint16_t*)vf_depth.get_data(), vf_depth.get_stride_in_bytes());

        //save_frame_raw_data(png_depth_file.str().c_str(), frame_depth);
        std::cout << "Saved " << png_depth_file.str() << std::endl;

        //color
        std::stringstream png_color_file;
        png_color_file << "dataset/image/" << "image" << file_no << ".png";
        stbi_write_png(png_color_file.str().c_str(), vf_color.get_width(), vf_color.get_height(),
            vf_color.get_bytes_per_pixel(), vf_color.get_data(), vf_color.get_stride_in_bytes());
        std::cout << "Saved " << png_color_file.str() << std::endl;

        file_no++;
    }

    return EXIT_SUCCESS;
}
catch (const rs2::error& e)
{
    std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;
    return EXIT_FAILURE;
}
catch (const std::exception& e)
{
    std::cerr << e.what() << std::endl;
    return EXIT_FAILURE;
}

Here is result
CASE 1: //rs2::align align_to(RS2_STREAM_DEPTH)

CASE 2;s2::align align_to(RS2_STREAM_COLOR); /////////////CASE 2

[hduonggithub]

Is there anyone can help me to write single band 16bit depth image correctly? Thanks

[cdb0y511]

Hi, I use opencv like

    frames.depthFrame_ =  std::make_shared<cv::Mat> ( Mat(Size(width, height), CV_16UC1, (void*)frame_depth.get_data(),    Mat::AUTO_STEP));

and imwrite works fine for me

[ev-mp]

@hduonggithub, the issue is with the export call:

stbi_write_png(png_depth_file.str().c_str(), vf_depth.get_width(), vf_depth.get_height(),
vf_depth.get_bytes_per_pixel(), (uint16_t*)vf_depth.get_data(), vf_depth.get_stride_in_bytes());

It assumes the input buffer being Big-endian according to PNG spec, while Z16 depth format is little-endian. So practically you need to swap the bytes order for each depth pixel (word) in the input buffer before calling stbi_write_png

#2147, #815

[hduonggithub]

@ev-mp Can you give me a sample code to swap the bytes? Thanks

[ev-mp]

@hduonggithub , you can do

std::vector<uint16_t> values(w*h);
memcpy((uint8_t*)values.data(),vf_depth.get_data(),values.size() * sizeof(uint16_t));
std::for_each(values.begin(), values.end(), [](uint16_t& val){ val=((val<<8)|(val>>8)); });
stbi_write_png(... ,values.data,()   ... )

[hduonggithub]

@ev-mp
I have modified the code as following. The result still looks incorrectly. Any idea?

    const int w = vf_depth.get_width();
    const int h = vf_depth.get_height();
    std::vector<uint16_t> values(w * h);
    memcpy((uint8_t*)values.data(), vf_depth.get_data(), values.size() * sizeof(uint16_t));
    std::for_each(values.begin(), values.end(), [](uint16_t& val) { val = ((val << 8) | (val >> 8)); });
    stbi_write_png(png_depth_file.str().c_str(), w, h, vf_depth.get_bytes_per_pixel(), values.data(), vf_depth.get_stride_in_bytes());

[ev-mp]

@hduonggithub , the input data is 16bpp. Is the tool you're using for snapshot it capable of handling 16bit data channels?
From the picture it is seen as if the render just presents the lower 8 bits of the data, hence the repetitive "wave" structure.
Most display/graphic card vendors still render True-color 24bit, or 8bit per channel, So I think it is the limitation of the graphic card/display you're using to present the result.
You can still use 16bit PNG with Processing software, such as Matlab. It is just not suitable to be presented with standard displays. For that we have a "colorizer" class in SDK that transfers 16bit single-channel depth data to 24bit RGB color space and makes "human-readable" images.

[hduonggithub]

@ev-mp
I did use EVNI/IDL software which is well known software for image processing. Below images are displayed in the same tool.

Test dataset shows correctly info: single band 16 bit image, BSQ format, Uint data type
Mine is 2 band image, BIP format and Byte data type.

I am so confused now. The data values look rounded and very small compared to test dataset (see profiles red color). Any help is really welcome. Thanks!

This is test dataset from open3d

This is mine

[ev-mp]

@hduonggithub , the difference is in the data header -

Mine is 2 band image, BIP format and Byte data type.

The PNG image appears stored incorrectly as 2-channel 8-bit rather than single-channel 16 bit data.
I looked up 'stbi' header and it seem to generate 8-bit per channel files only and therefore is not suitable to store 16-bit depth as native PNG.

The functions create an image file defined by the parameters. The image
is a rectangle of pixels stored from left-to-right, top-to-bottom.
Each pixel contains 'comp' channels of data stored interleaved with 8-bits
per channel, in the following order: 1=Y, 2=YA, 3=RGB, 4=RGBA. (Y is
monochrome color.)

We'll look for alternatives for stbi to deliver this capability for Z16, but in the meanwhile I suggest to serialize the raw data directly and handle it as a binary format:

librealsense/common/model-views.cpp

Lines 346 to 360 in f7cdf6e

	bool save_frame_raw_data(const std::string& filename, rs2::frame frame)
	{
	bool ret = false;
	auto image = frame.as<video_frame>();
	if (image)
	{
	std::ofstream outfile(filename.data(), std::ofstream::binary);
	outfile.write(static_cast<const char*>(image.get_data()), image.get_height()*image.get_stride_in_bytes());
	outfile.close();
	ret = true;
	}
	return ret;
	}

[hduonggithub]

@ev-mp Thanks. I will look forward to it. I will try your suggestion, I just do not know if open3d library can regconize this format or not. You may know.

I also will try using opencv as suggested by @cdb0y511

[hduonggithub]

@cdb0y511
Do you have any problem to compile when using imwrite? I think there is something missing here in opencv from realsense SDK package, so I cannot compile when using imwrite.

If I comment out the imwrite, the code was compiled successfully.
If uncomment out the imwrite, I got this error:

The code:---------------
Mat image16(Size(w, h), CV_16UC1, (void*)depth.get_data(), Mat::AUTO_STEP);
std::vector compression_params;
compression_params.push_back(CV_IMWRITE_PNG_COMPRESSION);
compression_params.push_back(9);
imshow("16 bit", image16);
imwrite("test.png", image16, compression_params);

The error:-------------------
Error LNK2019 unresolved external symbol "bool __cdecl cv::imwrite(class cv::String const &,class cv::debug_build_guard::_InputArray const &,class std::vector<int,class std::allocator > const &)" (?imwrite@cv@@YA_NAEBVString@1@AEBV_InputArray@debug_build_guard@1@AEBV?$vector@HV?$allocator@H@std@@@std@@@z) referenced in function main im-show C:\Program Files (x86)\Intel RealSense SDK 2.0\samples\im-show\rs-imshow.obj 1

Any help to fix this, Thanks!

[cdb0y511]

@hduonggithub
Hi, I don't meet this problem,
try

 cv::imwrite("test.png", image16); 

It does not relate with the SDK. I think you modified the sample directly.
so you may check your include or cmakelist again.

[hduonggithub]

@cdb0y511
I believe the imshow and imwrite should be in the same opencv package, right?

When I make the new project, using my opencv (4.4.0) (not using opencv from C:\Program Files (x86)\Intel RealSense SDK 2.0\third-party\opencv-3.4\include\opencv2), then I am able to compile and write depth image correctly in single band 16 bit format.
Here is my depth image

[cdb0y511]

I use 3.4 from GitHub. The warp from SDK third party may not include the full function of open CV ,I guess.

[RealSenseSupport]

Hi,

Will you be needing further help with this? If we don’t hear from you in 7 days, this issue will be closed.

Thanks