Pybind11 binding for rs2::frame fails with error as incompatible functional argument

[saroj31]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	{ D400 }
Firmware Version	(Open RealSense Viewer --> 05.13.00.50)
Operating System & Version	Linux (Ubuntu 20)
Kernel Version (Linux Only)	5.15.0-47-generic
Platform	PC/Raspberry Pi/ NVIDIA Jetson / etc..
SDK Version	{ legacy / 2.. }
Language	{C/python }
Segment	{others }

Issue Description

I am trying to add a new python module to convert the depth frame to RGB frame using my C++ code. I am trying to pybind11 here. I am facing problems here. I describe it in details below.

I am trying to make a python binding for my C++ code where I convert the Depth Frame to RGB Frame. I am doing this in C++ as I need some specific functions in C++ which I don't get in pyrealsense api.

The C++ code function with argument is below defined in mymath.hpp:

rs2::frame DtoRGBframe(rs2::frame in_depth) {

std::cout<<"Reached DtoRGBframe function"<<std::endl;
// declare filteres
rs2::threshold_filter thr_filter; // Threshold - removes values outside recommended range
rs2::colorizer color_filter; // Colorize - convert from depth to RGB color
rs2::disparity_transform depth_to_disparity(true); // Converting depth to disparity
rs2::disparity_transform disparity_to_depth(false);

float min_depth = 0.6f;
float max_depth = 3.5f;

//filter settings
thr_filter.set_option(RS2_OPTION_MIN_DISTANCE, min_depth);
thr_filter.set_option(RS2_OPTION_MAX_DISTANCE, max_depth);
color_filter.set_option(RS2_OPTION_HISTOGRAM_EQUALIZATION_ENABLED, 0);
color_filter.set_option(RS2_OPTION_COLOR_SCHEME, 9.0f);
color_filter.set_option(RS2_OPTION_MAX_DISTANCE, max_depth);
color_filter.set_option(RS2_OPTION_MIN_DISTANCE, min_depth);

// apply post processing filters
in_depth = thr_filter.process((in_depth));
in_depth = depth_to_disparity.process((in_depth));
in_depth = disparity_to_depth.process((in_depth));
in_depth = color_filter.process((in_depth));

return in_depth;
}
The binder.cpp for the above code is here:

#include <pybind11/pybind11.h>
#include <pybind11/cast.h>
#include "mymath.hpp"

namespace py = pybind11;

PYBIND11_MODULE(cmake_example, m) {

m.def("DtoRGBframe", &DtoRGBframe, R"pbdoc(
depth frame to RGB frame
)pbdoc");
I am able to successfully get the wheeler file for this package module and I am able to install the package with the module name as cmake_example.

below is the python test code snippet showing how I use my function:

frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
depth = depth_frame.as_depth_frame()
print(type(depth))
color_frame = frames.get_color_frame()
if not depth_frame or not color_frame:
continue

    # Convert images to numpy arrays
    rgb_depth_frame = cme.DtoRGBframe(depth.as_frame())
    depth_image = np.asanyarray(depth.get_data())
    color_image = np.asanyarray(color_frame.get_data())

When I run it I get the below error and the program fails

Traceback (most recent call last):
File "/home/snuc/home/snuc/Code_Repo/RemoteCode/cmake_example/build/../tests/test_basic.py", line 56, in
rgb_depth_frame = cme.DtoRGBframe(depth.as_frame())
TypeError: DtoRGBframe(): incompatible function arguments. The following argument types are supported:
1. (arg0: rs2::frame) -> rs2::frame

Invoked with: <pyrealsense2.frame Z16 #10 1667339936076.809570>

Can ANybody please help me here. What I feel I am not able to cast the python realsense object to the C++ object.

I tried to define the rs2 frame using pyclass_ .

below code for that:

py::class_rs2::frame(m, "rs2::frame")
.def(py::init<>());
I am knew to Pybind11. Thought it will be am easy work but it is so complex.

[MartyG-RealSense]

Hi @saroj31 The individual SDK functions featured in the above script (such as post-processing filters and colorizer) should be directly accessible through pyrealsense2 instead of having to write linkages between C++ and pyrealsense2.

Colorizer options and histogram equalization
#7767 (comment)
#7089 (comment)

POST-PROCESSING FILTERS

Threshold
#8170 (comment)

Disparity to Depth / Depth to Disparity
#11015 (comment)

---

I note that in the line below, it is commented that the colorizer is converting from depth to RGB color.

rs2::colorizer color_filter; // Colorize - convert from depth to RGB color

Technically, although the colorizer is coloring the depth coordinates according to their depth values and the settings of the colorizer, the coloring is not related to the RGB stream, so it is not converting from depth to RGB. It is instead just color-shaded depth data.

Streams in the RealSense SDK cannot be converted directly from one format to another (such as converting a depth stream into a color stream in real-time). You can though save depth data into a numpy array and then do subsequent conversion work on it, such as writing it to a PNG image file or an .AVI video file using OpenCV cv2 instructions, like the .avi writing pyrealsense2 script at #8528

[MartyG-RealSense]

Hi @saroj31 Do you require further assistance with this case, please? Thanks!

[saroj31]

Hi @MartyG-RealSense ,

Thanks for answering my query. Actually I am not trying to convert the depth image to rgb image, sorry for the misguiding comments in my code. We are actually working on getting a compressed depth image and then recover it to depth later. I used the method given by realsense i.e. https://dev.intelrealsense.com/docs/depth-image-compression-by-colorization-for-intel-realsense-depth-cameras, for my conversion and restore work. I cud achieve a good result in C++. We also needed a python version of that to use in our python applications.

My work is done in C++. I did not get the same result when I did it in Python. At least I read it as so, check the issue here: #9713. I could not find the equivalents of each C++ function. So we thought to do the heavy lifting work in C++ and just pass the realsense frames across the python and C++ applications. For that I explored the pybind11 bindings of C++ and python integrations, I tried to implement it on my side.

For the python integration we tried to use pybind11 to make our python bindings. We also found that realsense library is very rich in C++ code than the pyrealsense2. So we want to know the correct way or procedure to extend the python bindings of realsense such that we can both use the high performance features of C++ with our python applications. I tried to find some documentation of realsense on how the python bindings can be extended but could not find any so I asked about the problem I am facing here in the forum for help.

[MartyG-RealSense]

As pyrealsense2 is a wrapper of the C++ library, each C++ function should have a pyrealsense2 equivalent and so there is usually no need for RealSense users to create their own bindings to communicate between the two as it is already done for them.

As you have observed, depth image compression by colorization technique is more straightforward in C++ than it is in Python. The case #9713 that you quoted is an example of this.

The depth compression by colorization paper was a proof-of-concept experiment that is not recommendable as a means of depth data compression. #8117 discusses alternative approaches to compression.

[saroj31]

Thanks a lot for your answer and time, Martyz. I understand that the python bindings are enough. I will work on making a separate python application later.

For now we need to use python bindings for performance related boost too by using C++ application we can use multiple cameras and want to do Multithreading to have good frame rate. The above code is an attempt to fulfill that requirement. We want to use pybind11 as it is used in realsense side too. Also we want to exchange realsense frames from python to C++ code. I see it is done on your side in realsense side.

Can you help us here so that we can exchange realsense objects from Python to C++ code or vice versa. What is the necessary code needed ? The above code in my query is an attempt to do that. It wil be helpful if you can explain what is the wrong that I am doing in that code. Also if you can send me a snippet or point to your code where you are doing it in pyrealsense side so that I can get inspired from that code.

[MartyG-RealSense]

It is also possible to use multiple cameras and multithreading directly in pyrealsense2.

Mutiple cameras: https://github.com/ivomarvan/samples_and_experiments/tree/master/Multiple_realsense_cameras

Multithreading: #10113

The official pyrealsense2 wrapper uses pybind11 as its backend in its pybackend.cpp file.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/pybackend.cpp

As the RealSense SDK is open-source, you could conceivably use the source code of the current pyrealsense2 wrapper as a foundation to create your own custom version of pyrealsense2 that could then be built from the modified source code. The pyrealsense2 wrapper's files, including pybackend.cpp, can be found at the link below.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python

---

I do not have any guidance to offer about how you might write your own pybind11 binding for librealsense if you prefer not to use pyrealsense2 or transfer data from C++ to Python and vice versa though, unfortunately.

[MartyG-RealSense]

Hi @saroj31 Do you require further assistance with this case, please? Thanks!

[saroj31]

No. Nothing else. I will come back here later if I face any difficulty.

[MartyG-RealSense]

Thanks very much for the update. As you do not have a problem for now, I will close the case. It can be re-opened at a future date if you have further questions or you can create a new issue. Thanks again!