Getting unrectified images for both right and left through ROS into Unity

[mericgeren]

Hi there,

I want to send both images coming from both stereo modules of D435 camera through ROS and then, display both images in Unity. Can you help me on that please?

Thanks in advance,

Kindest regards.

[MartyG-RealSense]

Hi @mericgeren In a past case several years ago, a RealSense ROS user shared the method that they used to stream ROS topics into Unity. I have pasted it below.

https://community.intel.com/t5/Items-with-no-label/Using-the-Realsense-Unity-wrapper-to-visualise-pointcloud-from/m-p/1290448

---

Yes there is a way. Use the Software Device in the RealSense SDK. I did it by creating a wrapper class that used two software sensors that had video streams with all the intrinsic values from the camera info topics. I then used an instance of the RealSense Syncer class in replacement of the pipeline profile that has waitForFrames() called on it to feed framesets into the point cloud rendering pipeline. It's possible!

[mericgeren]

Thanks for the help you have offered. Now, i can get both IR streams from left and right stereo units in y8 format at the same time into Unity. I am curious about if it's possible to combine color stream and each IR stream to get colorized versions of them while keeping them unrectified?

[MartyG-RealSense]

Aligning color to infrared is not supported by the RealSense SDK, though a RealSense team member suggests at IntelRealSense/librealsense#1556 (comment) a method by which it might be achieved.

The other approach, described at IntelRealSense/librealsense#5093, is to align depth, color and infrared.

[mericgeren]

At the other approach, described at can i keep my IR streams from both left and right unrectified? Or will they automatically become rectified?

[MartyG-RealSense]

If a stream is not rectified then aligning it to another stream will not cause it to become rectified. Alignment, when performed using the SDK's align_to instruction, causes the origin coordinates to change to the origin of the sensor that is being aligned to.

For example, if depth is aligned to color then the 0,0,0 origin point of depth changes from the center of the left infrared sensor to the center of the RGB sensor.

[mericgeren]

But, i guess i still have to send that as IR to get both left and right images. So is it still be in Y8 format and thus still monochrome? Or Does doing this alignment just enable me to send images coming from left and right stereo units as two color images from both left and right? When i try to get two color streams for each of left and right unrectified streams after aligning color, IR and depth in their own seperate pipelines and configurations, i see this error: RuntimeError: xioctl(VIDIOC_S_FMT) failed, errno=16 Last Error: Device or resource busy. So can't i get two color streams for each of left and right IR unit?

P.S. Please let me know if sharing the code i wrote with you will help.

[MartyG-RealSense]

The D435 model only supports Y8 monochrome infrared streams. The D405, D415 and D455 support color from the left infrared sensor in RGB8 format but not from the right infrared sensor, so you cannot have color from both infrared sensors even on those camera models.

I do not require the code, thank you.

[mericgeren]

Thanks for the all the kind help you have offered. But, what about this solution (even though, i couldn't get much clue from it): IntelRealSense/librealsense#1556 (comment)

[MartyG-RealSense]

It sounds as though it is suggesting to align depth, infrared and color - like in IntelRealSense/librealsense#5093 - and then calculate the UV map for depth to color.

Intel have a section about UV map calculation in their Projection, Texture Mapping and Occlusion guide at the link below.

https://dev.intelrealsense.com/docs/projection-texture-mapping-and-occlusion-with-intel-realsense-depth-cameras#33-the-uv-map

[mericgeren]

Thanks for the all the help you offer. I know it isn't right place but, can you give me advice on how to compress Y8 IR image for transporting through ros? Actually, i wrote a code but, it both gives me fps rates like 1.630 or 0.735 and generally, most of time i get "no new messages" repeated. So, can't i compress IR images just like i did to color images?

[MartyG-RealSense]

The Intel documentation page for ROS at the link below indicates that the compressed-image-transport or compressed-depth-image-transport package should be installed in order for compressed infra topics to be published. The topic names published are slightly different depending on which package is used.

https://dev.intelrealsense.com/docs/ros-wrapper#compression-packages

For each of these packages, install the version that is specific to the ROS1 version that you are using (Kinetic, Melodic or Noetic).

compressed-image-transport
ros-kinetic-compressed-image-transport
ros-melodic-compressed-image-transport
ros-noetic-compressed-image-transport

compressed-depth-image-transport
ros-kinetic-theora-image-transport
ros-melodic-theora-image-transport
ros-noetic-theora-image-transport

If you are using ROS2 then change the ROS version name in the instruction to whichever ROS2 version is being used. For example:

ros-foxy-compressed-image-transport
ros-humble-compressed-image-transport

[MartyG-RealSense]

Hi @mericgeren Do you require further assistance with this case, please? Thanks!

[mericgeren]

Hi @MartyG-RealSense, sorry for not being able respond you until today. If you don't mind, can i return to you in regards to both UV mapping and compressed IR transport in the next week?

Kindest regards,

Meriç Geren

[MartyG-RealSense]

Yes, it's no problem at all to return to those subjects in the coming week.

[mericgeren]

Hello there,

In regards to compressed IR transport, i managed to get it into Unity and interestingly, even though i have the texture format R8 in Unity, i can see IR images in grayscale (before switching to compressed IR messages i would see images in red color)

But, in UV mapping, i got an issue. When i wrote my code and tried to run it. I got the error:

frames = pipeline.wait_for_frames()
RuntimeError: Frame didn't arrive within 5000

Thank you for all the help you have offered,

Kindest regards.

P.S. Please let me know if sharing the code i wrote will help you.

[MartyG-RealSense]

My Unity RealSense wrapper programming knowledge is limited, unfortunately. There is a past RealSense case about creating a UV map in Unity at IntelRealSense/librealsense#9588 though.

[mericgeren]

Thanks for the solution you have shared with me. It seems like calculating UV map for depth to color, and then, using this UV map to remap IR frames in Python has not been tried before.

Does the method mentioned in the post IntelRealSense/librealsense#1556 (comment) can be only used for a geometric alignment and can't be used for getting the colorized versions of left and right IR images just like the solution here IntelRealSense/librealsense#5963 (comment)?

[MartyG-RealSense]

Sincere apologies for the delay in responding further.

There are no further details about how to implement the method suggested in IntelRealSense/librealsense#1556 (comment) and it seems as though it was just a thought about how to approach the problem that was never actually tested in a real application.

[MartyG-RealSense]

Hi @mericgeren Do you require further assistance with this case, please? Thanks!

[mericgeren]

Hi there, sorry for my tardiness. For now i don't require further assistance with this case. So if you don't mind i am closing this as not planed. Sincere thanks for all the help you have offered.

With kindest regards.

[MartyG-RealSense]

You are very welcome. It's totally fine to close this issue. Thanks very much for the update!