Displaying Point Cloud from a Realsense D435 with Realsense Unity wrapper through ROS

[mericgeren]

---

Required Info
Camera Model	{ D435 }
Firmware Version	(2.50.0)
Operating System & Version	{Win (10) and Linux (Ubuntu 18.04)}
Kernel Version (Linux Only)	(5.3.0.59-generic)
Platform	2 PCs
SDK Version	{ 2.3.2 }
Language	{C#/python/unity }
Segment	{Robot/VR/others }

Issue Description

Hello there,

I am trying to get a pointcloud from Realsense D435 camera and a pc with linux via roslaunch realsense2_camera rs_camera.launch filters:=pointcloud through ROS, into another pc in the same LAN and there into unity to be displayed. I wrote a custom subscriber for getting, converting and rendering the point cloud from ROS into Unity. Although, i was able to display the point cloud in color, since point cloud was pretty big, it suffered from low FPS. So i wanted to downsampling the point cloud. But, this time it suffered from scarcity of points and thus, low visibility. Now i want to use Realsense Unity wrapper in the Unity side for displaying point cloud i got from ROS "camera/depth/color/points" topic. Could you explain how can i use the ROS point cloud message in a way that the rendering can be done in Realsense Unity wrapper entrirely?
Thanks in advance.

With kindest regards,

Meriç Geren

[MartyG-RealSense]

Hi @mericgeren After your previous case at #11862 about accessing ROS topics in Unity there is not any further information that I can suggest beyond what is already in that case, unfortunately.

Have you tried adding ordered_pc:=true to your ROS1 roslaunch instruction to publish an 'organized' point cloud instead of the default 'unorganized' one to see how it affects point cloud size, please?

[mericgeren]

Hello @MartyG-RealSense , unfortunately, adding ordered_pc:=true messes up colors. Only about 25% of the point cloud being rendered with correct colors. Other parts are too dark or too bright with bizzare patterns of colors. For the RealSense Unity wrapper, i have delved into how does RealSense Unity wrapper display pointcloud the issue is it seems like it uses a custom point class. But, when i checked out that class, couldn't get much clue about it. Also, just inspected these non-realsense pointcloud assets there. But, it seems like they can't take point cloud live. For them, you need to save the pointclouds into a file in certain format. Also, checked on PCX. It only gets these point coluds from files too if i am not mistaken.

[MartyG-RealSense]

At #1289 (comment) a RealSense team member offers advice about the pointcloud in Unity, whose code is in the Unity wrapper script file PointCloudGenerator.cs. The discussion adds that the pointcloud object is defined in the C# wrapper file wrappers/csharp/Intel.RealSense/Processing.cs

[mericgeren]

There is a thing that i couldn't understand: What is that "_UVMap" in PointCloudGenerator.cs? It seems like it's used in place of a shader there.

[MartyG-RealSense]

It looks as though the PointCloudGenerator.cs file comes from a version of the RealSense Unity wrapper that is several years old (the 400 Series cameras have been around since the start of 2018) and in the current Unity wrapper pointclouds are now handled by a processing-block file called RsPointCloud.cs

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/unity/Assets/RealSenseSDK2.0/Scripts/ProcessingBlocks/RsPointCloud.cs

[mericgeren]

Thanks for the help you have offered. From RsPointCloud.cs it seems like it has nothing to do with Unity at all. Also, found this: "community.intel.com/t5/Items-with-no-label/Using-the-Realsense-Unity-wrapper-to-visualise-pointcloud-from/m-p/1290448#M17727". But, when i look at how RealSense Unity Wrapper renders point cloud into unity, i see that it can only get the data from a stream coming directly from the camera.

[MartyG-RealSense]

As Unity cannot directly display the live data received from the camera, the wrapper converts the data into a material so that it can be visibly rendered on-screen. Information about this can be found at #9588

[mericgeren]

Thanks for all the kind help you have offered. Point cloud appears not to be working out for me. So if you don't mind i am closing this issue.

[MartyG-RealSense]

I don't mind at all. You are very welcome.