Help optimizing D455 for outdoor agricultural mapping with RTAB-Map

[VanjaLjubobratovic]

Hello,
I'm working on a project which aims to use RealSense D455 and RTAB-Map in ROS2 in order to create
an areal view map of agricultural terrains. The idea is that those systems would already be in place on an autonomous plant sprayer, used for something else, and they would create the map on the side so you can monitor the plants.

I have already developed and set up everything needed for that, but the results I'm getting are not really satisfying so I was wondering if anyone could point me in the right direction.

Here are my results including my mosaic and the point cloud I produce:
Mosaic
PLY Cloud
Screenshot of a part of the cloud in MeshLab
Image of the area so you have an idea what it actually looks like

Below you can see my rs_launch.py params in the form of a script I use for launching:

#!/bin/bash

resolution="1280x720x30"

LC_NUMERIC="en_US.UTF-8"; ros2 launch realsense2_camera rs_launch.py \
rgb_camera.profile:=$resolution \
depth_module.profile:=$resolution \
enable_gyro:=true \
enable_accel:=true \
unite_imu_method:=1 \
gyro_fps:=400 \
accel_fps:=200 \
enable_infra1:=false \
enable_infra2:=false \
enable_sync:=false \
align_depth.enable:=true \
pointcloud.enable:=false \
initial_reset:=true \
depth_module.enable_auto_exposure:=true \
decimation_filter.enable:=false \
temporal_filter.enable:=true \
spatial_filter.enable:=true \

Basically what I'm most interested in hearing is whether this looks like a camera problem, RTAB-Map problem or maybe both so I know what to look into.
Thanks!

[MartyG-RealSense]

Hi @VanjaLjubobratovic Is the main problem the purple color tinting on the RGB image? If it is then this is a known issue on the D455 camera model. It is not a bug, but a characteristic of the D455 hardware that can occur when the RGB sensor is over-saturated with bright sunlight.

Further information about this phenomenon can be found on the final page of the Specification Clarification document at the link below.

https://www.intelrealsense.com/download/13629/

D455 cameras manufactured after early 2022 had a design change to help reduce this effect. Other actions that can be taken include setting a Region of Interest in the bottom half of the image, or applying an ND filter product over the top of the camera lenses on the outside of the camera.

[VanjaLjubobratovic]

@MartyG-RealSense, no, sorry, I was probably a bit unclear.
My main problem is that I'm getting a lot of depth errors on the plants so they end up very fuzzy and smeared when you look from above (look at the garden area of the mosaic).

I realize it is hard to capture small and thin objects such as leaves in great detail, but I was wondering if, after all, there is something I can do to improve this or it just is what it is.

[MartyG-RealSense]

If it looks fuzzy and smeared then it may be more to do with the RGB color texture (which is being mapped onto the depth coordinates by align_depth) than it is to do with the depth data. You may get a clearer image if you maximize the RGB Sharpness setting to a value of '100'.

[VanjaLjubobratovic]

Ok, I'll try that out and get back to you.

[VanjaLjubobratovic]

@MartyG-RealSense So maybe we misunderstood each other. What I meant by "smearing" and "fuzzy" will be illustrated by the following 2 images:

Have a look at those plants along the wall and on the stairs. They look fairly decent.
But when you look at this area of the cloud from the side or from above...

You can also see it on the columns and other objects

So basically what I wanted to ask is what mistake have I made to produce such results. Can I configure the camera better? Is it not a camera problem at all and I should try to change RTAB-Map parameters? Is it just my poor recording and I should try to capture everything from all the possible angles?

Thank you for your time!

[MartyG-RealSense]

The effect resembles a phenomenon called point cloud spray that is discussed at IntelRealSense/librealsense#7021 (comment)

Applying a Visual Preset pre-made camera configuration (also known as a json file) can help to deal with it. #2445 provides information about applying a preset in ROS.

[VanjaLjubobratovic]

Ok, I'll try that out. Thanks.

[VanjaLjubobratovic]

@MartyG-RealSense So applying the high accuracy preset helps somewhat, but the problem doesn't really go away. I have also been trying to mitigate it by recording with a greater angle towards the ground and that does also somewhat help since the errors aren't completely perpendicular to the Z axis now, but still they do persist on the objects that are captured on the edges of the field of view.

After reading various different discussions on the issue where no definitive answer is given, would I be correct in assuming there isn't much more I can do to fix this?

[MartyG-RealSense]

You could potentially position a second camera adjacent to the first one with their fields of view overlapping so that the overall combined field of view is wider and can see more of the scene.

Alternatively, you could try using 1280x720 resolution if you are not doing so already so that the image 'zooms out' a little and can include a little more information at the edges.

[VanjaLjubobratovic]

I will test 720p along with the preset since I have been using 480p since the test shown in the first comment because it seems to be producing fewer errors.
I will report on the results and after that we can probably close this issue.

Also, a question about the 2 camera suggestion: is this accomplished with rs_multi_camera_launch or do I need to do something else?

[MartyG-RealSense]

Intel have a ROS guide for "stitching" pointcloud views from two cameras together into a single pointcloud in ROS1, though the process would not be quite the same in ROS2.

https://github.com/IntelRealSense/realsense-ros/wiki/Showcase-of-using-2-cameras

[MartyG-RealSense]

Hi @VanjaLjubobratovic Do you require further assistance with this case, please? Thanks!

[VanjaLjubobratovic]

Hi, sorry I haven't been able to test it. Bad weather lately. Keep the issue open for a little while longer, please.

[MartyG-RealSense]

That's no problem at all to keep the issue open for a further time period. Thanks very much for the update!

[VanjaLjubobratovic]

So after some experimentation I have managed to find a solution which I would call satisfactory.

In terms of camera settings:

• I've set temporal filter off as per documentation it can increase point cloud spray if the camera is moving
• I'm using the optimal profile which is 848x480x60 as per the documentation
• I'm using some of the camera presets. High Accuracy if there will be no filtering done on the cloud afterwards, and if you are filtering as I will be, maybe medium density can work great also, because it wont remove as much good points from your clouds.

In terms of camera operation:

• I'm recording at as big of an angle to the ground as I can since most of the errors are along the X axis, if the X axis points towards the ground, when I rotate the cloud to get an areal view, a lot of the spray will be covered up by good pixels

In terms of filtering:

• I have my custom node which is used to filter the point clouds produced by RTAB-Map and I'm using distance thresholding (0.5m to 4.0m). Already implemented by a RTAB-Map function for decoding messages on /rtabmap/mapData topic.
• My custom filter which calculates the median distance and standard deviation of all the points from the camera in the XY plane and removes those which don't fit into the specified interval. The effect is that point cloud edges are "trimmed". This removes a lot of outlier clusters where some outliers are being clustered together in the middle of empty space.
• PCL Statistical Outlier Removal filter which is specifically made to remove point cloud spray. It is used last because it is ineffective at removing clusters of outliers. Since I'm concerned with real time applications I was forced to make a multi threaded implementation which is decent for my needs.

The results of filtering on the point cloud RECORDED WITH ORIGINAL CAMERA SETTINGS:
BEFORE

AFTER

When all of the steps above are actually combined, the result should be quite good.
Here's my code if someone's interested in implementation details

I hope someone finds this useful.

[MartyG-RealSense]

It's great to hear that you achieved a solution, @VanjaLjubobratovic - thanks so much for sharing the details!

[VanjaLjubobratovic]

Since I have nothing more to add I'm gonna close this. Thanks for the help.