Need help alleviating "ghosting" or "smearing"

[t-kole-yipp]

I have an issue which requires a little bit of drawing to explain.
I'm essentially trying to chain multiple realsenses d435's together to create one big view from above.

This is going well, for the most part. This process takes place in Unity. By converting the depth images to point clouds, I can align the devices to create a single map. However, there is one effect that I can't seem to get rid of, and I need help to understand what is causing it and which settings to tweak to remove it as well as possible.

Consider the following scenario:

A large, 3m tall box is standing on the floor. A d435 is positioned 4 meters from the floor, looking down.
In our situation, the box would cast a big "shadow" on the floor, the part where the d434 cannot see the box. This is acceptable, as I'm not interested in the floor.

You would expect, in a perfect scenario, that the output would look like this:

Left, the pixel depth of the realsense, and right the reprojected pointcloud.
However, the actual output looks a bit more like this:

The depth seems to bleed out across the image.
It's barely noticeable when just looking at the depth image, but when reprojected, these "ghost points" can smear out well over a meter at times. Quite significant, as shown here:

Additionally, there seems to be a "magic" threshold where it occurs. If the distance between surface A and B is less than about 1.5 meters, the smearing occurs. If it's more than that value, it pops away.
I'd be fine with having no data there, if that's the other option.

The different presents the realsense viewer application provides, such as "high accuracy", help a little bit.
But it removes only a little bit of "smearing" while removing lots of good data.
What settings am I to tweak to get rid of this as much as possible?

Help is much appreciated!
Kind regards, T.

[MartyG-RealSense]

Hi @t-kole-yipp Given that the camera is 4 meters from the observed surface, the noise is not likely related to being too close to the observed object.

If the noise is being generated by the floor, the position of the camera is fixed and you do not care about capturing the floor, you may be able to clean up the data by applying a Threshold filter. This allows you to define a minimum and maximum observable depth distance, and depth data outside of this range is excluded from the image.

So if the floor is exactly 4 meters away and the camera is mounted in a fixed position, setting the threshold filter to a maximum distance of 3.95 meters may exclude the floor whilst still capturing the box.

Alternatively, you could set the maximum Threshold filter distance to 10 meters (the camera's full depth sensing range) to test whether the noise is caused by the floor only being partially captured (as my experience is that in the Viewer, the maximum distance of the Threshold filter tends to be set to 4 meters by default if this filter is enabled).

In the Viewer, you can find the Threshold filter by expanding open the Post Processing section of the options side-panel under the Stereo Module controls, and then expanding open the Threshold Filter option in the post-processing filter list.

A Threshold filter can also be programmed into your own application using C++, C# or Python code.

Bear in mind that the floor may be rendered poorly on the depth image if it is dark grey / black or has a reflective surface. In that situation, excluding the floor from the image with a sub-4m threshold setting may work best. A shadow on the floor from the box could also count as a dark grey texture that is difficult for the camera to read.

If the floor surface is not reflective and is not completely black then projecting a strong light source onto the area to brighten the surfaces may help to make them more depth-readable and reduce shadows.

There was a project where somebody built a makeshift conveyor with different shaped LEGO bricks passing along the conveyor under a camera mounted above. To deal with inconsistencies in lighting that made object detection more awkward, the project's designer "blasted" the conveyor with a powerful overhead light source to 'out-compete' other light sources in the area that may cause disruption by creating shadows.

Details of the project and a making-of article for it are in the link below.

#5393

---

Edit: having reviewed the case again, the problem that you are experiencing seems to more closely resemble the kind of ghost noise where the points get "dragged out".

#4553

In that conversation, a member of the RealSense team gives a list of advice on how to deal with the problem.

#3741 (comment)

[t-kole-yipp]

Thank you for the quick reply.
In your edit I think you have found what I'm experiencing, too.
I will try to tweak the settings as described, including the decimation filter, as I do not need the full depth resolution.
If this fixed the issue for me I will follow up with the results.

[MartyG-RealSense]

@t-kole-yipp Thanks very much - I look forward to your report.

[t-kole-yipp]

I have followed some of the advice given in the other posts, to see if I could improve the quality of the reprojected point cloud.
The devices are updated, calibrated, an no post-processing is enabled.

I started trying out different presets in a controlled environment. The camera is on the ceiling, and I've stacked some boxes to create some sort of cardboard scarecrow.

(images taken in Realsense Viewer. left: depth, right: pointcloud)

This is the default preset:

This one seems to be one of the worst of the presets. The surfaces are relatively clean, but there's this halo effect around the edges of objects, resulting in the "smearing" or "ghosting" as shown before.

Hand preset:

I suppose this one is more for up-close settings, but it was worth a shot.

High Accuracy preset:

The depth accuracy might be a little bit (but hardly noticeably) higher, but thin objects disappear in the thick black border around them.

High Density preset:

This one might be the best one for my use case yet. It's similar to default, but with less ghosting.

Medium Density preset:

This one seems to be somewhere in between High Accuracy and High Density.

The suggested "decimation" filter does hardly anything to remove the ghosting, I don't think I'll be enabling it for this use case.

One of the effects that is persistent throughout almost all presets is this one:

Below a certain depth threshold (maybe 30cm from object to floor), the object will blend into the floor completely, without a hard edge in between. I will be tweaking parameters more, but this might take some time, because it's not always clear which parameters do what.

If anyone has any more insights, please let me know!

[MartyG-RealSense]

Thanks very much for the test results. Yes, Medium Density is a balance between Fill factor and accuracy.

Do the results on High Density show any improvement if you increase the value of the Laser Power setting please? On the RealSense Viewer program, this can be found in the Stereo Module section of the options side-panel, under the Controls sub-section. Increasing laser power can reduce "sparseness" in depth data, whilst reducing laser power can increase sparseness.

[t-kole-yipp]

Increasing laser power seems to make the surfaces less "blobby", but doesn't help get rid of the ghosting.

[MartyG-RealSense]

Next, could you try turning off the Auto Exposure option in the Viewer and seeing if the image improves, please?

[t-kole-yipp]

Hi MartyG,

Enabling or Disabling Auto Exposure does not seem to do much.
The exposure setting makes the result much worse when it is too low, but does not increase significantly when it is high.

[MartyG-RealSense]

If the depth mis-readings surrounding the box are the result of the box's shadow, there is an interesting long discussion at the link below about shadows and the "false positive" depth readings they may generate.

#2129 (comment)

There was not a clear solution reached at the end of the conversation though, just the general suggestion that "custom post processing steps" may correct the problem.

The SR300 had a "depth confidence" function that the 400 Series stereo cameras do not have. It was once suggested that in the absence of that feature on the 400 Series, the Second Peak Threshold function of the 400 Series' "Advanced Mode" may be a substitute.

#3185 (comment)

[t-kole-yipp]

Hi MartyG, I've taken a look at the posts, maybe I can try tweaking Second Peak Threshold a bit to see if it improves.
I might just write a custom postprocessing step where if a pixel drops or raises more that x cm I discard it.

I'll respond if I get that to work.

[MartyG-RealSense]

Hi @t-kole-yipp Do you still require assistance with this case, please? Thanks!

[t-kole-yipp]

Hi Marty, thanks for checking in.
I'm scheduled to proceed with this next week, I'll keep you posted.

[MartyG-RealSense]

Okay, great - thanks so much for the update!

[t-kole-yipp]

@MartyG-RealSense

I tried to solve the ghosting problem at the source as much as possible, but I haven't had much luck.
I also tried to write a custom realsense postprocessing function to strip away strong gradients, but this had worse results than expected. Not only that, applying the filter to multiple inputs broke it completely. Different views were coming from different cameras, and they jittered like a broken VHS. (in C# - Unity to be specific, but that's for another Issue)

I've decided to focus my efforts more on alleviating the issue on my end, after the point clouds are rendered, by trying to get rid of the speckles in less blunt way.

If anyone has any further ideas, I'm open to suggestions!

[MartyG-RealSense]

@t-kole-yipp I have a sense of familiarity, as there is a different case today where some of the same concepts are being discussed: multiple cameras, point clouds and how to process the clouds to minimize ghost elements.

IntelRealSense/realsense-ros#1343 (comment)

[MartyG-RealSense]

Hi @t-kole-yipp Do you have an update for us, please? Thanks!

[t-kole-yipp]

Hi @MartyG-RealSense ,

None of the solutions posted here or in related threads seem to have helped significantly.
I'm "cleaning up" the reprojected point cloud as much as I can, by dilating less aggressively.
This gets my image clean enough for my use case, at the cost of some quality.
The "Medium Density" preset seems to be the best for me, I've decided to go with that one (plus additional laser power, as suggested earlier).

Thanks for the help so far.

[MartyG-RealSense]

Perhaps you could create your own custom preset file that copies-and-pastes the settings of the Medium Density preset but improves the fill rate (though not as much as the High Density preset).

[MartyG-RealSense]

Hi @t-kole-yipp Do you require further assistance with this case please? Thanks!

[MartyG-RealSense]

Hi @t-kole-yipp Do you require further assistance with this case please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.