Overlaying Edge Map Preset on top of Depth Image

[SeanyTea]

---

Required Info
Camera Model	D435
Firmware Version
Operating System & Version	Linux
Kernel Version (Linux Only)	5.15.76
Platform	Raspberry Pi
SDK Version	2.0
Language	Python
Segment

Issue Description

Hello, I have been playing around with the presets for the Intel Realsense D435, specifically the Edge Map preset. I would like to know if there is a way to access the depth image at the same time as the edge map. For instance, I can obtain the two images shown below using two different presets:

where the first one contains the edge map, and the second contains the depth info. Ideally, I would like to overlay the first on top of the second to enhance the edges/facial features.

I am aware that the following two lines control the edge preserving filter:
"param-texturecountthresh": "24", "param-texturedifferencethresh": "126",,

but am unsure if I can access the frame prior to applying this filter.

[MartyG-RealSense]

Hi @SeanyTea I can see a couple of possibilities for capturing before and after images.

1. Capture the original image. Then stop the pipeline, load the edge map json file and re-start the pipeline to apply the new configuration, then capture the edge-map image.

2. Capture the original image. Then change the TextureCountThreshold and TextureDifferenceThreshold values by accessing them via Advanced Mode with Python scripting and capture the subsequent edge map image. The RealSense SDK has a Python example script for Advanced Mode options at the link below.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/python-rs400-advanced-mode-example.py

If you need to display both the original and edge-map image simultaneously in real-time, an approach that may work is to define two pipelines, and use the depth frames from pipeline 1 in pipeline 2 and apply the TextureCountThreshold and TextureDifferenceThreshold settings changes to pipeline 2. A Python example for setting up two separate pipelines is at #5628 (comment) whilst a script example for two separate pipeline configurations and frame references is at #1735 (comment)

[SeanyTea]

Hi @MartyG-RealSense ,

Thanks for the response. I am looking to display these images in real-time, so I don't think the first approach works quite well as it takes ~ 1.5 seconds to reload the pipeline.

The multi-pipeline approach seems like an intuitive approach, but the references you listed appear to use either (a) multiple devices/cameras, or (b) two different sensors (i.e. RGB/IMU/depth), and thus two separate pipelines can be created. When I attempt to initialize two different pipelines, I run into either of the following errors at pipeline2.start(config):

1. RuntimeError: open(...) failed. UVC device is streaming! (Using pipeline1's config)
2. RuntimeError: No device connected. (Using its own separate config)

[MartyG-RealSense]

The script at #5628 (comment) should work with a single camera and put depth + color on one pipeline and the IMU on the other pipeline on its own. Do you receive errors if you test the script at that link, please?

Use of Raspberry Pi could also be a factor in your problem, as some RealSense users have had problems with accessing the IMU on Pi boards, as discussed at #11089

[SeanyTea]

I see, I should have clarified beforehand that we would like to make use of strictly the depth sensor, and disregard the RGB color camera altogether.

On a related note, but perhaps a side topic, would you be able to describe the details of the TextureCountThreshold? I cannot seem to find any documentation on it. Please let me know if that would be possible, and if so, if I should open a new issue.

If there are no other options to get the raw depth info prior to performing the edge detection, you can go ahead and close this issue. Thanks!

[MartyG-RealSense]

Advanced Mode functions are not documented, as they interact with each other in complex ways and so Intel controls them with machine-learning algorithms instead. RealSense users are free to experiment with the settings though to test how they affect the image.

In the case of Texture Count Threshold though, my Intel RealSense colleagues have provided a detailed explanation about it (and also Texture Difference Threshold) at #10608 (comment)

[MartyG-RealSense]

Hi @SeanyTea Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.