Very noisy RGBD input through pyrealsense2 but not through realsense-viewer

[fabiopoiesi]

Required Info
Camera Model	D400
Firmware Version	05.13.00.50
Operating System & Version	Linux Ubuntu 18.04
Kernel Version (Linux Only)	4.9.253-tegra
Platform	NVIDIA Jetson TX2
SDK Version	2.50.0
Language	python
Segment	Robot

Issue Description

When I access the camera from python I get very noisy inputs like this one (visualised with Open3D):

The same scene seen from the realsense-viewer is instead like this

It is not clear to me why this is happening.
Here is how I start the camera

self.__pipeline = rs.pipeline()
rs_config = rs.config()

rs_config.enable_stream(rs.stream.depth,
                                    self.__config.camera_info.depth_width,
                                    self.__config.camera_info.depth_height,
                                    rs.format.z16,
                                    30)

rs_config.enable_stream(rs.stream.color,
                                    self.__config.camera_info.color_width,
                                    self.__config.camera_info.color_height,
                                    rs.format.rgb8,
                                    30)

self.__pipeline.start(rs_config)

Here how I access the fames:

self.__align_to = rs.stream.depth
self.__align = rs.align(self.__align_to)
frames = self.__pipeline.wait_for_frames()
aligned_frames = self.__align.process(frames)
depth_frame = np.asanyarray(aligned_frames.get_depth_frame().get_data())
color_frame = np.asanyarray(aligned_frames.get_color_frame().get_data())

This is how I installed pyrealsense2:

git clone https://github.com/IntelRealSense/librealsense.git .
mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_EXAMPLES=false -DBUILD_PYTHON_BINDINGS=bool:true -DPYTHON_EXECUTABLE=/usr/bin/python3.6
make -j4
make install

[MartyG-RealSense]

Hi @fabiopoiesi The first difference between your image and the Viewer's is likely to be that you are performing a rare color to depth alignment with self.__align_to = rs.stream.depth instead of the usual depth to color alignment. On the D435 and D435i camera models where the field of view size of the color sensor is smaller than the depth sensor FOV, color to depth alignment can cause the image to stretch out to fit to the larger depth FOV size. You can test depth to color alignment to see whether it makes a positive difference by using self.__align_to = rs.stream.color

The Viewer also applies a range of depth colorization, color scheme and post-processing filter settings by default. In user-created scripts, these settings have to be deliberately programmed into the application.

#7767 (comment) and #7089 (comment) provide Python scripting resources for configuring the color scheme and histogram on / off status.

A RealSense team member provides an example of Python code for defining post-processing filters at #1672

[fabiopoiesi]

Thank you for your prompt reply.

I just discovered by chance that the cause of the noise was caused by the lights in the office! Incredible...

The lights are standard neon lights.
By turning them off I got this result from python without using the filters.

Problem solved then.

[MartyG-RealSense]

Thanks very much @fabiopoiesi for the update. It's excellent news that you found a solution.

Fluorescent lights such as ceiling strip lights can create interference on the image because they contain a heated gas that flickers at frequencies that are difficult to see with the human eye. There are a couple of approaches for dealing with this:

1. Use an FPS speed that is close to the operating frequency of the particular lights in your location. This may be 30 FPS or 60 FPS.

2. In European regions the lights may use a 50 Hz frequency that the RealSense SDK cannot compensate for with an FPS setting because it does not support 50 FPS. In that situation, you can set an option called power line frequency and define a value of '50' or '60' for it depending on whether your lights are operating at 60 Hz (such as in North American regions) or 50 Hz (European regions).

[fabiopoiesi]

I tried setting a different power line frequency, but it didn't help solving the problem when lights are on.
The problem occurs occasionally also with outdoor lighting.
I think this could be a combination of factors, I checked the infrared signal and noticed that the table surface is somewhat reflective and I can see the reflection of the IR projector.

I am exploring other parameters amongst the options.

Thank you for your support.

[MartyG-RealSense]

Yes, reflections from a surface make it more difficult for the camera to read depth detail. Adding a physical filter product over the lenses on the outside of the camera can significantly reduce this negative effect, which can also occur outdoors when observing reflective surfaces such as window panes. The image below shows the difference of observing glass without a filter and with one.

In the past week Intel also launched a new RealSense model based on the D435 called the D435f that has built-in light filters on its sensors. More information can be found at #10560

[rancheng]

Another option to try with pyrealsense2 is to save the customized json from realsense-viewer and load it from the code:

dev = cfg.get_device()
advnc_mode = rs.rs400_advanced_mode(dev)
advnc_mode.load_json(json_string)

as mentioned in issue 6075#

[fabiopoiesi]

very useful indeed!

thank you

[david-s-martinez]

hey @fabiopoiesi I have solved this problem at my research lab by sticking a polarizing filter made out of plastic on the rgb sensor and stereo module. I think they were intended for office windows. I believe you can find it at a hardware store, but there are also such filters for smartphone screens. I also disbled the autoexposure and set it to a fixed value since our application assumed stable light conditions. You can change this parameter from the realsense viewer and then import the configuration by means of a json file.

[MartyG-RealSense]

@davidmartinez13 Thanks very much for your advice to @fabiopoiesi :)

Hi @fabiopoiesi Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.