Disparity Map and Valid Depth Mask creation

[ghost]

Required Info
Camera Model	D435i
Firmware Version	05.12.05.00
Operating System & Version	Ubuntu 18.04, melodic
Kernel Version (Linux Only)	5.4.0.65 - generic
Platform	PC
SDK Version	2
Language	python, opencv
Segment	image processing

Issue Description

Hello dear Realsense team,
Im learning Opencv with Python3. The author of the book, uses OpenCV's Openni2 functions: CAP_OPENNI_DISPARITY_MAP and CAP_OPENNI_VALID_DEPTH_MASK to get the results:

I have learned from documentation how to transform depth to disparity. However, when I transform it, I get following result (I changed dtype=np.uint8) :

From the issues #1778, #7431 , it seems that to get correct disparity I need to cast the array to float, but I get this:

Data is as follows:

So I clearly don't understand what's the problem. If disparity cast to float, it looks like valid depth mask image from book. If it is uint8, wavy image.

Kindly, waiting for your reply!

[MartyG-RealSense]

Hi @Novruz97 Does the output of the Python / OpenCV disparity map generation tutorial in this link produce an output from your D435i that resembles what you are aiming for, as shown in the image below from the tutorial?

https://docs.opencv.org/3.4/dd/d53/tutorial_py_depthmap.html

[MartyG-RealSense]

Having considered your case again, it seems as though you are trying to generate the disparity map inside of the librealsense SDK instead of doing it outside of the SDK. There are not many scripting references for generating a disparity map, as your own research has likely shown.

I was able to find a C++ script by a RealSense user who took the approach of accessing the disparity mode of the camera through the camera's Advanced Mode. Though you are using Python, that user's approach may provide some useful insights.

https://support.intelrealsense.com/hc/en-us/community/posts/360051896473/comments/360014626993

[ghost]

Hello @MartyG-RealSense, thank you for replying!
I have worked on finding disparity map through streaming stereo cameras. Using your link given in first response, I have managed to get results as follows:
`
pipeline.start(config)

try:
while True:

    for frame in range(5):
            pipeline.wait_for_frames()


    frames = pipeline.wait_for_frames()
    ir_l = frames.get_infrared_frame(1)
    ir_r = frames.get_infrared_frame(2)

    if not ir_l and not ir_r:
        continue

    left_ir = np.asanyarray(ir_l.get_data())
    right_ir = np.asanyarray(ir_r.get_data())

    stereo = cv2.StereoBM_create(numDisparities=64, blockSize = 25)

    disparity = stereo.compute(left_ir,right_ir).astype(np.float32)/16.0
    
    
    cv2.imshow('disp',disparity/128)
    if cv2.waitKey(1) & 0xff == 27:
        break

finally:
print(disparity[100:200,100:200]/128)
print(disparity.dtype)
pipeline.stop()
cv2.destroyAllWindows()
`

Several things to note:

1. I couldn't understand the numDisparities (search range,numbers divisible by 16) and blockSize (linear size for algorithm comparison) fully. It seems that it depends on the resolution. How to determine the optimal numbers?

2. I cast the computed disparity to float and divided by 16, also cv2.imshow(file, (disparity-min_disp)/numDisp) as I have found in examples of opencv for python in stereo_match.py . Why I did it? I don't know. But if I do not divide it, the resulting data type of disparity computation is int16 and the image I will get:
   

3. Pointing to the beginning of my issue explanation, according to Opencv's Openni2 functions:
   https://github.com/opencv/opencv/blob/b19f8603843ac63274379c9e36331db6d4917001/samples/cpp/videocapture_openni.cpp#L18-L19

the disparity should be in pixels (even in realsense documentation). But I don't think either types (int16 nor float32) gives correct result. Below is disparity computation without casting to float and dividing by 16:

I am still working on the second reply link you gave me, getting disparity with advanced mode, though Im not good at c++.

If you could give some advice on the upper questions, will much appreciate!
If you need any information, please notice me.

[MartyG-RealSense]

The subject of programming disparity maps is outside of my knowledge, unfortunately so I do not have advice that I can offer about that.

In regard to numDisparities though, Intel have a Python example for demonstrating stereo depth that makes use of it. This may provide you with some useful insights.

https://github.com/IntelRealSense/librealsense/blob/master/doc/depth-from-stereo.md#software-stereo

The full source code of this example is below.

import numpy
import cv2

from matplotlib import pyplot as plt
from matplotlib import cm

left  = cv2.imread("l_active.png", cv2.IMREAD_GRAYSCALE)
right = cv2.imread("r_active.png", cv2.IMREAD_GRAYSCALE)

fx = 942.8        # lense focal length
baseline = 54.8   # distance in mm between the two cameras
disparities = 128 # num of disparities to consider
block = 31        # block size to match
units = 0.512     # depth units, adjusted for the output to fit in one byte

sbm = cv2.StereoBM_create(numDisparities=disparities,
                          blockSize=block)

# calculate disparities
disparity = sbm.compute(left, right)
valid_pixels = disparity > 0

# calculate depth data
depth = numpy.zeros(shape=left.shape).astype("uint8")
depth[valid_pixels] = (fx * baseline) / (units * disparity[valid_pixels])

# visualize depth data
depth = cv2.equalizeHist(depth)
colorized_depth = numpy.zeros((left.shape[0], left.shape[1], 3), dtype="uint8")
temp = cv2.applyColorMap(depth, cv2.COLORMAP_JET)
colorized_depth[valid_pixels] = temp[valid_pixels]
plt.imshow(colorized_depth)
plt.show()

[MartyG-RealSense]

Hi @Novruz97 Do you require further assistance with this case, please? Thanks!

[ghost]

Hello @MartyG-RealSense , sorry for late reply!
So far, I was able to convert that guy's C++ code of using advanced mode to change to Disparity Mode and get data in python, though I didn't understand the data/img I got..
Here is the code if anyone needs:
`

def GetDisparityMap():
    imageW = 1280
    imageH = 720
    fps = 30
    
    config = rs.config()
    config.enable_stream(rs.stream.depth, imageW, imageH, rs.format.z16, fps)

    pipe = rs.pipeline()
    pipe_profile = pipe.start(config)
    dev = pipe_profile.get_device()

    depth_sensor = dev.first_depth_sensor()
    depth_sensor.set_option(rs.option.emitter_enabled, 1.0) # emitter ON

    advanced_mode = rs.rs400_advanced_mode(dev)
    depth_table = advanced_mode.get_depth_table()
    depth_table.disparityMode = 1 # enable disparity mode
    advanced_mode.set_depth_table(depth_table) # write changes
    # Need to wait and confirm the settings, otherwise, disparity mode is incorrect
    depth_table = advanced_mode.get_depth_table() # get depth table again to confirm settings
    print('Disparity Mode: ' + f'{depth_table.disparityMode}')

    frameset = pipe.wait_for_frames()
    disparity_frame = frameset.get_depth_frame()
    data = np.asanyarray(disparity_frame.get_data())

    disparity = data.view(np.uint16).reshape((imageH, imageW))
  #norm = np.iinfo(disparity)
  #norm = disparity/norm.max
  #norm = 255 * norm
  #disp_uint8 = norm.astype(np.uint8)

    cv2.imwrite('Disparity Map.png',disparity)

    print(disparity)
    print(disparity.shape)
    print(disparity.dtype)

    pipe.stop()

`

I tried to normalize the data to shape to np.uint8, but it throws an integer type error.
Here is the img:

[MartyG-RealSense]

I researched your question about normalization carefully. Whilst the RealSense SDK provides the ability to switch between depth and disparity modes with depth_to_disparity and disparity_to_depth, this does not seem possible with this particular script.

If your goal is to normalize a numpy array, the reference in the link below did suggest though that dividing the array by its norm would normalize it.

https://www.kite.com/python/answers/how-to-normalize-an-array-in-numpy-in-python

[ghost]

@MartyG-RealSense Thank you for your help in this!
I could solve my problem by normalization!

[MartyG-RealSense]

Great news @Novruz97 - thanks for the update!