Skip to content

How it works

Jump to bottom
Bartosz Meglicki edited this page Apr 27, 2020 · 14 revisions

Program structure

  • process user input (width, height, framerate, depth units, time to capture)
  • init file for raw HEVC output
  • init Realsense D400 device
  • init VAAPI encoder with HVE
  • read depth data from the camera
  • encode to HEVC Main10 profile
  • write to raw HEVC file
  • cleanup

The encoding

Realsense and VAAPI devices are configured to work together (no software depth processing on the host)

  • VAAPI is configured for HEVC 10 bit per channel P010LE pixel format
  • Realsense is configured to ouput P016LE (Y plane) compatible depth data
  • P016LE data is binary compatible with P010LE data
  • the data output by Realsense is directly fed to VAAPI hardware encoder
  • the P010LE color data is filled with constant value
Pixel format Side Side
MSB 15 14 ... 7 6 5 4 3 2 1 0 LSB
P016LE x x ... x x x x x x x x
P010LE x x ... x x 0 0 0 0 0 0

The P010LE has data only in 10 MSB, zeroes in 6 LSB (ignored values).

We have 10 bits to encode 16 bit Realsense depth data which means range/precission trade-off:

  • the trade-off is controlled with Realsense Depth Units (0.0001 - 0.01)
  • the best precision/worst range is 6.4 mm/6.5472 m (for Depth Units 0.0001)
  • the worst precission/best range is 64 cm/654.72 m (for Depth Units 0.01)
  • all trade-offs in between are possible

For range: 65472 in binary is 10 ones followed by 6 zeroes.
For precision: we discard 6 bits of data (64 times worse precision than set in Depth Units).

The range above max P010LE encodable range is returned as 0. The clamping is achieved with rs400 advanced mode (no depth processing on the host).

Depth Units

Example Depth Units and resulting range/precision after encoding:

Depth Units Precision Range Required firmware
0.0001 6.4 mm 6.5472 m
0.01 64 cm 654.72 m
0.000078125‬ 5 mm 5.115 m >= 5.12.1.0
0.000046875 3 mm 3.069 m >= 5.12.1.0
0.0000390625 2.5 mm 2.5575 m >= 5.12.1.0
0.00003125 2 mm 2.046 m >= 5.12.1.0
0.000023438 1.5 mm 1.5345 m >= 5.12.1.0
0.000019531 1.25 mm 1.27875 m >= 5.12.1.0
0.000015625 1 mm 1.023 m >= 5.12.1.0
0.0000078125 0.5 mm 0.5115 m >= 5.12.1.0

The rule is:

  • divide your desired Precision by 64 to get Depth Units
  • multiply above Depth Units by 65472 to get Range
  • example for desired 1 mm precision:
    • 0.001 m / 64 = 0.000015625 Depth Units
    • 0.000015625 * 65472 = 1.023 m Range

Decoding

There will come time when you will want to decode data and make some use of it.

See Decoding wiki page for hints.

Visual Example

The video shows point cloud streaming based on the methodology introduced in this repository.

Hardware Accelerated Point Cloud Streaming

Beyond depth encoding

The next logical step is adding texture to depth encoding.

RNHVE already does that. If you are interested see:

Clone this wiki locally