encoding depth stream

[bmegli]

There is HEVC 3D extension that supports encoding depth maps.

According to this article HEVC/3DVC generally outperforms the tree-based compression methods.

One could possibly use reference 3D HEVC for software encoding and decoding.

As far as I know, those extensions are not supported by VAAPI, at least not on the Kaby Lake.

[bmegli]

In our case we need:

• hardware encoder/decoder
• with efficient pipeline
• it doesn't have to be perfect (lossy, reasonable artifacts are also ok)

Here we mean video codec used for depth map encoding. It will not work perfectly, there are some articles describing that.

Solution sketch

On Kaby Lake It should be possible to take advantage of:

• VAAPI 10 bit HEVC encoding:

~$ vainfo --display drm --device /dev/dri/renderD129
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Kaby Lake - 2.1.0
vainfo: Supported profile and entrypoints
...
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
...

• with P010LE pixel format

[AVHWDeviceContext @ 0x55c82d8b37e0] Matched "Intel i965 driver for Intel(R) Kaby Lake - 2.1.0" as known driver "Intel i965 (Quick Sync)".
25 : nv12
...
335 : p010le

• and RealsenseD435 Depth Units configured properly

It may be necessary to change the “Depth Units” in the advanced mode api. By default the D4 VPU
provides16bit depth with a depth unit of 1000um (1mm). This means the max range will be ~65m

With P010LE pixel format we have 1024 distinct values. This may be used for example for:

• 1 cm precision and 0-10 m range
• 0.5 cm precision and 0-5 m range
• ...

The Realsense Depth Units may be configured so that:

• we have 16 bit value
• corresponding to our needs with higher accuracy and P016LE
• P016LE may be put in place for P010LE (by design, the least significant digits will be ignored)

Conclusion

What we need is:

• extending HVE for HEVC support (see #4)
• configuring Realsense Depth Units so that it outputs correct range data to encode as P010LE (P016LE really)
• feeding P010LE Y plane with Realsense depth data, UV plane with dummy
• extending HVD for HEVC support
• extending RNHVE to support depth streaming apart from color/infrared
• extending UNHVD for depth data or creating separate project that decodes and feeds point cloud data to Unity

[bmegli]

Extending HVE for HEVC support

HVE was extended with various codecs support (including HEVC) in ac3a4c1.

P010LE example was added.

[bmegli]

Configuring Realsense Depth Units

We want the 10 MSB of P010LE Y plane, uint16 data to encode the depth.

• 1000 value for 5 m the Depth Units would be 5000um (5mm)
• but we need 1000 shifted by 6 bit left, which is 64000 and gives Depth Units shifted by 6 bits right which is 78,125 um, which is 78,125/1000000 m = 0,000078125 m

When trying to input it in realsense-viewer I am confronted with the message that 0.0001 (100 um, 0.1 mm) is the minimal value.

0.0001 m Depth Unit will give:

• 6.5536 m range
• 0,0064 m precision which is 6,4 mm (the smallest number that can be represented by 10 MSB of P010LE)

We could achieve any range/precision in software but this is good enough and we are trying to avoid software processing of depth/video data.

---

Is this really a hardware limit or just somebody on the Realsense team constrained it because nobody imagined such scenario (setting Depth Unit above device precision, not because device can have such precision but because it is needed for pure hardware processing)?

---

Concluding:

• we set 'Depth Unit' to 0.0001 m
• with 6.5536 m range
• and 6.4 mm precision

This can be done by loading json with rs2 settings through rs400::advanced_mode

[bmegli]

In the proof of concept we loaded all settings from json file.

I would rather set only the relevant Depth Units setting manually with RS2_OPTION_DEPTH_UNITS

Something along the line:

	rs2::pipeline_profile profile = pipe.start(cfg);
	rs2::depth_sensor depth_sensor = profile.get_device().first<rs2::depth_sensor>();

	if(!depth_sensor.supports(RS2_OPTION_DEPTH_UNITS))
	{
		cerr << "ERROR - device doesn't support RS2_OPTION_DEPTH_UNITS";
		exit(EXIT_FAILURE);
	}		
	depth_sensor.set_option(RS2_OPTION_DEPTH_UNITS, 0.0001f);

[bmegli]

This became self contained example and deserves own repository

[bmegli]

The code was moved to realsense-depth-to-vaapi-hevc10