accelerated textured depth streaming

[bmegli]

Related to #1

Potential Encoding

Currently only luminance Y plane of HEVC Main10 P010LE (encoding details) is used.

This means that P010LE chroma (interleaved U/V) channel is still unused.

The U/V plane is:

• half the size of Y (H/2 height and same stride as Y)
• with P010LE this means 16 bits for each color value of which only 10 MSBs are used

Funnily enough this is exactly the size of D435 infrared greyscale data (which is 8 bit per sample with Y8 format).

If the stride of infrared matches its width it should be possible to:

• directly map infrared data to interleaved U/V plane data
• at the cost of losing every second value (P010LE 16 bits of which only 10 MSBs are used)
• the 2 excess LSB bits should not affect the encoding too much (LSB) and may be ignored

This is somewhat like 4:2:2 chroma sampling (if you consider IR "to be" the U/V chroma.

The decoding

The decoding would use the same mechanism as currently with additional notes:

• the left D435 imager is already aligned with depth data (no CPU expansive alignment process needed)
• the vertices' colors may be extracted during unprojection ignoring every second value (e.g. left or mean)
• and mapped directly to Unity Mesh vertex color data on native side (as currently with point clouds)
  • SetColors(NativeArray<T>)
  • NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray

Final notes

This should not affect latency in any significant way but may affect the required bitrate

• the trivial chroma U/V plane is encoded, decoded and ignored anyway
• increasing chroma complexity will affect the bitrate

It is difficult to say anything about the quality of texture encoding without performing some tests

• the encoded and ignored 2 LSB bits from every second value may have minor impact
• the resulting "4:2:2" sampling may have some impact (every second horizontal value)

[bmegli]

Working proof-of-concept between:

• NHVD textured-depth branch
• UNHVD textured-point-clouds branch

Some first subjective impressions:

• textured point clouds need more bitrate
• the effect for human perception is profound and worth it

Some quick screenshots, same view.

Point cloud only:

Textured point cloud:

[bmegli]

This is already implemented, merged into master branches and working between:

• NHVE
• UNHVD

It would be nice at add some (short and simple) video showing the functionality.

[bmegli]

Done, documented, video added.