accelerated network point cloud decoder

[bmegli]

The last left step in HVS accelerated depth streaming

This means:

• receiving and decoding HEVC Main10 depth data from new RNHVE with NHVD
• turning depth map into point cloud using camera matrix
• feeding point cloud to Unity Mesh data
• visualizing point cloud

Just to get the idea of amount of data:

• with Realsense D435 streaming 848:480 at 30 fps
• we get up to 848*480 = 407 040 points per frame
• meaning 12 211 200 dynamically generated points per second

[bmegli]

Receiving and decoding HEVC Main10 depth data from new RNHVE with NHVD

This is just a matter of configuring NHVD for HEVC.

Turning depth map into point cloud using camera matrix

• unprojection using pinhole camera model.
• camera matrix from the sender side with librealsense (intrinsics)
• I would start with CPU implementation (reference)
• then move to OpenCL accelerated GPU implementation

The CPU will have to bash through 400k points unprojection one by one, this looks like typical task for GPU.

Feeding point cloud to Unity Mesh data

Unity mesh has a limit of 65k points. This means we need up to 7 Meshes to accommodate single 848x480 depth image.

Edit: not anymore - Unity now supports 32 bit index format

I see two ways to do this efficiently:

1. From version 2019.3 Unity supports Mesh::SetVertices(NativeArray<T> inVertices)
2. There is old way with NativeRenderingPlugin, specifically this code

I really like the first approach:

• the code follows the current NHVD workflow
• works cross-API (OpenGL, DirectX, etc)
• probably resolves to direct graphics API call for vertex buffer update (like second approach)

The native 400k vertices data may be wrapped without ever touching data with: NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray

The second approach:

• has different and invasive workflow
• needs separate implementation for each graphics API (OpenGL, DirectX)
• in my case it would end up in OpenGL
• is as performant as possible (direct OpenGL)

Meshes:

• implement Unity side abstraction
• of circular buffer of vertices
• that below abstraction would have Meshes with 65k points
• that can be fed data from native array each frame

Having it implemented as circular buffer allows:

• controlling the decay of depth frames
• showing either just recent
• or more decaying gradually

Visualizing point cloud

Unity doesn't support point clouds so this means implementation of shader

Edit: Unity now supports meshes with points topology

This again mimics current implementation of NHVD:

network -> hardware decoder -> frame update -> fill Unity object with native data

[bmegli]

Receiving and decoding HEVC Main10 depth data from new RNHVE with NHVD

Turns out this needed adding additional configuration in HVD, NHVD, UNHVD

[bmegli]

Turning depth map into point cloud using camera matrix

The math behind Realsense D400 projection and deprojection

[bmegli]

Turning depth map into point cloud using camera matrix

• created HD2PC Hardware Depth to Point Cloud C library placeholder
• implemented rough software unprojection in initial commit

Edit:

• library was renamed to HDU Hardware Depth Unprojector

[bmegli]

Feeding point cloud to Unity Mesh data

Implemented proof of concept in:

• NHVD nhpcd branch
• UNHVD unhpcd branch

I see two ways to do this efficiently [...]

Used first approach. Works in realtime.

[bmegli]

Visualizing point cloud

Implemented in the same UNHVD commit as above.
Rendering through shader, there is just too little control with Mesh point topology.

So the final proof-of-concept workflow is:

• native side (thread)
  • network -> hardware decoder -> unprojector
• managed side
  • LateUpdate -> wrap data in NativeArray -> fill mesh -> shader

[bmegli]

Finished & merged.

The unprojection step is now in software through HDU, on my laptop this uses less then 2% of CPU and ~5 ms. To decide whether this is worth OpenCLing. A simple SIMD on x/y/z should give 2-3 times speedup even without GPU.

Encoding with 2 Mb/s bitrate gives reasonable results which proofs the concept of realtime hardware accelerated point cloud streaming.

This generally finishes:
bmegli/hardware-video-streaming#1

The rest is maintenance/polishing.