How it works

Overview

Native plugin unhvd-native is responsible for:

• receiving video/depth data on UDP port and hardware decoding (NHVD library)
• optionally unprojecting depth map to point cloud (HDU library)
• serving latest video frames/point cloud through easy interface

Unity side:

• UNHVD script is a wrapper around native unhvd-native library
• RawImageVideoRenderer script may be used for streaming to UI
• VideoRenderer script may be used for streaming to scene object
• PointCloudRenderer script may be used for streaming point clouds
• GPUPointCloudRenderer script does above with accelerated unprojection

Workflow

Video

Native library has a worker thread that collects frames from network, pushes frames to hardware for decoding.

Native library has functions that share frame data under mutex.

Example:

• unhvd_frame_begin function shares data under mutex
• unhvd_frame_end function releases the mutex

Unity side in LateUpdate script calls begin and end function. Between those calls it fills Texture data with native memory (never touched from managed side).

Data is never copied on native library side and managed Unity side.

Point Clouds

Point clouds are streamed as depth maps with optional color.

See depth encoding and infrared encoding explanations.

PointCloudRenderer

After hardware decoding there is additional step with unprojection and color mapping.
The data is unprojected to Unity compatible float3 and color32 arrays.
Same internal mesh data layout is set in Unity with Mesh.SetVertexBufferParams (Unity 2019.3)

The arrays are:

• shared between begin and end functions
• wrapped with NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray
• used for filling Mesh with Mesh.SetVertexBufferData(NativeArray<T>, ... ) (Unity 2019.3)

This never touches or copies data from managed side.

GPUPointCloudRenderer

There is no unprojection step on the native library side.
Depth and color frames are shared like in Video workflow for filling textures.
Unity side LateUpdate consumes textures with compute shader.
Compute shader performs unprojection on GPU.
The output is used by classic shaders to render the point cloud.

Data is never copied on native library side and managed Unity side.
After filling textures data is not touched by CPU again.

---

All Video and PointCloud pipelines are graphics API agnostic (OpenGL, DirectX, Vulkan, etc.). This is left as responsibility of Unity.