How it works

Overview

Native plugin unhvd-native is responsible for:

• receiving video/depth data on UDP port and hardware decoding (NHVD library)
• optionally unprojecting depth map to point cloud (HDU library )
• serving latest video frame/point cloud through easy interface

Unity side:

• UNHVD script is a wrapper around native unhvd-native library
• RawImageVideoRenderer script may be used for streaming to UI
• VideoRenderer script may be used for streaming to scene object
• PointCloudRenderer script may be used for streaming point clouds

Workflow

Video

Native library has a worker thread that collects frames from network, pushes frames to hardware for decoding.

Native library has functions that share frame data under mutex.

Example:

• unhvd_frame_begin function shares data under mutex
• unhvd_frame_end function releases the mutex

Unity side in LateUpdate script calls begin and end function. Between those calls it fills Texture data with native memory (never touched from managed side).

Data is never copied on both native and managed sides.

Point clouds

Point clouds are streamed as depth maps with optional color.

See depth encoding and infrared encoding explanations.

After hardware decoding there is additional step with unprojection and color mapping.
The data is unprojected to Unity compatible float3 array and color32 array.

The arrays are:

• shared between begin and end functions
• wrapped with NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray
• used for filling Mesh with:
  • Mesh::SetVertices(NativeArray<T> vertices) (Unity 2019.3)
  • Mesh::SetColors(NativeArray<T> colors) (Unity 2019.3)

This again never touches or copies data from managed side.

Both Video and PointCloud pipelines are graphics API agnostic (OpenGL, DirectX, Vulkan, etc.). This is left as responsibility of Unity.