How it works

Overview

Native plugin (NHVD) is responsible for:

• receiving video data on UDP port (MLSP library)
• hardware video decoding (HVD library)
• optionally unprojecting depth map to point cloud (HDU library )
• serving latest video frame/point cloud through easy interface

Unity side:

• NHVD script is a wrapper around native NHVD library
• RawImageVideoRenderer script may be used for streaming to UI
• VideoRenderer script may be used for streaming to scene object
• PointCloudRenderer script may be used for streaming point clouds

Workflow

Video

Native library has a worker thread that collects frames from network, pushes frames to hardware for decoding.

Native library has functions that share frame/point cloud data under mutex.

Example:

• nhvd_frame_begin function shares data under mutex
• nhvd_frame_end function releases the mutex

Unity side in LateUpdate script calls begin and end function. Between those calls it fills Texture data with native memory (never touched from managed side).

Data is never copied on both native and managed sides.

Point clouds

Point clouds are streamed as depth maps with optional greyscale color.

See depth encoding and infrared encoding explanations.

After hardware decoding there is additional step with unprojection (Depth to Point Cloud). The depth data is unprojected to Unity compatible float3 array and color32 array.

The arrays are:

• shared between begin and end functions
• wrapped with NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray
• used for filling Mesh with:
  • Mesh::SetVertices(NativeArray<T> vertices) (Unity 2019.3)
  • Mesh::SetColors(NativeArray<T> colors) (Unity 2019.3)

This again never touches data from managed side and never copies data.

Both Video and PointCloud pipelines are graphics API agnostic (OpenGL, DirectX, Vulkan, etc.), this is left as responsibility of Unity.