[Proposal] Dynamic native modules

[gbryant-arm]

Introduction

It became apparent with #568 and #571 that integrating complex software with Veracruz is tedious. One reason for this is that complex software has its own build system, which isn't necessarily compatible with ours and requires some adjustments, e.g. installing additional packages in the build image or changing our build scripts.
On another note, there hasn't been any effort yet towards separating native modules from the runtime's TCB, which is a security liability.

We propose the introduction of dynamic native modules to:

• streamline the development of native modules by building them as separate artefacts
• reduce the runtime's TCB by executing native modules in a different memory space, for which isolation is offloaded to the kernel
  kernel; the user's TCB stays the same but they can decide which modules they trust

High-level considerations

Threat model

The crux of dynamic native modules resides in our capacity to sandbox them.
There is a tradeoff between I/O performance, usability and isolation, directly determined by our choice of sandboxing. For now, we propose to focus on applications that don't require high throughput between WASM programs and native modules, as opposed to e.g. short-lived services where a significant initialization overhead is not an option, or high I/O applications.
We propose the following threat model:

• Native modules should run in their own memory space, isolated from each other and from the Runtime Manager
• Native modules should have their own view of the host's filesystem, though they should be able to read the input from the WASM program and write the results after computation. This should be the only way for native modules to communicate with WASM programs or other native modules
• It should be possible to restrict the syscalls that can be performed by native modules. This should be specified in the policy. Note that we don't want to restrict native modules too much to the point of degrading their usability

Native module packaging, provisioning and attestation

Not unlike a Linux package, a native module is a program shipped as an archive of binaries, dynamic libraries and other files required for computation that aren't provided by the participants (e.g. database). We could opt for a standard format like Debian packages or tar/zip balls.
The native module must implement a standard ABI ((de)serialization) to read the caller's (WASM program) input and write the results after computation.
The native module must specify an entry point (i.e. what binary should be run when the service is invoked).
The native module's package location (e.g. github release and version/hash) and target (executing platform) must be specified in the policy.
The package must be fetched by the Runtime Manager and attested (file hash) before computation, and should probably be cached in the runtime, possibly pushed to the (untrusted) disk to speed context initialization.
The policy must specify which programs can access the native module's interface file (e.g. /services/tflite).

Execution model

The execution is similar to a FaaS:

1. The WASM program invokes the service by writing to the native module's interface file (e.g. /services/tflite)
2. The Runtime Manager intercepts the write, checks the WASM program's permissions against the policy
3. The Runtime Manager initializes the sandbox: exposes the WASM program's input to the sandbox, then runs the service as a separate process within the sandbox. We assume the native module has already been checked against the policy and unpacked
4. The native module executes in the sandbox, using its partial view of the filesystem for the computation, then writes the results to its filesystem (ideally shared with the WASM program's for performance reasons)
5. The Runtime Manager cleans up the native module context: kills the service and tears down the sandbox (includes removing the native module's filesystem)

Technical choices

Sandbox

Hypervisor-based solutions: overkill
Docker: complex, big overhead starting/killing containers, requires a modern and full-fledged operating system
sandbox2: self contained, doesn't depend on any recent kernel features; policy mostly describes allowed/forbidden syscalls and fs access, but also supports eBPF

ABI (serialization, IPC)

Between WASM and native modules: keep using postcard on top of the fs
Between Runtime Manager and sandbox: Runtime Manager execs sandbox with optional CLI arguments (one-way communication)

[hugovincent]

Some questions:

• Do we need the dynamic/arbitrary module pull functionality at this time? I think these considerations (packaging, attestation, fetching/caching, encoding in the policy etc) are largely orthogonal to the sandboxing and interface considerations and security implications. I (meta-)propose separating into two separate proposals.
  • How do native modules get "registered" with the Veracruz runtime? Where does the code that provides the synthetic special files on the VFS etc come from and how is it packaged?
• How does native module code access files (or file-likes/synthetics) on the Veracruz VFS? This could be for example with something like FUSE, or could be implemented as part of the sandbox (trap FS syscalls from native module, check path/name, either pass through to kernel or service syscall directly from VFS), or the execution model could pre-arrange VFS contents of relevance to be copied to host FS files visible to the sandbox. Note that the latter may give rise to concurrency hazards for some use-cases.
  • Another option would be to move the VFS to kernel-provided backing (e.g. ramfs) so that it can be accessed by both the runtime/wasm program and native modules. cc @ShaleXIONG
  • What is the approach for access control to the Veracruz VFS by native modules?
• Are we sure that restricting native modules to only interacting with Wasm programs via the VFS is sufficiently expressive for our use-cases? It's certainly a lot simpler than RPC-style functionality to natively call module code from Wasm programs (and vice versa).
• On the execution model: is synchronous batch mode as you describe (wasm calls native module, native module runs to completion and is torn down, execution returns to wasm) sufficient for our use-cases? Again, it's much simpler and less error-prone than concurrent module+wasm execution or even interspersed but singly threaded execution, but limiting.

[hugovincent]

Another existential consideration: do we want uniformity / homogeneity between different types of sandboxes such that for example a pipeline of both wasm and native programs can be constructed? This has implications for the sandbox API surface too, especially if we add a Morello sandbox type as well.