how to get to swingset-on-XS

[warner]

@dckc @kriskowal and I had a meeting last week to figure out the steps necessary to get our system using XS in a meaningful way. Here's the dependency diagram we assembled from that meeting. I'm not 100% sure of the syntax, so I intend to rewrite it with something more obvious. In this image, I believe a link from B -> A means "A depends upon B", and a parallel link from C -> A which uses a dotted line means "A depends upon one of (B or C)". Also the bold ovals mark the path figured should be the shortest one.

original source

[warner]

Here are some notes on what some of the nodes mean, not complete:

harden

hardenOnXS: we need a working harden() function. We might achieve this either with hardenShim or sesCompartmentOnXS

• hardenShim: get the make-hardener shim (https://github.com/Agoric/SES-shim/tree/master/packages/make-hardener) working under XS, to give us a working harden() function. XS has an extended form of Object.freeze() for use when marking early objects as suitable for copying into ROM, but it doesn't have the semantics we need (and indeed we must vat code from using it, because that might cause surprises)
• sesCompartmentOnXS: get our full Compartment shim working on XS (https://github.com/Agoric/SES-shim/tree/master/packages/ses), including the harden that it provides inside Compartments (??)

module loading

dynamicModuleLoading: we need to make importBundle work to load our kernel, vats, and contracts. The bundles themselves might be in "nestedEvaluate" format (a JSON-serialized object, which has one property which is a JSON-serialized object, with one property per module name), or they might be an Endo "archive" (a ZIP file with a manifest and one member per module).

• nestedEvaluate is our current mechanism: bundle-source creates this format by default, using rollup to locate and assemble all the files. rollup also converts each file from it's original format (ESM or CommonJS) into a form that uses require (it's not exactly CommonJS, but it's certainly not a module, and does not require support for import in the loading environment)
  • When import-bundle loads the nestedEvaluate format, it needs the ability to create a new Compartment, but then it uses compartment.evaluate(string) to load the pieces and turn the source code into objects. This is the low-tech solution: it does not require implementation of any portion of our Compartment module-loader API
  • Alternatively (and preferably), we could use an Endo archive format instead of nestedEvaluate. This requires our Compartment instances to implement the new import method, which entails the whole stack of new Compartment module-loader APIs (static module records, loader hooks, etc). The XS Compartment does not do this, so we'd either need to get Moddable to add it (and the spec we're writing is still pretty raw), or use our SES-shim to get our implementation working under XS. This is the high-tech solution: the archives would be cleaner, and there's some hope that more debugging information would be retained (there's less string mangling going on).

CommonJS

All of our own code is written in an ESM format, however our package.json does not mark them as such (no type: 'module'), and we still rely upon using -r esmto load everything (#527). A handful of our active dependencies are written in CommonJS format, so we need ourbundle-source+import-bundle` solution to accomodate both. User (contract) code is likely to include even more, since we want users to have access to anything from NPM that happens to be SES-compatible (i.e. does not depend upon ambient authority or sloppy mode).

rollup knows how to handle either ESM or CommonJS files as input, so when we use the nestedEvaluate format for our bundles, the source file formats don't matter.

(@kriskowal please confirm) The Endo make-archive tool knows how to parse both ESM and CommonJS for their dependencies, so it is capable of creating our kernel/vat bundles out of modules with both formats. However to actually load those, the runtime code (compartment.import, the moduleLoader it uses, the loadHook) must be able to understand that format. This involves a parser. The SES-shim's Compartment implementation uses Babel (or Babel-lite?), some portions of which involve dependencies upon Node.js built-ins, or assume access to file IO. Getting Babel/Babel-lite to run under XS seems difficult.

[warner]

@kriskowal gave me a big brain-dump of the xsnap project he and @dckc have been working on. Here's my attempt to capture it.

First off, the ultimate plan looks like this:

We'll have one kernel process, X xs-vat-worker processes, and Y actual vats. The vats can share a process to reduce the number of processes (performance). Vats will not share a process when we're unwilling to let them be mutually vulnerable to platform bugs like SES breaches, engine-level memory corruption, resource-exhaustion attacks that our metering doesn't catch, and Spectre/Meltdown -style confidentiality violations. We also use multiple processes to enable parallelization of reloads, since we expect each process to be single-threaded.

The protocol spoken between the kernel process and each xs-vat-worker process contains "deliveries" (send message X to vat Y) and "syscalls" (vat Y sends syscall X to the kernel and expects a response). It also contains meta requests like "create vat Y from snapshot S1", "save vat Y to snapshot S2", and some bootstrap controls.

In this plan, each vat has a separate XSMachine C struct. The "dispatch unit" has its own XSMachine, and exclusive access to the message pipes up to the kernel process. The dispatch unit is given access to C functions that can create the vat XSMachine instances, invoke functions on its global (to inject deliveries), and control the snapshot machinery.

the immediate plan

As a step towards that ultimate plan, we're starting with a simpler version that replaces the "dispatch unit" with a separate library (written in Node.js, rather than XS). Each vat gets its own XS-based process, with exactly one XSMachine each. We draw the process boundaries one level lower in the hierarchy.

The XS process still uses netstring-framed commands on pipes (listens on fd3, emits on fd4), but these operate at a lower level than the kernel-to-worker protocol described above. The commands are roughly:

• evaluate a string
• invoke a specific function on the start-compartment global object, wait for the engine to quiesce (promise queue is empty), and return some result
• perform GC
• snapshot the heap and write it to a file

In addition, the process can be started with an argument that loads the heap from a named snapshot file, rather than starting with an empty environment.

The xsnap package will contain the C code that implements this process, as well as a Node.js library that:

• knows the location of the binary
• offers a function which spawn()s the binary, wires up the message pipes, and implements the protocol
• exposes functions like c = create(snapshot), await c.evaluate(string), results = c.deliver(delivery), c.snapshot(filename)

Then we'll probably write an xs-vat-worker library, whose instances live inside the kernel process, just like the existing xs-node-worker approach. When creating a new vat, this will use the xsnap library to:

• create a new (empty) XS child
• evaluate a pre-rollup'ed SES-shim library, to add lockdown() to the global scope
• evaluate more code to execute lockdown()
• evaluate a pre-rollup'ed supervisor library in the child
  • this supervisor will accept calls to a property it adds to the global object
  • the supervisor will include liveslots, and code to build a vat from some bundle object it gets over the wire

This library will accept kernel delivery objects, but will invoke the evaluate and deliver APIs provided by the xsnap library.

[dckc]

fixed in #2225

xsnap currently doesn't explose "perform GC"; presumably the need for that will become apparent in upcoming work.

[dckc]

Experiment: 57 SwingSet tests fail with managerType: 'xsnap'

I tried changing local to xsnap in loadVat.js (as detailed below) and running all the SwingSet tests: 57 tests failed

Many of them seem due to a documented limitation: setup()-based vats must use a local Manager

For example:

  test/test-vpid-kernel.js:58

   57:   }                                       
   58:   await kernel.createTestVat(name, setup);
   59:   return { log, getSyscall };             

  Rejected promise returned by test. Reason:

  Error {
    message: 'setup()-based vats must use a local Manager',
  }

The one change I made w.r.t. 37a658c :

$ git diff
diff --git a/packages/SwingSet/src/kernel/loadVat.js b/packages/SwingSet/src/kernel/loadVat.js
index ed267fc3..e46f2a2b 100644
--- a/packages/SwingSet/src/kernel/loadVat.js
+++ b/packages/SwingSet/src/kernel/loadVat.js
@@ -293,7 +293,7 @@ export function makeVatLoader(stuff) {
       ...creationOptions,
       setup,
       enableSetup: true,
-      managerType: 'local',
+      managerType: 'xsnap',
     };
     const manager = await vatManagerFactory(vatID, managerOptions);
     addVatManager(vatID, manager, managerOptions);

---

p.s. for follow-up, see #2260