decide+implement API for durable virtual objects/collections: makeDurableKind?

[warner]

What is the Problem Being Solved?

Our plan for vat upgrade requires user code to use "Durable" to mark their precious-for-upgrade data. Instead of plain "virtual objects", they must use "durable objects" (aka "durable virtual objects", we're still seeking out a comfortable name). Instead of #2004 "virtual collections", they must use "durable collections".

For durable collections, the API is the same as for non-durable collections, so we only need to pick new names. I think @FUDCo is going with something like makeScalarDurableMapStore vs makeScalarBigMapStore, etc, for the four kinds of collections we currently have (makeScalarBig{Weak,}{Map,Set}Store). These collection objects can be serialized to the #4325 "baggage" to be retrieved by the new version during upgrade.

For non-durable virtual objects, we define makeInstance = makeKind(behavior), which creates the instance maker from a behavior record. For durable virtual objects, we need a way for the new version to establish replacement behavior, which requires some sort of handle that is distinct from the mere instance-making authority. @FUDCo 's current proposal is:

const handle = allocateDurableKind();
const makeInstance = makeDurableKind(handle, behavior);
baggage.set('name', handle);

The handle is an unforgeable serializable durable object. If the kind creates instances with vrefs like o+12/NN, then the handle would serialize to just o+12. It is technically exportable, although it's unlikely you would ever send it off-vat. Since it's durable, it can be stored in the baggage and retrieved by the next version. That next version can call makeDurableKind with the new behavior:

const handle = baggage.get('name');
const makeInstance = makeDurableKind(handle, newBehavior);

Description of the Design

Once we agree upon the API, the implementation needs are:

• define the durable versions of the collection creation functions, add to VatData
  • this is mostly trivial, the main work is to forbid non-durable data as keys/values
• define makeDurableKind, make the handle serializable
• docs to explain to userspace authors how to use the durability

We need to sketch out enough of the upgrade process to see what a vat's version2 would look like, and decide that we're happy with the approach. We don't need to implement baggage or upgrade for this ticket, but upgrade needs to follow smoothly from the API choices we make here.

Security Considerations

The "handle" should be closely held, to make sure only the right portion of the new vat's code gets the ability to commandeer the durable objects.

Storage Considerations

We need to decide how and when the vat's non-durable data gets deleted during upgrade, which may influence the implementation of the durable stores. I see two constraints:

• it should happen before the new-version userspace gets to run, or at least the new userspace should not be able to reach the non-durable data
• if the new version fails (i.e. a crash during upgrade() (or whatever design better vat-facing upgrade protocol (buildRootObject()? upgrade()?) #4382 decides), the old data should be retained, so the old version of the vat can continue operation

One option is that liveslots and the kernel have an agreement, e.g. "all vatstore keys starting with vsd. are durable and retained, all keys starting with vs. are non-durable and deleted). Then the kernel begins the upgrade-vat event by deleting all the non-durable keys. These deletions are stored in the crank buffer, and unwound if the upgrade crank fails.

Another option is that liveslots is responsible for enumerating and deleting all the non-durable keys itself. This reduces coupling between liveslots and kernel, however it would involve O(N) syscalls that'll be pretty expensive.

@FUDCo pointed out that each kind has an ID (o+NN), and each kind is either durable or not, and the metadata to indicate durability is stored in the vatstore. So if the kernel knows about the algorithm liveslots uses, the DB has enough information to figure out which keys should be kept. I'm concerned about reducing the coupling, though, so I'd be more comfortable with a protocol like vsd.*.

Test Plan

unit tests

[FUDCo]

Note: We already implement the durable collection creation functions and already implement the exclusion of non-durable stuff from them and from durable virtual objects.