feat: implement the durable kind API

Closes issue #4495

packages/ERTP/globals.d.ts

@@ -1,6 +1,7 @@
 interface VatData {
   defineKind: function;
   defineDurableKind: function;
+  makeKindHandle: function;
   makeScalarBigMapStore: function;
   makeScalarBigWeakMapStore: function;
   makeScalarBigSetStore: function;

packages/SwingSet/docs/virtual-objects.md

@@ -1,39 +1,49 @@
-# TODO (update to include new stores and durable objects stuff)
+# TODO (update to include new stores stuff)
 
 # Vat Secondary Storage
 
 The kernel currently provides two secondary storage mechanisms for the use of (application) code running in a vat:
 
-- Virtual objects
+- Virtual and durable objects
 - Persistent stores
 
-Each of these is accessed via a global made available to vat code.
+These are accessed via properties of the `VatData` global made available to vat code, or more stylishly by importing from `@agoric/swingset-vat/src/storeModule.js` (that's a working placeholder module which we will be replacing it with a more ergonomic package name once we figure out what that should be).
 
-## Virtual Objects
+The APIs described here all have to do with storing data on disk.  However, you should understand an important distinction made in these APIs between the labels "virtual" and "durable".  In our usage here, things that are "virtual" will automatically swap their state to disk storage and thus don't eat up RAM space in the running vat process even if they grow large in number.  In contrast, things that are "durable" are not only stored on disk but survive the lifetime of the vat process holding them and may be retrieved later in a future version of the vat.
 
-A virtual object is an object with a durable identity whose state is automatically and transparently backed up in secondary storage.  This means that when a given virtual object is not in active use, it need not occupy any memory within the executing vat.  Thus a vat can manage an arbitrarily large number of such objects without concern about running out of memory.
+## Virtual and Durable Objects
 
-A virtual object has a "kind", which defines what sort of behavior and state it will possess.  A kind is not exactly a data type, since it comes with a concrete implementation, but it indicates a family of objects that share a set of common behaviors and a common state template.
+A virtual or durable object (which we'll abbreviate VDO to save breath) is an object whose state is automatically and transparently backed up in secondary storage.  This means that when a given VDO is not in active use, it need not occupy any memory within the executing vat.  Thus a vat can manage an arbitrarily large number of such objects without concern about running out of memory.
 
-A vat can define new kinds of virtual object by calling the `defineKind` function provided as a vat global:
+A VDO has a "kind", which defines what sort of behavior and state it will possess.  A kind is not exactly a data type, since it comes with a concrete implementation, but it indicates a family of objects that share a set of common behaviors and a common state template.
 
-  `defineKind(descriptionTag, init, actualize, finish)`
+A vat can define new kinds of VDOs by calling the `defineKind` or `defineDurableKind` functions:
 
-The return value from `defineKind` is a maker function which the vat can use to create instances of the newly defined object kind.
+  `maker = defineKind(descriptionTag, init, actualize, finish)`
+or
+  `maker = defineDurableKind(kindHandle, init, actual, finish)`
+
+The return value from `defineKind` or `defineDurableKind` is a maker function which the vat can use to create instances of the newly defined VDO kind.
 
 The `descriptionTag` parameter is a short description string for the kind.  This is the same kind of tag string you would use in a call to `Far`.  It will appear on `.toString()` representations of corresponding `Presence` objects that are exported to remote vats.
 
-The `init` parameter is a function that will be called when new instances are first created. It is expected to return a simple JavaScript object that represents the initialized state for the new virtual object instance.  Any parameters passed to the maker function returned by `defineKind` are passed directly to the `init` function.
+A `kindHandle` is a type of durable object that can be used to identify the kind in a later incarnation of the vat.  The usage of a kind handle rather than a simple tag string is the main thing that distinguished the two kind definition functions.  You obtain a kind handle for use in `defineDurableKind` by calling
+
+  `kindHandle = makeKindHandle(descriptionTag)`
+
+where `descriptionTag` is exactly the same as the same named parameter of `defineKind`.  The difference is that a kind handle is itself that a durable object that may be stored for later retrieval, and used in a future call to `defineDurableKind` to associate new behavior with the kind in question.
+
+The `init` parameter is a function that will be called when new instances are first created. It is expected to return a simple JavaScript object that represents the initialized state for the new VDO instance.  Any parameters passed to the maker function returned by `defineKind`/`defineDurableKind` are passed directly to the `init` function.
 
-The `actualize` parameter is a function that binds an in-memory instance (the "Representative") of the virtual object with the virtual object's state, associating such instances with the virtual object's behavior.  It is passed the virtual object's state as a parameter and is expected to return either:
+The `actualize` parameter is a function that binds an in-memory instance (the "Representative") of the VDO with the VDO's state, associating such instances with the VDO's behavior.  It is passed the VDO's state as a parameter and is expected to return either:
 
 1. A new JavaScript object with methods that close over the given state.  This returned object will become the body of the new instance.  This object can be empty; in such a case it can serve as a powerless but unforgeable "marker" handle.
 
 2. A new JavaScript object populated with objects as described in (1).  These will become facets of the new instance.  The returned object will be an object mapping to the facets by name.
 
-The `actualize` function is called whenever a new virtual object instance is created, whenever such an instance is swapped in from secondary storage, and whenever a reference to a virtual object is received as a parameter of a message and deserialized.  The `actualize` function for a given VDO kind must always return the same shape of result: if it returns a single instance body, it must always return a single instance body; if it returns a object full of facets, it must always return an object with the exact same facet names.
+The `actualize` function is called whenever a new VDO instance is created, whenever such an instance is swapped in from secondary storage, and whenever a reference to a VDO is received as a parameter of a message and deserialized.  Note that for any given VDO kind, the shape of the value returned by the `actualize` function may not vary over successive calls.  That is, if it's a single facet, it must always be a single facet, and if it's multiple facets it must always be the same set of multiple facets.
 
-The `finish` parameter will, if present (it is optional), be called exactly once as part of instance initialization.  It will be invoked immediately after the `actualize` function is called for the first time.  In other words, it will be called after the instance per se exists but before that instance is returned from the maker function and thus becomes available to whoever requested its creation.  `finish` is passed two parameters: the virtual object's state (exactly as passed to the `actualize` function) and the virtual object itself. The `finish` function can modify the object's state in the context of knowing the object's identity, and thus can be used in cases where a validly initialized instance requires it to participate in some kind of cyclical object graph with other virtual objects.  It can also be used, for example, to register the object with outside tracking data structures, or do whatever other post-creation setup is needed for the object to do its job.  In particular, if one or more of the object's methods need to refer to the object itself (for example, so it can pass itself to other objects), the `finish` function provides a way to capture that identity as part of the object's state.
+The `finish` parameter is optional. It is a function that, if present, be called exactly once as part of instance initialization.  It will be invoked _immediately_ after the `actualize` function for that instance is called for the very first time.  In other words, it will be called after the instance per se exists but before that instance is returned from the maker function to whoever requested its creation.  `finish` is passed two parameters: the VDO's state (exactly as passed to the `actualize` function) and the VDO itself. The `finish` function can modify the object's state in the context of knowing the object's identity, and thus can be used in cases where a validly initialized instance requires it to participate in some kind of cyclical object graph with other VDOs.  It can also be used, for example, to register the object with outside tracking data structures, or do whatever other post-creation setup is needed for the object to do its job.  In particular, if one or more of the object's methods need to refer to the object itself (for example, so it can pass itself to other objects), the `finish` function provides a way to capture that identity as part of the object's state.
 
 For example:
 
@@ -64,7 +74,7 @@ For example:
   const makeCounter = defineKind('counter', initCounter, actualizeCounter, finishCounter);
 ```
 
-This defines a simple counter object with two properties in its state, a count and a name.  You'd use it like this:
+This defines a simple virtual counter object with two properties in its state, a count and a name.  You'd use it like this:
 
 ```javascript
   const fooCounter = makeCounter('foo');
@@ -78,7 +88,7 @@ This defines a simple counter object with two properties in its state, a count a
   console.log(`${barCounter.getName()} count is ${barCounter.getCount()`); // "new bar count is 1"
 ```
 
-Suppose you instead wanted to provide the the increment and decrement capabilities as independent facets.  A simplified version of the above (without the name property, counter registry, and `reset` method) might look like:
+Suppose you instead wanted to provide a durable version with the increment and decrement capabilities made available as independent facets.  A simplified version of the above (without the name property, counter registry, and `reset` method) might look like:
 
 ```javascript
   const initFacetedCounter = () => ({ counter: 0 });
@@ -101,7 +111,8 @@ Suppose you instead wanted to provide the the increment and decrement capabiliti
     };
   }
 
-  const makeFacetedCounter = defineKind('counter', initCounter, actualizeCounter);
+  const facetedCounterKind = makeKindHandle('durable counter');
+  const makeFacetedCounter = defineDurableKind(facetedCounterKind, initCounter, actualizeCounter);
 ```
 
 Which you'd use like:
@@ -154,6 +165,6 @@ Additional important details:
 
   `state.zot.push(4);`
 
-- A virtual object can be passed as a parameter in messages to other vats.  It will be passed by presence, just like any other non-data object you might send in a message parameter.
+- A VDO can be passed as a parameter in messages to other vats.  It will be passed by presence, just like any other non-data object you might send in a message parameter.
 
-- A virtual object's state may include references to other virtual objects. The latter objects will be persisted separately and only deserialized as needed, so "swapping in" a virtual object that references other virtual objects does not entail swapping in the entire associated object graph.
+- A VDO's state may include references to other VDOs. The latter objects will be persisted separately and only deserialized as needed, so "swapping in" a VDO that references other VDOs does not entail swapping in the entire associated object graph.

packages/SwingSet/src/liveslots/liveslots.js

@@ -1073,6 +1073,7 @@ function build(
     VatData: {
       defineKind: vom.defineKind,
       defineDurableKind: vom.defineDurableKind,
+      makeKindHandle: vom.makeKindHandle,
       makeScalarBigMapStore: collectionManager.makeScalarBigMapStore,
       makeScalarBigWeakMapStore: collectionManager.makeScalarBigWeakMapStore,
       makeScalarBigSetStore: collectionManager.makeScalarBigSetStore,

packages/SwingSet/src/liveslots/virtualObjectManager.js

@@ -3,6 +3,8 @@
 
 import { assert, details as X, q } from '@agoric/assert';
 import { Far } from '@endo/marshal';
+import { parseVatSlot } from '../lib/parseVatSlots.js';
+
 // import { kdebug } from './kdebug.js';
 
 /**
@@ -412,6 +414,8 @@ export function makeVirtualObjectManager(
   /**
    * Define a new kind of virtual object.
    *
+   * @param {string} kindID  The kind ID to associate with the new kind.
+   *
    * @param {string} tag  A descriptive tag string as used in calls to `Far`
    *
    * @param {*} init  An initialization function that will return the initial
@@ -492,8 +496,7 @@ export function makeVirtualObjectManager(
    * reference to the state is nulled out and the object holding the state
    * becomes garbage collectable.
    */
-  function defineKindInternal(tag, init, actualize, finish, durable) {
-    const kindID = `${allocateExportID()}`;
+  function defineKindInternal(kindID, tag, init, actualize, finish, durable) {
     let nextInstanceID = 1;
     const propertyNames = new Set();
 
@@ -648,11 +651,47 @@ export function makeVirtualObjectManager(
   }
 
   function defineKind(tag, init, actualize, finish) {
-    return defineKindInternal(tag, init, actualize, finish, false);
+    const kindID = `${allocateExportID()}`;
+    return defineKindInternal(kindID, tag, init, actualize, finish, false);
+  }
+
+  let kindIDID;
+  const kindDescriptors = new WeakMap();
+
+  function reanimateDurableKindID(vobjID, _proforma) {
+    const { subid: kindID } = parseVatSlot(vobjID);
+    const raw = syscall.vatstoreGet(`vom.kind.${kindID}`);
+    assert(raw, X`unknown kind ID ${kindID}`);
+    const durableKindDescriptor = JSON.parse(raw);
+    const kindToken = Far('kind', {});
+    kindDescriptors.set(kindToken, durableKindDescriptor);
+    return kindToken;
+  }
+
+  function makeKindHandle(tag) {
+    if (!kindIDID) {
+      kindIDID = `${allocateExportID()}`;
+      syscall.vatstoreSet('kindIDID', kindIDID);
+      vrm.registerKind(kindIDID, reanimateDurableKindID, () => null, true);
+    }
+    const kindID = `${allocateExportID()}`;
+    const kindIDvref = `o+${kindIDID}/${kindID}`;
+    const durableKindDescriptor = { kindID, tag };
+    const kindHandle = Far('kind', {});
+    kindDescriptors.set(kindHandle, durableKindDescriptor);
+    registerValue(kindIDvref, kindHandle, false);
+    syscall.vatstoreSet(
+      `vom.kind.${kindID}`,
+      JSON.stringify(durableKindDescriptor),
+    );
+    return kindHandle;
   }
 
-  function defineDurableKind(tag, init, actualize, finish) {
-    return defineKindInternal(tag, init, actualize, finish, true);
+  function defineDurableKind(kindToken, init, actualize, finish) {
+    const durableKindDescriptor = kindDescriptors.get(kindToken);
+    assert(durableKindDescriptor);
+    const { kindID, tag } = durableKindDescriptor;
+    return defineKindInternal(kindID, tag, init, actualize, finish, true);
   }
 
   function countWeakKeysForCollection(collection) {
@@ -674,6 +713,7 @@ export function makeVirtualObjectManager(
   return harden({
     defineKind,
     defineDurableKind,
+    makeKindHandle,
     VirtualObjectAwareWeakMap,
     VirtualObjectAwareWeakSet,
     flushCache: cache.flush,

packages/SwingSet/src/storeModule.js

@@ -3,6 +3,7 @@
 export const {
   defineKind,
   defineDurableKind,
+  makeKindHandle,
   makeScalarBigMapStore,
   makeScalarBigWeakMapStore,
   makeScalarBigSetStore,