implement kernel-side `syscall.dropImport` handler, refcounting, GC action delivery #3109

warner · 2021-05-17T08:03:12Z

What is the Problem Being Solved?

This issue covers the last big push of GC work for #3106.

Once #3108 is done, we need the kernel to respond to vats calling syscall.dropImport by clearing the reachability flag, which will trigger refcount changes.

The overall tasks are:

modify the implemention of syscall.dropImport to clear the reachability flag in the clist
impement both "reachable" and "recognizable" refcounts on all kernel objects
update these refcounts during clist additions/removals
track the refcounts that touch zero, add them to a new maybeFreeKrefs ephemeral set
process that set at end-of-delivery, to see what GC actions should be scheduled
add those actions to a new durable GC Actions set (in the kernelDB)
scan the pending actions before each new delivery, to prioritize GC work
deliver the highest-priority non-negated GC Action to the appropriate vat
update all unit tests as necessary, add new ones

Test Plan

Unit tests in test-kernel.js and test-gc-kernel.js.

The text was updated successfully, but these errors were encountered:

warner · 2021-05-31T22:50:02Z

I was mis-assigning the overall refcounting work to the wrong ticket (#2946), so I'm re-purposing this one to cover that work, in addition to its original small slice (just syscall.dropImport).

This adds a new DB key for each kernel object, which contains a comma-separated pair of (reachable, recognizable) reference counts. The vatKeeper functions that add/remove clist entries now update these counts: * the 'reachable' count is incremented whenever the reachable flag in the clist entry is changed from false to true, and decremented in the other direction * increment/decrementRefCount take flags to indicate whether the kref is being uses as an import, or an export (since exports don't count towards object refcounts), and whether both reachable+recognizable counts are being changed or merely the recognizable count * deleteCListEntry takes a flag to disable decref, which will be used when deleting an object export (since the refcounts will be zero by that point) `deadKernelPromises` and `purgeDeadKernelPromises` were generalized into `maybeFreeKrefs` and `processRefcounts()`. Object processing is disabled for now, by commenting out the point where krefs would be added to the durable GC Actions set. This will continue retire Promises as before, but will leave all Objects in place. refs #3109

This makes the final step to activate swingset garbage collection (at least pieces covered by #3106): * implement kernel-side handling of the three GC syscalls (`dropImports`, `retireImports`, `retireExports`), some of which happens during translation, the remainder in `kernelSyscalls.js` * implement kernel-side translators for the GC deliveries (`dropExports`, `retireExports`, `retireImports`) * populate the GC Actions set during `processRefcounts()` * change `c.step()/run()` to execute any pending GC Action before consulting the run-queue * add test-gc-kernel.js to exercise basic checks Also, a few miscellaneous error messages were improved. closes #3109 refs #3106

This adds a new DB key for each kernel object, which contains a comma-separated pair of (reachable, recognizable) reference counts. The vatKeeper functions that add/remove clist entries now update these counts: * the 'reachable' count is incremented whenever the reachable flag in the clist entry is changed from false to true, and decremented in the other direction * increment/decrementRefCount take flags to indicate whether the kref is being uses as an import, or an export (since exports don't count towards object refcounts), and whether both reachable+recognizable counts are being changed or merely the recognizable count * deleteCListEntry takes a flag to disable decref, which will be used when deleting an object export (since the refcounts will be zero by that point) `deadKernelPromises` and `purgeDeadKernelPromises` were generalized into `maybeFreeKrefs` and `processRefcounts()`. Object processing is disabled for now, by commenting out the point where krefs would be added to the durable GC Actions set. This will continue retire Promises as before, but will leave all Objects in place. refs #3109

This makes the final step to activate swingset garbage collection (at least pieces covered by #3106): * implement kernel-side handling of the three GC syscalls (`dropImports`, `retireImports`, `retireExports`), some of which happens during translation, the remainder in `kernelSyscalls.js` * implement kernel-side translators for the GC deliveries (`dropExports`, `retireExports`, `retireImports`) * populate the GC Actions set during `processRefcounts()` * change `c.step()/run()` to execute any pending GC Action before consulting the run-queue * add test-gc-kernel.js to exercise basic checks Also, a few miscellaneous error messages were improved. closes #3109 refs #3106

This adds a new DB key for each kernel object, which contains a comma-separated pair of (reachable, recognizable) reference counts. The vatKeeper functions that add/remove clist entries now update these counts: * the 'reachable' count is incremented whenever the reachable flag in the clist entry is changed from false to true, and decremented in the other direction * increment/decrementRefCount take flags to indicate whether the kref is being uses as an import, or an export (since exports don't count towards object refcounts), and whether both reachable+recognizable counts are being changed or merely the recognizable count * deleteCListEntry takes a flag to disable decref, which will be used when deleting an object export (since the refcounts will be zero by that point) `deadKernelPromises` and `purgeDeadKernelPromises` were generalized into `maybeFreeKrefs` and `processRefcounts()`. Object processing is disabled for now, by commenting out the point where krefs would be added to the durable GC Actions set. This will continue retire Promises as before, but will leave all Objects in place. refs #3109

This makes the final step to activate swingset garbage collection (at least pieces covered by #3106): * implement kernel-side handling of the three GC syscalls (`dropImports`, `retireImports`, `retireExports`), some of which happens during translation, the remainder in `kernelSyscalls.js` * implement kernel-side translators for the GC deliveries (`dropExports`, `retireExports`, `retireImports`) * populate the GC Actions set during `processRefcounts()` * change `c.step()/run()` to execute any pending GC Action before consulting the run-queue * add test-gc-kernel.js to exercise basic checks Also, a few miscellaneous error messages were improved. closes #3109 refs #3106

This adds a new DB key for each kernel object, which contains a comma-separated pair of (reachable, recognizable) reference counts. The vatKeeper functions that add/remove clist entries now update these counts: * the 'reachable' count is incremented whenever the reachable flag in the clist entry is changed from false to true, and decremented in the other direction * increment/decrementRefCount take flags to indicate whether the kref is being uses as an import, or an export (since exports don't count towards object refcounts), and whether both reachable+recognizable counts are being changed or merely the recognizable count * deleteCListEntry takes a flag to disable decref, which will be used when deleting an object export (since the refcounts will be zero by that point) `deadKernelPromises` and `purgeDeadKernelPromises` were generalized into `maybeFreeKrefs` and `processRefcounts()`. Object processing is disabled for now, by commenting out the point where krefs would be added to the durable GC Actions set. This will continue retire Promises as before, but will leave all Objects in place. refs #3109

This makes the final step to activate swingset garbage collection (at least pieces covered by #3106): * implement kernel-side handling of the three GC syscalls (`dropImports`, `retireImports`, `retireExports`), some of which happens during translation, the remainder in `kernelSyscalls.js` * implement kernel-side translators for the GC deliveries (`dropExports`, `retireExports`, `retireImports`) * populate the GC Actions set during `processRefcounts()` * change `c.step()/run()` to execute any pending GC Action before consulting the run-queue * add test-gc-kernel.js to exercise basic checks Also, a few miscellaneous error messages were improved. closes #3109 refs #3106

This is a relatively independent portion of the kernel-side GC work. "GC Actions" are generated by the upcoming `processRefcounts()` function that the kernel will run after each delivery is complete. For example, if the kernel observes an object's "reachable" refcount drop to 0 during a crank, and the count is still zero when `processRefcounts()` runs, it will push a `dropExports` action (for the exporting vat and dropped kref) into the GC Actions set. The GC Actions set is stored durably in the kernelDB, and consulted by a call to `processNextGCAction()` just before pulling anything else off the run-queue. This call must select the highest-priority action to take, and check to see if the kref in question has been re-exported since the action was enqueued (removing it from the queue if so). It then returns the action to take, or `undefined` if there are none. This commit adds the implementation of `processNextGCAction()`, and the unit test which exercises its various possibilities. It also adds a new key to the kernelDB (named `gcActions`) which holds the durable set, and several new kernelKeeper methods to manipulate it, including `get`, a simple `add` function, and a `get` to replace the set wholesale (since `processRefcounts` can be more efficient if it updates the durable set in a single operation rather than a long sequence of adds). It does not change the behavior of the kernel: nothing yet adds GC Actions (this will happen in `processRefcounts()`, nor does anything example the durable set to act upon any queued actions (this will happen in `getNextMessage()`). refs #3109

This adds a new DB key for each kernel object, which contains a comma-separated pair of (reachable, recognizable) reference counts. The vatKeeper functions that add/remove clist entries now update these counts: * the 'reachable' count is incremented whenever the reachable flag in the clist entry is changed from false to true, and decremented in the other direction * increment/decrementRefCount take options to indicate whether the kref is being uses as an import, or an export (since exports don't count towards object refcounts), and whether both reachable+recognizable counts are being changed or merely the recognizable count * deleteCListEntry takes a flag to disable decref, which will be used when deleting an object export (since the refcounts will be zero by that point) `deadKernelPromises` and `purgeDeadKernelPromises` were generalized into `maybeFreeKrefs` and `processRefcounts()`. A few other cleanups/improvements were made: * `kernel.dump()` adds `.objects` with refcounts * `kernelObjectExists()` was added as a stable way to ask the question refs #3109

warner added the enhancement New feature or request label May 17, 2021

warner mentioned this issue May 17, 2021

The SwingSet GC Plan (save RAM for unreachable objects) #3106

Closed

9 tasks

warner self-assigned this May 17, 2021

warner added the SwingSet package: SwingSet label May 17, 2021

warner changed the title ~~implement kernel-side syscall.dropImport handler~~ implement kernel-side syscall.dropImport handler, refcounting, GC action delivery May 31, 2021

This was referenced May 31, 2021

chore(swingset): add GC Actions #3222

Merged

add kernelKeeper index to efficiently track the importing vats of each kernel object #3223

Open

warner mentioned this issue Jun 2, 2021

gcAndFinalize might be insufficient on Node.js in test-transcript.js #3240

Open

warner mentioned this issue Jun 7, 2021

match incref/decref calls within kernel #3264

Closed

warner closed this as completed in d05eeb2 Jun 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement kernel-side `syscall.dropImport` handler, refcounting, GC action delivery #3109

implement kernel-side `syscall.dropImport` handler, refcounting, GC action delivery #3109

warner commented May 17, 2021 •

edited

Loading

warner commented May 31, 2021

implement kernel-side syscall.dropImport handler, refcounting, GC action delivery #3109

implement kernel-side syscall.dropImport handler, refcounting, GC action delivery #3109

Comments

warner commented May 17, 2021 • edited Loading

What is the Problem Being Solved?

Test Plan

warner commented May 31, 2021

implement kernel-side `syscall.dropImport` handler, refcounting, GC action delivery #3109

implement kernel-side `syscall.dropImport` handler, refcounting, GC action delivery #3109

warner commented May 17, 2021 •

edited

Loading