amortize kernel-source bundling process during large unit tests #1643
Assignees
Labels
Comments
warner
added a commit
that referenced
this issue
Aug 30, 2020
Using `rollup` to turn a tree of vat-defining source files into a bundle object is somewhat slow. It seems to depend on the amount of code being bundled: the worst case is the kernel sources, which take 1.5s . The kernel needs a number of other vats which are automatically bundled at runtime. The total process takes about 2.7s on my machine. We have unit tests that run many swingsets in a single test file. The most extreme case is test-message-patterns.js, which calls `buildVatController` 60 times in a row. It takes a really long time to run, 3-6 minutes, and the majority is the bundling step. Parallelizing this doesn't help (JS is a single-threaded language, it's not using non-blocking filesystem operations, I don't know whether it's CPU or IO bound but either would be a problem). This patch exports a new `buildKernelBundles()` function to let these multi-swingset tests build the bundles once per test file. It also adds `kernelBundles` to the `buildVatController(.., .., runtimeOptions)` options bag to pass these bundles back in (with sensible defaults). The controller gets all the built-in bundles from this collection, rather than running `bundleSource` each time. It also rearranges the console setup slightly, to make the `console` object available earlier within `buildVatController`. I needed this to debug problems with unpacking the other `runtimeOptions`. Also, now that we're building our own `console` object, we no longer need to explicitly harden the native console's (unique) prototype object. refs #1643
|
I implemented this in #1644 and enabled it for several swingset tests that build a lot of swingsets in a single test, so we've probably picked the low-hanging fruit on this one. Closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the Problem Being Solved?
Swingset-using unit tests take too long to run.
Several swingset unit tests, like
test-message-patterns.js, build many instances of the kernel (about 60) and run a small number of operations in each. The kernel needs several sounds bundles to do its job:kernel.js(the kernel source itself)To avoid a manual
yarn buildstep (which would cause really confusing errors if you forget to run it),buildVatController()function bundles these sources into bundle objects at runtime. But the bundling process (which usesrollup) is fairly slow, depending upon the amount of code being bundled: the kernel source takes about 1.5s to bundle on my machine. When we're building a lot of swingsets in a single test file, this times adds up.test-message-patterns.jstakes about 230s when run serially. By building all the bundles once and sharing them between calls tobuildVatController(), this can be reduced to about 15s.Description of the Design
buildVatController()already accepts aruntimeOptionsbag, but currently only acceptsdebugPrefixandverboseoptions. I'm going to add akernelBundlesoption, as well as a new exportedbuildKernelBundles()function. The default behavior will be forbuildVatControllerto bundle the sources itself, but tests can callbuildKernelBundles()and get back an object (clonable data) that can be passed instead.cc @FUDCo
The text was updated successfully, but these errors were encountered: