investigate object leak in devnet AMM loadgen

[warner]

Our current codebase (a2900b4 aka 5507-ignore-vatcancelled, which is current trunk (102a5b6) plus some telemetry changes and a giant xsnap.js hack to ignore the vatCancelled Promise as a workaround for #5507), in conjunction with the AMM load generation code, appears to leak objects somewhere. Over time, the size of the kernel object table keeps growing.

We've seen this sort of thing before (#2013, at least), but there were enough other memory-consumption bugs that interfered with the analysis that we haven't really dived into it properly yet. One time, the problem turned out to be an XS promise resolution bug (which retained the resolution values of old promises). It could also be a bug in the AMM or Zoe (holding on to stale objects somehow), or in the way that the load generator is using those services. We're also seeing (#5507) a Promise.race/v8/ES bug in which old resolutions are retained as long as one of the input promises remains unsettled, which might be involved.

This ticket is to collect information and analysis results. We can close it when the current testnet AMM loadgen shows a flat kernel object table size.

[mhofman]

Possibly related: #3488

[warner]

This might be caused by #5671 , so we should circle back once we understand and/or fix the vat-bank issue.

[mhofman]

Have we confirmed there is an actual growth of the kernel object table vs the missing decrement fixed by #5652?

In my loadgen tests I am not seeing any object leak while performing both AMM & Vault operations, but I might not be exercising everything.

[warner]

I think you're right, the missing decrement was swamping our ability to see any kernel-wide object leaks. If your loadgen is showing flat object counts, then I think we can close this.

The instagoric load profile (in which dozens of clients are doing a lot of initialization work at the same time, and thus takes hours and hours to get to a steady state) is likely to produce more kernel objects until all clients are stable, which will look a lot like an object leak, but is actually just the warmup phase. And the one current known issue already has a ticket (#5671), which we'll investigate separately.