point-in-time, interactive state examination at epoch N via progressive HTML

[raulk]

statediff already spiders down the state tree, dives into ADT data structures, and binds them to high-level objects. This allows us to go from a IPLD state tree, into a big typed struct that represents the state in a way that's inspectable (and diffable).

It uses a read-through CBORStore that falls back to RPC calls to fetch objects from a client.

The goal is to use these mechanics to enable point-in-time, interactive state examination at a given epoch.

The experience/flow we planned is:

1. Via CLI, call statediff providing an RPC endpoint and an epoch to examine.
2. statediff populates an initial, single-page, high-level HTML, with nodes that can be expanded.
3. As you expand those nodes, we progressively enhance in the HTML with the new content.

[raulk]

cc @momack2 @anorth @whyrusleeping @jbenet

[momack2]

Sounds super cool (and useful) to me!

The name "diff" implies to me that I can actually input two different epochs and compare where the state changed - is that right? If so, it'd be useful as well for documenting to the community what parts of state are being updated in a network upgrade.

[anorth]

Thanks! So just confirming some possible use cases:

• If I have a dump of a state tree in a CAR file, I can load that up (wrapping the CAR in a CBORStore) and output an HTML file, all headless and without requiring a running Lotus node or anything like that
• If I have two such CAR files, I can do the same to produce a navigable HTML diff

[raulk]

Yeah! We originally devised this tool to perform a three-way diff when a test vector fails, between the expected state, actual state, and initial state. See how it's used here: https://github.com/filecoin-project/lotus/blob/master/conformance/runner_test.go#L222-L232.

But the core of this tool is the recursion down the state tree and the interpretation of each node. So the goal of this issue is to repurpose/repackage that logic.

Just had a chat with @willscott. After digesting the great exchange of ideas, my understanding is that this tool needs to:

1. Provide a view of the entire state tree at a particular epoch, where the state tree is supplied by:

• providing a CAR with a single root —or—
• by pointing to a running Lotus node, and using the ChainReadObj RPC call to load objects from the state tree.

2. Provide a diff of the state trees between two epochs, where the state trees are supplied by:

• providing two CARs with a single root each, or one CAR with two roots —or—
• by pointing to a running Lotus node, just like above.

Both features should support two output modes:

• JSON, to use things like jq, dump into MongoDB, send to Elasticsearch, etc.
• HTML, which is what this issue is about.

In order to make the HTML performant and usable, we desire to:

• Embed the data and UI into a single-page HTML, with no dependencies, nor requiring Webpack or other JS bundlers. We can use Vue without a compiler, loading any libraries we need through a CDN.
• Generate it quickly.
  • According to @willscott, the loading and interpretation of a full state tree could take one or more minutes. So paying the cost of interpreting the entire state tree upfront may not lead to good UI.
  • Instead, we could embed the raw CAR (instead of the interpreted JSON) into the HTML, and interpret it by calling out to the Go interpreter via WASM.
  • We could maintain this WASM bundle and either host it in filecoin.io, or embed it in the HTML and load it as indicated here.
  • Alternatively, we could use the js libraries HAMT, but we'd have to reimplement the specs-actors ADT package (data structures), in JS, and maintain the mapping logic in two places (JS and Go), which I don't think will scale well.

@willscott does this capture the most salient points of our discussion?

[willscott]

Yep, agree with the write up of the design space.

• Not tied to vue. I suspect that we will run into babel / other needs for translation as soon as there's any heavy-lifting at the JS rendering level, that may mitigate the 'without complier' part. Not overly opposed to a js bundler if it can be run as part of go:generate in generating the binary and not at html-building / specific instance time. In other words, I'd be okay with the golang binary for statediff including a pre-compiled template.

• I'm a bit hesitant about wasm packaging. Downsides include

  • includes the golang runtime, so the wasm module will be at least a few megs
  • security boundary complexity, like issues loading from a file:// or localhost url.
  • I'm not convinced we end up with a single canonical state tree parser out of this anyway - i believe we already have a couple others in various places.

[raulk]

I suspect that we will run into babel / other needs for translation as soon as there's any heavy-lifting at the JS rendering level, that may mitigate the 'without complier' part.

If we plan to use things like Typescript, PostCSS, CSS-in-JS, fancy languages (e.g. JSX), etc. we will need to use a bundler and build tool. But if all we're using Vue for is to handle DOM changes and events (we don't need anything more than that), this will never be a shortcoming for us. Vue is designed to be imported at runtime as a library. You can use its compiler for fancier things, but we don't need them. It's very different to React and Svelte, which DO require using compilers.

includes the golang runtime, so the wasm module will be at least a few megs

If we serve it from filecoin.io, it will be cacheable and gzipped. It won't evolve very frequently, so the cache hit rate will be large.

security boundary complexity, like issues loading from a file:// or localhost url.

I don't think we will encounter this issue if we embed the CAR in the HTML, in a <script /> section.

I'm not convinced we end up with a single canonical state tree parser out of this anyway - i believe we already have a couple others in various places.

Which other state tree binding libraries exist?

I honestly don't see a way around using Go for this. There may be some js IPLD libs for foundational pieces, but they are not as nearly as tested as the Go counterparts. And the specs-actors specific data structures are not implemented, AFAIK. It's a huge risk, effort and maintenance burden to duplicate all this logic in JS.

[anorth]

This all sounds great, thanks.