How do we visualize and analyze results?

[drmorr0]

Right now the only way to "look at" the results from a simulation are through Prometheus/Grafana (or some other observability stack that you have installed in your simulation cluster). This is not really sufficient for doing data analysis or comparisons between simulations. How can we do this better?

Questions we'd like to be able to answer:

1. Compare/visualize a "diff" between two simulation runs (or between a simulated run and a "real" run)
2. Compute some set of (possibly user-defined?) metrics across one or more simulations
3. Perform ad-hoc queries of the data/logs/etc to identify "what went wrong" or "why did this do that"

Some ideas:

1. Export to some format that can be studied inside Jupyter notebooks? There's no easy way to, say, export from Prometheus to Jupyter, so we'd have to do this all on our own
2. Same as (1), except maybe using something like d3.js?
3. Write some kind of Grafana plugin(s)?
4. ....?

[drmorr0]

We ended up saving to parquet files that can be analyzed later using, e.g, https://github.com/acrlabs/datakube