[meta] Plots: improve infrastructure for analyzing perf metrics

[wwwillchen]

Building off the work in #1936, I'll work on the following tasks:

Improve quality

• Run this on a compute instance in Asia to test the hypothesis that the high variance for international sites is driven by network latency
• Testing effect of launching (or not) a new Chrome for every test (Plots: testing effect of launching (or not) a new Chrome for every test. #1982)

Improve visualizations

• Create a way to view archived results with a URL (e.g. github pages)
• Enable comparisons over different versions of Lighthouse (allows you to easily see how a commit will change the metrics on real-world sites)

Plots as CLI

• Create flags to configure which sites to analyze
• Create a --out flag to customize output path

Other

• Create a suite of tests to ensure /plots/ doesn't break
• Add scores to Metrics (https://github.com/GoogleChrome/lighthouse/blob/master/lighthouse-core/lib/traces/pwmetrics-events.js)
• Make plotly a devDependency within /plots/ so it can be run locally (currently plotly is fetched from CDN)
• Integrate with Travis CI to tag a PR with a before-and-after analysis (optional?)

[slohmes]

Hello, I'm excited to see this feature grow!

Following the link in the Lighthouse plots readme (https://googlechrome.github.io/lighthouse/plots/) gives me a 404. I don't see this reported in any open issues - is it new, or expected? I'm happy to either open an issue to remove the link from the documentation, if the page is not currently supported, or to open an issue to fix the plots page, if it should be working (or maybe that should be included in this issue).

Does anyone have clarity around the expected status of the plots page?

[paulirish]

@slohmes i just fixed the 404: https://googlechrome.github.io/lighthouse/plots/

we really like plots too, but we haven't been improving it much recently. would be happy for you to explore it and see if it works for you.

[brendankenny]

no more plots 😢