jepsen: make the Jepsen tests first-class citizens

[knz]

Make the Jepsen tests a first-class citizen

• create a dashboard linked/embedded in the main web site that displays the status from the latest tests with the date/time of the last test run jepsen: Dashboard #19695
• deploy separate Jepsen testing for each released CockroachDB version concurrently

Better automate the tests

• make the tests use pre-allocated VMs instead of terraforming on each run.
• upload the perf statistics gathered by Jepsen to a separate repo so we can see how a test performance changes over time. jepsen: Track performance over time #19696
• use different running times for different tests and different nemesis combinations. jepsen: More nemeses, flexible configuration #19697

Ease the troubleshooting process

• make it simpler to re-launch a failed test using the exact same configuration (e.g. a single link to download a script that does the same thing).

Fix bugs/limitations in the test framework

• make nemeses and test initialization less flaky by using cockroach quit more, use signals less, take inspiration from the CLI interactive tests.
• jepsen: failure due to unexpected retry error in Sets test jepsen: failure due to unexpected retry error in Sets test #15733
• jepsen: the Register test is improperly configured jepsen: the Register test is improperly configured #15734
• jepsen: the start-stop nemesis is flaky/incorrect jepsen: the start-stop nemesis is flaky/incorrect #15736
• jepsen: the Sequential test is probably incorrect jepsen: the Sequential test is probably incorrect #15737
• triage remaining issues in nightlies: Jepsen tests intermittently failing #14171
• jepsen: (re-)enable the big-skews nemesis jepsen: (re-)enable the big-skew nemeses #15893
• enable the split nemesis (and maybe others?) jepsen: More nemeses, flexible configuration #19697

Extend the test suite

• tests: possible linearizability anomaly when running linearizable tests: jepsen-test linearizabile monotonic timestamps #5517
• add testing using a nemesis that introduces data corruption or disk errors storage: Test behavior when disk fills up #19656
• run for longer than 3m per test (jepsen: Bank test failed #17815 needs several hours to reproduce)

[knz]

cc @bdarnell @spencerkimball

[bdarnell]

jepsen-io/jepsen#178 fixes the register tests and reduces some other flakiness. The sequential tests are still failing 100% of the time, and the most common failure in the other tests is #16450.

[bdarnell]

The jepsen tests are in pretty good shape these days. I think that aside from the automatic filing of issues (#19694), they're already "first-class citizens". I've added separate issues for most of the remaining points here, so I'm closing this meta-issue. Feel free to open new issues for the points abandoned here if you think they're worth keeping.