Automated test framework can run scripts on launched clusters. Add offline stake operations test case and script.

[danpaul000]

Problem

• Cluster framework needs greater flexibility as we add more flavors of test cases.
• We don't have test/example scripts for the various staking operations, particularly the offline signing maneuvers.

Summary of Changes

1. Re-factor the automation framework to be able to run arbitrary scripts on a configurable testnet after the cluster has launched. (system-test/stake-operations-testcases/offline_stake_colo.yml is an example of a testcase that calls a CUSTOM_SCRIPT after successful cluster launch.)

2. Distinguish between testcases that should run a sleep time, partition looping logic, or a script

3. Create test script (system-test/stake-operations-testcases/offline_stake_operations.sh) that points at a running cluster and uses a nonce account and the offline signing workflow to do all staking operations. This can be run from the automation framework or manually without any environment dependencies, other than having solana and solana-keygen in your PATH.

4. Add a buildkite-readable testcase to wrap up the creation of a colo or GCE testnet and run the stake operations against it.

5. Add SKIP_PERF_RESULTS flag to buildkite testcases to skip printing TPS, confirmation time and slot rate for non-performance (ie staking ops) cases where we don't care about those values. Reducing output clutter.

Fixes #6194

[danpaul000]

@garious I bashed this problem pretty hard. This could go in as written as a functional sanity test, but I think it could serve external parties better if broken up into some smaller sample scripts, or turned into how-to documentation. wdyt?

[garious]

Nice! Got a way to ensure this doesn't bitrot? If it executes super fast, it'd be good to see this in CI. Otherwise, part of a nightly run.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

Pull request has been modified.

[danpaul000]

Depends on: #8780

Working this stake integration script into a nightly run as part of a larger test framework refactor.

[danpaul000]

Well this ballooned a little bit, but is in good shape now. FYI @mvines @t-nelson on summary point #1.

[t-nelson]

Lookin' good! Just a couple Qs

[t-nelson]

Are the config changes besides the TEST_TYPE additions intentional or debug artifacts?

[danpaul000]

Debug artifacts mostly. I just threw the "sanity-testcases" dir in there to run short tests that exercise the functionality of the testing framework, rather than the limits of the cluster. So I can point a buildkite job on a PR against this file rather than an expensive/long nightly-style testcase.