go/oasis-test-runner: configurable sanitychecker

[ptrus]

Daily longterm tests have been failing since validator-0 was falling behind the rest of the network due to the supplementary-sanity checker. This moves the supplementary-sanity checker to a new (client-1) node, which can fall behind without affecting rest of the network.

Also increase the timeout between governance workload iterations, as currently the amount of proposals was quickly pilling up over time (and both queries-workload and sanity-checker, go through all (even expired) proposal and votes).

TODO:

• run the longterm job to ensure the CI instances can handle the test suite with the additional node: https://buildkite.com/oasisprotocol/oasis-core-long-tests/builds/543

[codecov]

Codecov Report

Merging #3719 (fcbcc97) into master (4956fbd) will increase coverage by 0.10%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3719      +/-   ##
==========================================
+ Coverage   66.90%   67.01%   +0.10%     
==========================================
  Files         401      401              
  Lines       39855    39855              
==========================================
+ Hits        26665    26708      +43     
+ Misses       9412     9372      -40     
+ Partials     3778     3775       -3     

Impacted Files	Coverage Δ
go/oasis-node/cmd/common/pprof/pprof.go	66.66% <100.00%> (+46.29%)	⬆️
go/consensus/tendermint/apps/governance/query.go	83.33% <0.00%> (-16.67%)	⬇️
go/worker/common/committee/runtime_host.go	64.28% <0.00%> (-6.25%)	⬇️
go/consensus/api/submission.go	69.01% <0.00%> (-5.64%)	⬇️
go/storage/api/root_cache.go	71.11% <0.00%> (-4.45%)	⬇️
go/storage/database/database.go	73.39% <0.00%> (-1.84%)	⬇️
go/consensus/tendermint/governance/governance.go	58.67% <0.00%> (-1.66%)	⬇️
go/consensus/tendermint/abci/mux.go	58.60% <0.00%> (-1.55%)	⬇️
go/consensus/tendermint/apps/staking/fees.go	46.37% <0.00%> (-1.45%)	⬇️
...onsensus/tendermint/apps/governance/state/state.go	71.83% <0.00%> (-1.41%)	⬇️
... and 26 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4956fbd...fcbcc97. Read the comment docs.

[ptrus]

@kostko This is ready for review. The last 2 failed runs were due to runtime staking messages being resubmitted (related to compute node being restarted at an unlucky time), causing the operation to re-apply (since the runtime doesn't use nonces).

I think we should tackle those in a subsequent PR as this one got big enough and it's kinda hard to keep testing the changes here while the daily tests are also being run. This PR however should fix the permanent test failures that are currently happening on the daily tests runs.

[kostko]

Nice work, thanks!

[kostko]

Would be nice if we could unify/cleanup this as it is the same for every Node, but this is also true for the Node constructor above so probably best for a separate issue/PR.

[ptrus]

👍 I'll open an issue