go/e2e/txsource: periodically restart runtime nodes

[ptrus]

TODO:

• setup 2 keymanagers for the long-running test and also enable restarts there
• setup 4 compute nodes
• keep one of each type of nodes always running (so we can observe the memory/cpu/disk usage of that node, compared to the others)
• revert the scripts/daily_txsource.sh change before merging

Fixes:

• .buildkite/longtests.pipeline.yml missing new artifact: extracted into: ci: extract go artifacts upload code into a script #3125
• go/consensus/tendermint/roothash: ignore non tracked runtimes when reindexing

[codecov]

Codecov Report

Merging #3124 into master will decrease coverage by 0.18%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #3124      +/-   ##
==========================================
- Coverage   68.72%   68.53%   -0.19%     
==========================================
  Files         374      374              
  Lines       36990    37000      +10     
==========================================
- Hits        25421    25358      -63     
- Misses       8331     8396      +65     
- Partials     3238     3246       +8     

Impacted Files	Coverage Δ
go/consensus/tendermint/roothash/roothash.go	73.29% <0.00%> (-0.69%)	⬇️
go/consensus/api/transaction/results/results.go	0.00% <0.00%> (-100.00%)	⬇️
go/oasis-node/cmd/common/metrics/disk.go	65.51% <0.00%> (-20.69%)	⬇️
go/consensus/tendermint/api/errors.go	86.66% <0.00%> (-13.34%)	⬇️
go/worker/compute/executor/committee/state.go	74.07% <0.00%> (-11.12%)	⬇️
go/runtime/host/sandbox/sandbox.go	66.54% <0.00%> (-10.79%)	⬇️
go/oasis-node/cmd/common/metrics/resource.go	84.00% <0.00%> (-8.00%)	⬇️
go/worker/storage/service_external.go	47.31% <0.00%> (-6.46%)	⬇️
...n/crypto/signature/signers/memory/memory_signer.go	71.42% <0.00%> (-4.77%)	⬇️
go/worker/common/committee/runtime_host.go	65.71% <0.00%> (-4.77%)	⬇️
... and 27 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02a2481...f26c4bb. Read the comment docs.

[kostko]

Nice catch!

[ptrus]

hm got a: failed to run scenario: log watcher compute-3/log: log assertion failed: timeout detected in the short-txsource test on CI: https://buildkite.com/oasisprotocol/oasis-core-ci/builds/918#2cdbef61-4395-4526-894a-bc040620f727 (where we don't do any restarts, so only change are few more nodes). Maybe would need beefier instance for the test due to increased number of nodes.

The reason seems to be that one of the merge workers were behind a bit, will merge this (since it only happened once on 10+ runs) and re-address if it occurs more.

[kostko]

Based on a quick look at the logs, the timeout has to do with one of the storage nodes returning "permission denied" for some updates. Need to investigate this more.