go/oasis-test-runner/oasis: Add a keymanager replication test

[Yawning]

• Add a basic replication test.
  • Spin up a replica.
  • Explicitly query the replica or something.
• Fix bugs.
  • Oh my god, the key manager worker host doesn't support enclave rpc.
  • Key manager client should support km->km.
  • Fix the key manager grpc access control.

[codecov]

Codecov Report

Merging #2843 into master will decrease coverage by 0.02%.
The diff coverage is 66.50%.

@@            Coverage Diff             @@
##           master    #2843      +/-   ##
==========================================
- Coverage   67.60%   67.57%   -0.03%     
==========================================
  Files         349      350       +1     
  Lines       33796    33960     +164     
==========================================
+ Hits        22849    22950     +101     
- Misses       8009     8051      +42     
- Partials     2938     2959      +21     

Impacted Files	Coverage Δ
go/keymanager/api/grpc.go	50.98% <39.47%> (-33.64%)	⬇️
go/worker/keymanager/init.go	71.42% <58.33%> (-1.46%)	⬇️
go/keymanager/client/client.go	82.56% <62.50%> (+7.56%)	⬆️
go/worker/keymanager/watcher.go	65.71% <65.71%> (ø)
go/worker/keymanager/handler.go	67.92% <71.42%> (+6.38%)	⬆️
go/worker/keymanager/worker.go	64.58% <95.65%> (+6.07%)	⬆️
go/consensus/tendermint/keymanager/keymanager.go	86.74% <100.00%> (ø)
go/keymanager/api/api.go	70.00% <100.00%> (+1.03%)	⬆️
go/oasis-node/cmd/node/node.go	54.06% <100.00%> (+0.11%)	⬆️
go/worker/common/runtime_host.go	66.18% <100.00%> (+0.74%)	⬆️
... and 37 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e83231b...5378569. Read the comment docs.

[Yawning]

Well, replication works. The rust side simple encryption test client will panic when the query goes to the replica, but that's an orthogonal problem. Either this doesn't retry, or the retry logic is busted.

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RpcFailure(RpcStatus { status: Unknown, details: Some("worker/keymanager: error from runtime: decrypt error") })', tests/clients/simple-keyvalue-enc/src/main.rs:99:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[kostko]

The rust side simple encryption test client will panic when the query goes to the replica, but that's an orthogonal problem. Either this doesn't retry, or the retry logic is busted

Rust part of the retry mechanism should work and is also covered in unit tests. Currently the Rust client will only retry three times though so depending on the situation this may not be enough. It could also be a problem with the round robin node selection policy.

Fixt