teamcity: failed test: TestRestoreReplicas

[cockroach-teamcity]

The following tests appear to have failed on master (testrace): TestRestoreReplicas

You may want to check for open issues.

#1460924:

TestRestoreReplicas
...886 util/stop/stopper.go:542  quiescing; tasks left:
2      mtc send
I190829 19:57:33.506401 104242 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190829 19:57:33.507115 104886 util/stop/stopper.go:542  quiescing; tasks left:
1      mtc send
I190829 19:57:33.509041 104623 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190829 19:57:33.511032 104815 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190829 19:57:33.512346 104409 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190829 19:57:33.513789 104595 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
W190829 19:57:33.517047 91810 internal/client/txn.go:524  [liveness-hb] failure aborting transaction: node unavailable; try another peer; abort caused by: result is ambiguous (error=failed to send RPC: store is stopped [exhausted])
I190829 19:57:33.517453 91810 storage/node_liveness.go:836  [liveness-hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (error=failed to send RPC: store is stopped [exhausted])
I190829 19:57:33.521054 104106 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
W190829 19:57:33.523166 91810 internal/client/txn.go:524  [liveness-hb] failure aborting transaction: node unavailable; try another peer; abort caused by: node unavailable; try another peer
W190829 19:57:33.524171 91810 storage/node_liveness.go:484  [liveness-hb] failed node liveness heartbeat: node unavailable; try another peer
I190829 19:57:33.529638 103968 storage/client_test.go:1359  [txn=79bbdafe] test clock advanced to: 171.000000313,0
I190829 19:57:33.538832 103968 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190829 19:57:33.548278 104885 util/stop/stopper.go:542  quiescing; tasks left:
4      rpc heartbeat
I190829 19:57:33.552500 104885 util/stop/stopper.go:542  quiescing; tasks left:
1      rpc heartbeat
W190829 19:57:33.553726 91604 storage/raft_transport.go:620  while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
    soon.go:35: condition failed to evaluate within 45s: node not live
        goroutine 91229 [running]:
        runtime/debug.Stack(0x62e81e0, 0xc002df0500, 0xc007ee5ca0)
        	/usr/local/go/src/runtime/debug/stack.go:24 +0xab
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x62e81e0, 0xc002df0500, 0xc007ee5ca0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:36 +0x87
        github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restartStore(0xc003086380, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:1055 +0x149
        github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restart(0xc003086380)
        	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:1108 +0xc1
        github.com/cockroachdb/cockroach/pkg/storage_test.TestRestoreReplicas(0xc002df0500)
        	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:373 +0x7ea
        testing.tRunner(0xc002df0500, 0x538f150)
        	/usr/local/go/src/testing/testing.go:865 +0x164
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:916 +0x65b

Please assign, take a look and update the issue accordingly.

[tbg]

#40351