teamcity: failed test: TestRestoreReplicas

[cockroach-teamcity]

The following tests appear to have failed on master (testrace): TestRestoreReplicas

You may want to check for open issues.

#1457168:

TestRestoreReplicas
... 23:25:40.518295 118376 storage/store.go:3569  [s1,r1/1:/M{in-ax}] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W190827 23:25:40.519041 118374 storage/raft_transport.go:620  while processing outgoing Raft queue to node 2: store 2 was not found:
I190827 23:25:40.519247 118049 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190827 23:25:40.521912 118286 util/stop/stopper.go:542  quiescing; tasks left:
4      mtc send
1      [async] intent_resolver_ir_batcher
1      [async] intent_resolver_gc_batcher
I190827 23:25:40.522950 117878 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190827 23:25:40.527069 118286 util/stop/stopper.go:542  quiescing; tasks left:
1      mtc send
I190827 23:25:40.530013 117500 storage/client_test.go:1357  [txn=c3bc3c1a] test clock advanced to: 171.000000313,0
I190827 23:25:40.533486 117732 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190827 23:25:40.534257 117950 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190827 23:25:40.539522 118233 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190827 23:25:40.545167 117500 internal/client/txn.go:634  async rollback failed: failed to send RPC: sending to all 2 replicas failed; last error: <nil> failed to send RPC: store is stopped
I190827 23:25:40.549221 118377 internal/client/txn.go:634  async rollback failed: node unavailable; try another peer
I190827 23:25:40.546111 106161 storage/node_liveness.go:836  [liveness-hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (error=failed to send RPC: store is stopped [exhausted])
W190827 23:25:40.552780 106161 storage/node_liveness.go:484  [liveness-hb] failed node liveness heartbeat: context canceled
I190827 23:25:40.563399 118285 util/stop/stopper.go:542  quiescing; tasks left:
3      rpc heartbeat
I190827 23:25:40.570946 118285 util/stop/stopper.go:542  quiescing; tasks left:
2      rpc heartbeat
I190827 23:25:40.572715 118285 util/stop/stopper.go:542  quiescing; tasks left:
1      rpc heartbeat
W190827 23:25:40.574744 105711 storage/raft_transport.go:620  while processing outgoing Raft queue to node 1: rpc error: code = Canceled desc = grpc: the client connection is closing:
    soon.go:35: condition failed to evaluate within 45s: node not live
        goroutine 105684 [running]:
        runtime/debug.Stack(0x6223380, 0xc0025eaf00, 0xc00051c8a0)
        	/usr/local/go/src/runtime/debug/stack.go:24 +0xab
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x6223380, 0xc0025eaf00, 0xc00051c8a0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:36 +0x87
        github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restartStore(0xc001944380, 0x0)
        	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:1053 +0x149
        github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).restart(0xc001944380)
        	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:1106 +0xc1
        github.com/cockroachdb/cockroach/pkg/storage_test.TestRestoreReplicas(0xc0025eaf00)
        	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:373 +0x7ea
        testing.tRunner(0xc0025eaf00, 0x524dbf8)
        	/usr/local/go/src/testing/testing.go:865 +0x164
        created by testing.(*T).Run
        	/usr/local/go/src/testing/testing.go:916 +0x65b

Please assign, take a look and update the issue accordingly.

[tbg]

#40351