roachtest: tpccbench/nodes=3/cpu=16 failed

[cockroach-teamcity]

roachtest.tpccbench/nodes=3/cpu=16 failed with artifacts on master @ a44a9b1ffce25f51026b494a1dcb393cfc5361f3:

(monitor.go:149).Wait: monitor failure: full command output in run_111030.156103855_n1_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-44184

[DarrylWong]

Import failed with:

W241108 11:15:12.123814 1 ccl/workloadccl/fixture.go:577  [-] 6  error enabling automatic stats: pq: update-setting: result is ambiguous: error=ba: Put [/Table/6/1/"sql.stats.automatic_collection.enabled"/0], EndTxn(parallel commit) [/Table/6/1/"sql.stats.automatic_collection.enabled"/0], [txn: f973c1c1], [protect-ambiguous-replay] RPC error: grpc: error reading from server: EOF [code 14/Unavailable] [exhausted] (last error: failed to send RPC: sending to all replicas failed; last error: ba: Put [/Table/6/1/"sql.stats.automatic_collection.enabled"/0], EndTxn(parallel commit) [/Table/6/1/"sql.stats.automatic_collection.enabled"/0], [txn: f973c1c1], [protect-ambiguous-replay] RPC error: grpc: error reading from server: EOF [code 14/Unavailable])
Error: importing fixture: importing table customer: pq: pausing due to error; use RESUME JOB to try to proceed once the issue is resolved, or CANCEL JOB to rollback: exhausted retries: grpc: error reading from server: EOF [code 14/Unavailable]
run_111030.156103855_n1_cockroach-workload-f: 2024/11/08 11:15:12 cluster.go:2489: > result: COMMAND_PROBLEM: exit status 1

Node 1 was having trouble staying connected, logs are full of:

W241108 11:15:39.376260 599 kv/kvserver/liveness/liveness.go:667 â‹® [T1,Vsystem,n1,liveness-hb] 1573 +An inability to maintain liveness will prevent a node from participating in a
W241108 11:15:39.376260 599 kv/kvserver/liveness/liveness.go:667 â‹® [T1,Vsystem,n1,liveness-hb] 1573 +cluster. If this problem persists, it may be a sign of resource starvation or
W241108 11:15:39.376260 599 kv/kvserver/liveness/liveness.go:667 â‹® [T1,Vsystem,n1,liveness-hb] 1573 +of network connectivity problems. For help troubleshooting, visit:

Maybe just an infra flake but I'll send over to kv to take a closer look.

[srosenberg]

It does look like network just dropped out on n1.

ip	node
10.1.0.10	n1
10.1.0.101	n2
10.1.0.100	n3

From n2,

E241108 11:12:09.957964 861 2@rpc/peer.go:642 ⋮ [T1,Vsystem,n2,rnode=1,raddr=‹10.1.0.10:26257›,class=raft,rpc] 269  disconnected (was healthy for 1.001s): grpc: ‹rpc error: code = Unavailable desc = error reading from server: EOF› [code 14/Unavailable]
W241108 11:12:09.957968 15966 kv/kvserver/raft_transport.go:1067 ⋮ [T1,Vsystem,n2] 270  while processing outgoing Raft queue to node 1: recv msg error: grpc: ‹grpc: the client connection is closing› [code 1/Canceled]:
W241108 11:12:12.638239 955 kv/kvserver/raft_transport.go:1067 ⋮ [T1,Vsystem,n2] 271  while processing outgoing Raft queue to node 1: recv msg error: grpc: ‹error reading from server: read tcp 10.1.0.101:37850->10.1.0.10:26257: read: connection reset by peer› [code 14/Unavailable]:

From n1,

W241108 11:12:08.684625 23933 kv/kvserver/closedts/sidetransport/receiver.go:135 ⋮ [n1,remote=2] 409  closed timestamps side-transport connection dropped from node: 2 (grpc: ‹context canceled› [code 1/Canceled])
W241108 11:12:08.875542 254 2@gossip/gossip.go:1467 â‹® [T1,Vsystem,n1] 410  first range unavailable; trying remaining addresses
I241108 11:12:08.875683 24091 gossip/client.go:112 ⋮ [T1,Vsystem,n1] 411  unvalidated bootstrap gossip dial to ‹10.1.0.100:26257›
I241108 11:12:08.887057 24091 gossip/client.go:136 ⋮ [T1,Vsystem,n1] 412  started gossip client to n0 (‹10.1.0.100:26257›)
I241108 11:12:08.934968 24197 rpc/heartbeat.go:169 â‹® [-] 413  failing ping request from node n2
E241108 11:12:09.067784 2706 2@rpc/peer.go:642 ⋮ [T1,Vsystem,n1,rnode=2,raddr=‹10.1.0.101:26257›,class=raft,rpc] 414  disconnected (was healthy for 2m0.694s): grpc: ‹error reading from server: EOF› [code 14/Unavailable]

[srosenberg]

Also, we're hitting AmbiguousReplayProtection [1].

[1] #107658