roachtest: admission-control/tpcc-olap/nodes=3/cpu=8/w=50/c=96 failed

[cockroach-teamcity]

roachtest.admission-control/tpcc-olap/nodes=3/cpu=8/w=50/c=96 failed with artifacts on master @ a226099dfee6ac0159c0c40a54aaa753d646cf6e:

test artifacts and logs in: /artifacts/admission-control/tpcc-olap/nodes=3/cpu=8/w=50/c=96/run_1
(admission_control_tpcc_overload.go:135).verifyNodeLiveness: failed to fetch liveness metrics: status: 503 Service Unavailable, content-type: application/json, body: {
  "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "code": 14,
  "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "details": [
  ]
}, error: <nil>

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=true , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-28138
Epic: CRDB-28893

[erikgrinaker]

All nodes were still running, but gRPC connections appeared to be failing on n1 and n2. The check itself is trying to fetch time series metrics via the gRPC JSON gateway for about 30 seconds:

cockroach/pkg/cmd/roachtest/tests/admission_control_tpcc_overload.go

Lines 124 to 136 in fc68b0f

	if err := retry.WithMaxAttempts(ctx, retry.Options{
	MaxBackoff: 500 * time.Millisecond,
	}, 60, func() (err error) {
	response, err = getMetrics(adminURLs[0], now.Add(-runDuration), now, []tsQuery{
	{
	name: "cr.node.liveness.heartbeatfailures",
	queryType: total,
	},
	})
	return err
	}); err != nil {
	t.Fatalf("failed to fetch liveness metrics: %v", err)
	}

cockroach/pkg/cmd/roachtest/tests/ts_util.go

Lines 102 to 103 in d26f0c3

	err := httputil.PostJSON(http.Client{Timeout: 500 * time.Millisecond}, url, &request, &response)
	return response, err

05:56:02 test_impl.go:344: test failure #1: full stack retained in failure_1.log: (admission_control_tpcc_overload.go:135).verifyNodeLiveness: failed to fetch liveness metrics: status: 503 Service Unavailable, content-type: application/json, body: {
  "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "code": 14,
  "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "details": [
  ]
}, error: <nil>

However, we're also seeing node readiness checks fail with the same error on n1 and n2:

teardown: 05:56:02 cluster.go:1346: checking for dead nodes
teardown: 05:56:03 cluster.go:1362: n3: err=<nil>,msg=11335
teardown: 05:56:03 cluster.go:1362: n2: err=<nil>,msg=11258
teardown: 05:56:03 cluster.go:1362: n1: err=<nil>,msg=11551
teardown: 05:56:03 cluster.go:1362: n4: err=<nil>,msg=skipped
teardown: 05:56:03 test_runner.go:1131: n1:/health?ready=1 status=503 body={
  "error": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "code": 14,
  "message": "connection error: desc = \"transport: error while dialing: gRPC connection unexpectedly re-dialed: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "details": [
  ]
}
teardown: 05:56:03 test_runner.go:1131: n2:/health?ready=1 status=503 body={
  "error": "connection error: desc = \"transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "code": 14,
  "message": "connection error: desc = \"transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)\"",
  "details": [
  ]
}
teardown: 05:56:03 test_runner.go:1139: n3:/health?ready=1 status=200 ok
teardown: 05:56:03 test_runner.go:1126: n4:/health?ready=1 error=Get "http://34.171.146.150:26258/health?ready=1": dial tcp 34.171.146.150:26258: connect: connection refused

@tbg The connection reuse errors seem suspect. I know you've been working in the RPC area lately, any of this seem related?

[tbg]

#99261 (comment)

I'll have to double check but I don't think I merged any of my changes related to this already, so this is looking like a pre-existing bug, at least not newly introduced over the last few weeks but possibly new in 23.1, we'll have to investigate.