roachtest: c2c/shutdown/src/coordinator failed

[cockroach-teamcity]

roachtest.c2c/shutdown/src/coordinator failed with artifacts on master @ 76da6c7dfb4d71ad614715ec5e230a6cc76fbd0e:

test c2c/shutdown/src/coordinator failed during post test assertions (see test-post-assertions.log) due to (test_runner.go:1122).func1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Cockroach CA")
test artifacts and logs in: /artifacts/c2c/shutdown/src/coordinator/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-28911

[cockroach-teamcity]

roachtest.c2c/shutdown/src/coordinator failed with artifacts on master @ 1eb628e8c8a7eb1cbf9bfa2bd6c31982c25cbba0:

test c2c/shutdown/src/coordinator failed during post test assertions (see test-post-assertions.log) due to (test_runner.go:1122).func1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Cockroach CA")
test artifacts and logs in: /artifacts/c2c/shutdown/src/coordinator/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[lidorcarmel]

bisected to 3573201 which is pr #105006
I think I trust my bisection (meaning, I don't think it's flaky but I didn't really run this frequently)
cc @herkolategan somehow this pr messed up certificates?

[lidorcarmel]

that said, other c2c tests are succeeding! still looking.

[herkolategan]

@lidorcarmel I'm going to look into this, because the test that PR, #105006, re-enabled does some certificate manipulation. Including changing permissions etc. It could be the culprit.

[blathers-crl]

cc @cockroachdb/test-eng

[lidorcarmel]

@herkolategan I might have got this one wrong :( It doesn't fail anymore so maybe it is flaky? and if I was bisecting using a flaky test then that's.. garbage. Anyway I tried 3 runs now with #105006 and all 3 succeeded. Hopefully you didn't waste time on this! I'll look again tomorrow.

[renatolabs]

Pretty sure the failure is due to a bug in the certificate management by the test infrastructure, as evidenced by the error message in the logs:

grpc: ‹connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Cockroach CA\")"› [code 14/Unavailable]

It was recently fixed by #105514, so I believe that is why you don't see the failure anymore. I'm going to close this issue, roachtest will create another one if it fails again.

[herkolategan]

@herkolategan I might have got this one wrong :( It doesn't fail anymore so maybe it is flaky? and if I was bisecting using a flaky test then that's.. garbage. Anyway I tried 3 runs now with #105006 and all 3 succeeded. Hopefully you didn't waste time on this! I'll look again tomorrow.

@lidorcarmel No problem, thanks for the heads-up. I spotted the same issue @renatolabs mentioned and thought it was the cause (I made a note on that issue #105514 that it's likely the cause of this one). Hence that should fix this even if it was flaky.