roachtest: rebalance/by-load/leases/mixed-version failed

[cockroach-teamcity]

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 911f1ce389342459592d89fe7b78bcba3a2d265a:

(mixedversion.go:592).Run: mixed-version test failure while running step 9 (run "rebalance load run"): CPU not evenly balanced after timeout: within bounds mean=69.0 tolerance=20.0% (±13.8) bounds=[55.2, 82.8]
	stores=[s1: 66 (-4.1%), s2: 78 (+14.4%), s3: 61 (-10.3%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-37692

[kvoli]

mean=69.0 tolerance=20.0% (±13.8) bounds=[55.2, 82.8]
stores=[s1: 66 (-4.1%), s2: 78 (+14.4%), s3: 61 (-10.3%)]

It appears all stores are within the bounds, might be a testing issue.

[kvoli]

The test is unable to gather the CPU usage information, causing the test to timeout before balancing is achieved (or noticed to be achieved).

06:34:15 cluster.go:2429: running cmd `./cockroach auth-session lo...` on nodes [:4]; details in run_063415.317963092_n4_cockroach-authsessio.log
06:34:15 httpclient.go:210: ./cockroach auth session login failed on node 4: failed to authenticate: COMMAND_PROBLEM: exit status 1

This may be an issue with mixed version secure clusters and the roachtest framework?

run_063413.087691684_n1_cockroach-authsessio: 06:34:13 cluster.go:2432: > ./cockroach auth-session login root --port={pgport:1} --certs-dir ./certs --format raw
teamcity-14772140-1712728228-03-n4cpu4:[1]: ./cockroach auth-session lo...
ERROR: connection lost.

EOF
Failed running "auth-session login"
run_063413.087691684_n1_cockroach-authsessio: 06:34:13 cluster.go:2457: > result: Error for Node 1: COMMAND_PROBLEM: exit status 1
(1) Node 1. Command with error:
  | ```
  | ./cockroach auth-session login root --port={pgport:1} --certs-dir ./certs --format raw
  | ```
  | stdout: <empty>
  | stderr:ERROR: connection lost.
  |
  | EOF
  | Failed running "auth-session login"
Wraps: (2) COMMAND_PROBLEM
Wraps: (3) exit status 1
Error types: (1) *hintdetail.withDetail (2) errors.Cmd (3) *exec.ExitError

[kvoli]

This is a similar issue to #119064. The fix there was to run the test in insecure mode. Should we do the same here @DarrylWong?

[DarrylWong]

👍 That sounds reasonable to me. A little surprised it took this long to fail with this though.