roachtest: acceptance/version-upgrade failed

[cockroach-teamcity]

roachtest.acceptance/version-upgrade failed with artifacts on master @ bb5c97b5d2073a27e7ec88654b05ac913112f610:

test artifacts and logs in: /artifacts/acceptance/version-upgrade/run_1
(mixedversion.go:366).Run: mixed-version test failure while running step 29 (run "test schema change step"): output in run_052958.923353501_n1_workload-run-schemac: ./workload run schemachange {pgurl:1-4} --concurrency 2 --max-ops 10 --verbose 1 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_052959.759759262_n1_workload-run-schemac.log: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-26117

[renatolabs]

Same familiar error:

"message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: operation \"show cluster setting version\" timed out after 2m0.001s (given timeout 2m0s): value differs between local setting ([18 8 8 22 16 2 24 0 32 11]) and KV ([18 8 8 22 16 2 24 0 32 10]); try again later (\u003cnil\u003e after 1m59.141621974s) (SQLSTATE XXUUU)",

(and once again, no migrations around the time of failure are actually taking more than 2 minutes).

@fqazi @ajwerner was #99665 expected to stop this kind of failure from happening?

[ajwerner]

That one is interesting!

[ajwerner]

No it has nothing to do with that failure mode. The deal here is that somewhere in the cluster setting land we're getting divergence between the in-memory value and the KV value.

[ajwerner]

To be more concrete, the way the code here works is that it reads the value from KV and checks the value in memory. They do not move in lock-step (the in-memory value moves first). If the in-memory version does not match the kv version, we retry. We keep retrying because of different mismatches. I think we expect mis-matches nowadays while any migrations are on-going.

I don't know quite what the right thing to do here is. Maybe we want to return an in-memory value if we've seen the KV value ever move past it?

[renatolabs]

No it has nothing to do with that failure mode.

So you're saying that #99590 (fixed by #99665) and this issue are unrelated and just happen to have the same error message as outcome? Can you explain how one could have identified the distinction?

I think we expect mis-matches nowadays while any migrations are on-going.

What do you mean by nowadays? The behavior you described (which seems to be the logic in show_cluster_setting.go) is basically unchanged since at least 22.2. I've looked through a few issues where we got this same error message and it seems that it generally pointed towards some other problem that needed fixing.

[ajwerner]

I guess the truth is that they both failed because the migrations too more than 2 minutes. The difference is that here they were making progress, but were sort of slow and there they weren't making progress at all. I only fixed the proximate issue that was causing the migration to be hung. You're right that in effect the issue is unresolved.

[ajwerner]

I wonder if what we should do is note when the version in memory is the fence version successor to the version in KV and then just return the KV version.

[ajwerner]

@renatolabs this is what I have in mind: #99967

[ajwerner]

I can't get this one to fail on master, at least after 20 runs.

[ajwerner]

I wonder if some of the fixes merged by Raphael recently changed things.

[renatolabs]

Closing this one as it's very likely fixed by #99967. We'll take another look if it fails again.