roachtest: schemachange/mixed-versions failed ["show cluster setting version" timed out: value differs between local setting and KV]

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-24.2 @ 310388de99f5b0b708a6b286581d79bb4df6f34d:

(mixedversion.go:710).Run: mixed-version test failure while running step 12 (run "run schemachange workload and validation in mixed version"): full command output in run_072406.811148213_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed #129458 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.2 release-blocker]
• roachtest: schemachange/mixed-versions failed [index contains stored column "" with unknown ID 0] #129395 roachtest: schemachange/mixed-versions failed [index contains stored column "" with unknown ID 0] [C-test-failure O-roachtest O-robot T-sql-foundations branch-master release-blocker]
• roachtest: schemachange/mixed-versions failed #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]
• roachtest: schemachange/mixed-versions failed [pg_lsn in mixed version] #129104 roachtest: schemachange/mixed-versions failed [pg_lsn in mixed version] [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.2.10-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41550

[rafiss]

E240822 07:26:12.364841 1 workload/cli/run.go:591  [-] 3  ***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: operation "show cluster setting version" timed out after 2m0.001s (given timeout 2m0s): value differs between local setting (24.2-upgrading-step--01) and KV (24.1-upgrading-to-24.2-step-012) (SQLSTATE XXUUU)
Error: ***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: operation "show cluster setting version" timed out after 2m0.001s (given timeout 2m0s): value differs between local setting (24.2-upgrading-step--01) and KV (24.1-upgrading-to-24.2-step-012) (SQLSTATE XXUUU)

[rafiss]

The error value differs between local setting (24.2-upgrading-step--01) and KV (24.1-upgrading-to-24.2-step-012) indicates that the local setting is set to a fence version, as evidenced by the odd internal version number. The strange thing is that it looks like the internal version is -1. That's why it got formatted with the extra dash (step--01).

I read some code and found that that's possible because of the FenceVersionFor logic:

cockroach/pkg/upgrade/helpers.go

Lines 24 to 48 in 614ce3e

	// FenceVersionFor constructs the appropriate "fence version" for the given
	// cluster version. Fence versions allow the upgrades infrastructure to safely
	// step through consecutive cluster versions in the presence of Nodes (running
	// any binary version) being added to the cluster. See the upgrade manager
	// above for intended usage.
	//
	// Fence versions (and the upgrades infrastructure entirely) were introduced
	// in the 21.1 release cycle. In the same release cycle, we introduced the
	// invariant that new user-defined versions (users being crdb engineers) must
	// always have even-numbered Internal versions, thus reserving the odd numbers
	// to slot in fence versions for each cluster version. See top-level
	// documentation in pkg/clusterversion for more details.
	func FenceVersionFor(
	ctx context.Context, cv clusterversion.ClusterVersion,
	) clusterversion.ClusterVersion {
	if (cv.Internal % 2) != 0 {
	log.Fatalf(ctx, "only even numbered internal versions allowed, found %s", cv.Version)
	}
	// We'll pick the odd internal version preceding the cluster version,
	// slotting ourselves right before it.
	fenceCV := cv
	fenceCV.Internal--
	return fenceCV
	}

When the internal version is 0, which it would be for all final versions, the corresponding fence version will have an internal version of -1.

The change in #99967 was meant to allow the local and KV version to be off by one fence version. But the logic in checkClusterSettingValuesAreEquivalent does not account for the -1 internal version described above.

This should be fixable with a change to checkClusterSettingValuesAreEquivalent. ccing @RaduBerinde since you've been interacting with the cluster version code a lot. Does my reasoning make sense to you? If so, my proposed fix is here: #129520

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-24.2 @ a956ba3b4efcb80ffe9a2ca7b818094975285942:

(mixedversion.go:710).Run: mixed-version test failure while running step 13 (run "run schemachange workload and validation in mixed version"): full command output in run_070757.449659469_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed #129535 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.2.1-rc release-blocker]
• roachtest: schemachange/mixed-versions failed [index contains stored column "" with unknown ID 0] #129395 roachtest: schemachange/mixed-versions failed [index contains stored column "" with unknown ID 0] [C-test-failure O-roachtest O-robot T-sql-foundations branch-master release-blocker]
• roachtest: schemachange/mixed-versions failed #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]
• roachtest: schemachange/mixed-versions failed [pg_lsn in mixed version] #129104 roachtest: schemachange/mixed-versions failed [pg_lsn in mixed version] [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.2.10-rc]

This test on roachdash | Improve this report!

[blathers-crl]

Based on the specified backports for linked PR #129520, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}