Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tabledesc_test: deflake TestIndexDoesNotStorePrimaryKeyColumnMixedVersion #116791

Conversation

Xiang-Gu
Copy link
Contributor

@Xiang-Gu Xiang-Gu commented Dec 19, 2023

Previously, this test would flake under --stress and the root cause is because the upgrade precondition check that it relies on actually checks for corrupt descriptor at AOST '-10s', and it's possible that the corrupt descriptor was injected within the last 10s to the system, and thus the check produced a "false negative" (meaning it saw there isn't no corruption at AOST '-10s' and mistakenly assumed there isn't corruption now) and thus mistakenly allowed the cluster upgrade to proceed.

Fixes #116622
Release note: None

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@Xiang-Gu Xiang-Gu marked this pull request as ready for review December 19, 2023 20:25
@Xiang-Gu Xiang-Gu requested a review from a team as a code owner December 19, 2023 20:25
@Xiang-Gu Xiang-Gu force-pushed the deflake/TestIndexDoesNotStorePrimaryKeyColumnMixedVersion branch from 499641c to 6db9546 Compare December 19, 2023 22:10
@Xiang-Gu Xiang-Gu requested a review from rafiss December 19, 2023 22:14
@@ -115,6 +114,8 @@ func TestIndexDoesNotStorePrimaryKeyColumnMixedVersion(t *testing.T) {

// Assert cluster version upgrade is blocked.
require.Equal(t, [][]string{{"1000023.1"}}, tdb.QueryStr(t, "SHOW CLUSTER SETTING version;"))
// disable AOST for upgrade precondition check to ensure it sees the injected, corrupt descriptor.
upgrades.TestingSetFirstUpgradePreconditionAOST(false)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be:

defer upgrades.TestingSetFirstUpgradePreconditionAOST(false)()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch! Fixed

…sion

Previously, this test would flake under `--stress` and the root cause
is because the upgrade precondition check that it relies on actually
checks for corrupt descriptor at AOST '-10s', and it's possible that
the corrupt descriptor was injected within the last 10s to the system,
and thus the check produced a "false negative" (meaning it saw
there isn't no corruption at AOST '-10s' and mistakenly assumed there
isn't corruption now) and thus mistakenly allowed the cluster upgrade
to proceed.

The AOST '-10s' trick in the upgrade was implemented to help with
performance, and, for now, I think that outweighs the possibility of
producing such false-negatives, so this fix instead modified the test such
that it waits 10s after injecting the corrupt descriptor to ensure it
will be seen by the time the upgrade logic runs. The downside that comes
with this decision is this test will now take ~10-11s to run and we need
to skip it under `--stress` (bc 1000 iterations / # of parallelism * 10s
can still be larger than 1h, the default timeout for nightly stresses).

Release note: None
@Xiang-Gu Xiang-Gu force-pushed the deflake/TestIndexDoesNotStorePrimaryKeyColumnMixedVersion branch from 6db9546 to de78ad1 Compare December 19, 2023 23:30
@rafiss
Copy link
Collaborator

rafiss commented Dec 20, 2023

bors r+

@craig
Copy link
Contributor

craig bot commented Dec 20, 2023

Build succeeded:

@craig craig bot merged commit bab4160 into cockroachdb:master Dec 20, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sql/catalog/tabledesc: TestIndexDoesNotStorePrimaryKeyColumnMixedVersion failed
3 participants