kv: deflake TestFollowerReadsWithStaleDescriptor #108366
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes cockroachdb#108349. This commit deflakes TestFollowerReadsWithStaleDescriptor by disabling kvprober. As of 769ba1c, kvprober is now metaphorically enabled in all tests. When enabled, it can cause unexpected updates to the range cache, which trip up the test's assertion that a given query is not served on a newly added follower replica. The failure was easy to reproduce if I added a sleep after the non-voter addition. This change deflakes the test (even with this sleep) by disabling kvprober. Release note: None
nvanbenschoten
requested review from
rachitgsrivastava and
DarrylWong
and removed request for
a team
|
This change is |
andrewbaptist
requested changes
Aug 9, 2023
craig bot
pushed a commit
that referenced
this pull request
Aug 9, 2023
107355: cli: enhance diagnosing contention with redacted debug zips r=fqazi a=fqazi Previously a redacted zip, we would exclude retries and key information for contended keys since they could contain PII data. This patch does the following: - Adds a new builtin is_system_table_key which allows us to know if a key belongs to a system table - Modified redacted debug zips to include data for system keys in contention tables conditionally (if they belong to system tables) - Include retries and last_retry_reason information for queries in cluster insights to help diagnose contention Fixes: #104593 108454: Revert "kvprober: metamorphically enable / configure kvprober" r=andrewbaptist,joshimhoff a=nvanbenschoten This reverts (most of) commit 769ba1c. That commit metamorphically enabled kvprober. This has been observed to be destabliziing to unit tests. When the metamorphic constant is enabled (50% of the time) and when kvprober is fast enough, random ranges will see extra requests that they aren’t expecting. This adds nondeterminism which can trip up tests in any number of different ways. All of the following flakes have been tracked back to kvprober: Fixes #107864. Fixes #108242. Fixes #108441. Fixes #108349. Fixes #108124. Closes #108366. Release note: None Co-authored-by: Faizan Qazi <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]>
nvanbenschoten
deleted the
nvanbenschoten/deflakeTestFollowerReadsWithStaleDescriptor
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #108349.
This commit deflakes TestFollowerReadsWithStaleDescriptor by disabling kvprober. As of 769ba1c, kvprober is now metaphorically enabled in all tests. When enabled, it can cause unexpected updates to the range cache, which trip up the test's assertion that a given query is not served on a newly added follower replica.
The failure was easy to reproduce if I added a sleep after the non-voter addition. This change deflakes the test (even with this sleep) by disabling kvprober.
Release note: None