Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gc: inject bugs in GC to verify randomized test coverage. #98156

Closed
aliher1911 opened this issue Mar 7, 2023 · 2 comments
Closed

gc: inject bugs in GC to verify randomized test coverage. #98156

aliher1911 opened this issue Mar 7, 2023 · 2 comments
Assignees
Labels
branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 GA-blocker O-qa

Comments

@aliher1911
Copy link
Contributor

aliher1911 commented Mar 7, 2023

In #90830 GC got the ability to use clear range requests when removing multiple subsequent keys.

We rely on randomized tests to verify correctness outside of junits to improve test coverage.
To increase confidence in randomized tests, add faults to GC and ensure that randomized tests pick up discrepancies.

Jira issue: CRDB-25107

@aliher1911 aliher1911 added O-qa branch-master Failures and bugs on the master branch. GA-blocker T-kv-replication branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Mar 7, 2023
@aliher1911 aliher1911 self-assigned this Mar 7, 2023
@blathers-crl
Copy link

blathers-crl bot commented Mar 7, 2023

cc @cockroachdb/replication

@aliher1911
Copy link
Contributor Author

Tried:

  • injecting off by 1 in batch splits
  • removing head or tail of the points batch
  • adjusting gc key timestamps
  • distorting "isGarbage" checks

Randomized tests pick up failures all the time.

Found code branch in clear range case that was never/rarely engaged in randomized test.
Added new test config to ensure those cases are also tested.

craig bot pushed a commit that referenced this issue Apr 12, 2023
100396: workload/kv: additional parameters for workload generation r=kvoli a=aliher1911

This PR adds following features:

This commit adds --sfu-wait-delay option to configure delay between 'select for update' and upsert statements. Tidy up of RuntimeOnly flags to avoid fixture from exposing timeouts etc for data generations options.

Refactors validation and configuration of generator operations and extracts validated parameters to generator and sequence for later use.

Adds --key-size flag to allow creation of primary keys of arbitrary sizes. When key size is used, strings would be used as keys. Key is prefixed by uint64 zero padded string to maintain sort order and allow pre splitting ranges same way as int keys. The rest of the key is filled with random characters with generator seeded with numeric part of the key. That allows us to keep keys less compressible than constants and at the same time allow updates if key cycle is short.

Adds support for string key imports in fixtures to speed up data load.

Epic: none

Release note: None

101245: gc: add explicit cases for clear range in randomized tests r=erikgrinaker a=aliher1911

Previously there was very small chance that clear range requirements are met in randomized gc testing. This commit adds explicit test cases where clear range thresholds set specifically low to ensure test sees different ways of splitting requests into batches.

Release note: None

Touches #98156

101295: loqrecovery: deflake TestRetrieveApplyStatus r=erikgrinaker a=aliher1911

Test expects fully upreplicated 5 node cluster while TestCluster does not guarantee 5 replica config to propagate and upreplciate before proceeding. This fix adds explicit check during startup to wait appropriate cluster state.

Release note: None

Fixes #98623

Co-authored-by: Oleg Afanasyev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 GA-blocker O-qa
Projects
None yet
Development

No branches or pull requests

1 participant