Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asim: introduce rand testing framework with default-only settings #107255

Merged
merged 1 commit into from
Jul 26, 2023

Conversation

wenyihu6
Copy link
Contributor

@wenyihu6 wenyihu6 commented Jul 20, 2023

This patch lays the backbone of the randomized testing framework. Currently, it
only supports default configuration for all options, implying that there is no
randomization yet. Additionally, it refactors some of the existing structure in
data_driven_test. Note that this should not change any existing behavior, and
the main purpose is to make future commits cleaner.

TestRandomized is a randomized testing framework designed to validate
allocator by creating randomized configurations, generating corresponding
allocator simulations, and validating assertions on the final state.

Input of the framework (fields in the testSetting struct):

  1. numIterations (int, default: 3): specifies number of test iterations to be
    run, each with different random configurations generated
  2. duration (time.Duration, default: 30min): defined simulated duration of
    each iteration verbose (bool, default: false): enables detailed simulation
    information failing output
  3. randSeed (int64, default: 42): sets seed value for random number
    generation
  4. assertions ([]SimulationAssertion, default: conformanceAssertion with 0
    under-replication, 0 over-replication, 0 violating, and 0 unavailable):
    defines criteria for validation assertions
  5. randOptions: guides the aspect of the test configuration that should be
    randomized. This includes:
  • cluster (bool): indicates if the cluster configuration should be randomized
  • ranges (bool): indicates if the range configuration should be randomized
  • load (bool): indicates if the workload configuration should be randomized
  • staticSettings (bool): indicates if the simulation static settings should
    be randomized
  • staticEvents (bool): indicates if static events, including any delayed
    events to be applied during the simulation, should be randomized

RandTestingFramework is initialized with a specified testSetting and
maintained its state across all iterations. Each iteration in
RandTestingFramework executes the following steps:

  1. Generates a random configuration based on whether the aspect of the test
    configuration is set to be randomized in randOptions
  2. Executes a simulation and store any assertion failures in a buffer

Part of: #106311
Release note: None

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@wenyihu6 wenyihu6 force-pushed the rand-framework branch 12 times, most recently from 1982e47 to 4806d6e Compare July 24, 2023 19:48
@wenyihu6 wenyihu6 marked this pull request as ready for review July 24, 2023 20:57
@wenyihu6 wenyihu6 requested a review from a team as a code owner July 24, 2023 20:57
@wenyihu6 wenyihu6 requested a review from kvoli July 24, 2023 20:57
Copy link
Collaborator

@kvoli kvoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

See review comments on #107354

I don't think there are any which block this first commit.

Reviewed 8 of 8 files at r1, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @wenyihu6)

This patch lays the backbone of the randomized testing framework. Currently, it
only supports default configuration for all options, implying that there is no
randomization yet. Additionally, it refactors some of the existing structure in
data_driven_test. Note that this should not change any existing behavior, and
the main purpose is to make future commits cleaner.

TestRandomized is a randomized testing framework designed to validate
allocator by creating randomized configurations, generating corresponding
allocator simulations, and validating assertions on the final state.

Input of the framework (fields in the testSetting struct):
1. numIterations (int, default: 3): specifies number of test iterations to be
run, each with different random configurations generated
2. duration (time.Duration, default: 30min): defined simulated duration of
each iteration verbose (bool, default: false): enables detailed simulation
information failing output
3. randSeed (int64, default: 42): sets seed value for random number
generation
4. assertions ([]SimulationAssertion, default: conformanceAssertion with 0
under-replication, 0 over-replication, 0 violating, and 0 unavailable):
defines criteria for validation assertions
5. randOptions: guides the aspect of the test configuration that should be
randomized. This includes:
- cluster (bool): indicates if the cluster configuration should be randomized
- ranges (bool): indicates if the range configuration should be randomized
- load (bool): indicates if the workload configuration should be randomized
- staticSettings (bool): indicates if the simulation static settings should
be randomized
- staticEvents (bool): indicates if static events, including any delayed
events to be applied during the simulation, should be randomized

RandTestingFramework is initialized with a specified testSetting and
maintained its state across all iterations. Each iteration in
RandTestingFramework executes the following steps:
1. Generates a random configuration based on whether the aspect of the test
configuration is set to be randomized in randOptions
2. Executes a simulation and store any assertion failures in a buffer

Part of: cockroachdb#106311
Release note: None
@wenyihu6
Copy link
Contributor Author

Most comment don't apply to this commit, except for the GetNumStores part. But it doesn't need to be included in the first commit, so I've removed it and will add it in the next one.

@wenyihu6
Copy link
Contributor Author

TFTR!

bors r=kvoli

@craig
Copy link
Contributor

craig bot commented Jul 26, 2023

This PR was included in a batch that timed out, it will be automatically retried

@craig
Copy link
Contributor

craig bot commented Jul 26, 2023

Build succeeded:

@craig craig bot merged commit ea9e64e into cockroachdb:master Jul 26, 2023
@wenyihu6 wenyihu6 deleted the rand-framework branch July 27, 2023 03:51
@wenyihu6 wenyihu6 self-assigned this Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants