-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
asim: introduce rand testing framework with default-only settings #107255
Conversation
1982e47
to
4806d6e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See review comments on #107354
I don't think there are any which block this first commit.
Reviewed 8 of 8 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @wenyihu6)
This patch lays the backbone of the randomized testing framework. Currently, it only supports default configuration for all options, implying that there is no randomization yet. Additionally, it refactors some of the existing structure in data_driven_test. Note that this should not change any existing behavior, and the main purpose is to make future commits cleaner. TestRandomized is a randomized testing framework designed to validate allocator by creating randomized configurations, generating corresponding allocator simulations, and validating assertions on the final state. Input of the framework (fields in the testSetting struct): 1. numIterations (int, default: 3): specifies number of test iterations to be run, each with different random configurations generated 2. duration (time.Duration, default: 30min): defined simulated duration of each iteration verbose (bool, default: false): enables detailed simulation information failing output 3. randSeed (int64, default: 42): sets seed value for random number generation 4. assertions ([]SimulationAssertion, default: conformanceAssertion with 0 under-replication, 0 over-replication, 0 violating, and 0 unavailable): defines criteria for validation assertions 5. randOptions: guides the aspect of the test configuration that should be randomized. This includes: - cluster (bool): indicates if the cluster configuration should be randomized - ranges (bool): indicates if the range configuration should be randomized - load (bool): indicates if the workload configuration should be randomized - staticSettings (bool): indicates if the simulation static settings should be randomized - staticEvents (bool): indicates if static events, including any delayed events to be applied during the simulation, should be randomized RandTestingFramework is initialized with a specified testSetting and maintained its state across all iterations. Each iteration in RandTestingFramework executes the following steps: 1. Generates a random configuration based on whether the aspect of the test configuration is set to be randomized in randOptions 2. Executes a simulation and store any assertion failures in a buffer Part of: cockroachdb#106311 Release note: None
Most comment don't apply to this commit, except for the |
TFTR! bors r=kvoli |
This PR was included in a batch that timed out, it will be automatically retried |
Build succeeded: |
This patch lays the backbone of the randomized testing framework. Currently, it
only supports default configuration for all options, implying that there is no
randomization yet. Additionally, it refactors some of the existing structure in
data_driven_test. Note that this should not change any existing behavior, and
the main purpose is to make future commits cleaner.
TestRandomized is a randomized testing framework designed to validate
allocator by creating randomized configurations, generating corresponding
allocator simulations, and validating assertions on the final state.
Input of the framework (fields in the testSetting struct):
run, each with different random configurations generated
each iteration verbose (bool, default: false): enables detailed simulation
information failing output
generation
under-replication, 0 over-replication, 0 violating, and 0 unavailable):
defines criteria for validation assertions
randomized. This includes:
be randomized
events to be applied during the simulation, should be randomized
RandTestingFramework is initialized with a specified testSetting and
maintained its state across all iterations. Each iteration in
RandTestingFramework executes the following steps:
configuration is set to be randomized in randOptions
Part of: #106311
Release note: None