-
Notifications
You must be signed in to change notification settings - Fork 375
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[NO-TICKET] Raise benchmark default durations
**What does this PR do?** This PR goes through our existing benchmarks, and for those that used a default duration of 10 or 12 seconds, raises the duration to 30s. **Motivation:** We've observed most of these benchmarks having flaky regressions/improvements on PRs that shouldn't affect them. I suspect that the running the benchmarks for such short time may be contributing to flakiness. For instance, one of the benchmarks that flakes the most often is `profiler_sample_serialize.rb`, and concidentally that's a benchmark where each iteration does a lot of work, and thus in a typical 10 second run we may only see around 70 iterations. Hopefully by running the benchmarks for slightly longer we'll have more consistent results. **Additional Notes:** There is a downside to this -- because our benchmarks are currently executed sequentially in CI, this will make the benchmark run took quite longer than it used to. Hopefully this trade-off is reasonable; if not, we can re-evaluate. **How to test the change?** Check the latest benchmarks CI run, and confirm the tests are running for 30 seconds, rather than 10/12.
- Loading branch information
Showing
6 changed files
with
12 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters