Skip to content

Commit

Permalink
[NO-TICKET] Raise benchmark default durations
Browse files Browse the repository at this point in the history
**What does this PR do?**

This PR goes through our existing benchmarks, and for those that used
a default duration of 10 or 12 seconds, raises the duration to 30s.

**Motivation:**

We've observed most of these benchmarks having flaky
regressions/improvements on PRs that shouldn't affect them.

I suspect that the running the benchmarks for such short time may
be contributing to flakiness. For instance, one of the benchmarks
that flakes the most often is `profiler_sample_serialize.rb`, and
concidentally that's a benchmark where each iteration does a lot
of work, and thus in a typical 10 second run we may only see around
70 iterations.

Hopefully by running the benchmarks for slightly longer we'll have
more consistent results.

**Additional Notes:**

There is a downside to this -- because our benchmarks are currently
executed sequentially in CI, this will make the benchmark run took
quite longer than it used to.

Hopefully this trade-off is reasonable; if not, we can re-evaluate.

**How to test the change?**

Check the latest benchmarks CI run, and confirm the tests are running
for 30 seconds, rather than 10/12.
  • Loading branch information
ivoanjo committed Aug 27, 2024
1 parent c515a5e commit ec4fe81
Show file tree
Hide file tree
Showing 6 changed files with 12 additions and 12 deletions.
4 changes: 2 additions & 2 deletions benchmarks/profiler_allocation.rb
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ def export(flush)
class ProfilerAllocationBenchmark
def run_benchmark
Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand All @@ -41,7 +41,7 @@ def run_benchmark
3.times { GC.start }

Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand Down
12 changes: 6 additions & 6 deletions benchmarks/profiler_gc.rb
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def create_profiler

def run_benchmark
Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand All @@ -46,7 +46,7 @@ def run_benchmark
end

Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand All @@ -72,7 +72,7 @@ def run_benchmark
end

Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand All @@ -91,7 +91,7 @@ def run_benchmark
Datadog::Profiling.wait_until_running

Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand All @@ -105,7 +105,7 @@ def run_benchmark
Datadog.configure { |c| c.profiling.enabled = false }

Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand All @@ -124,7 +124,7 @@ def run_benchmark
Datadog::Profiling.wait_until_running

Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_hold_resume_interruptions.rb
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def create_profiler

def run_benchmark
Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_sample_loop_v2.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def thread_with_very_deep_stack(depth: 500)

def run_benchmark
Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/profiler_sample_serialize.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ def create_profiler

def run_benchmark
Benchmark.ips do |x|
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 10, warmup: 2 }
benchmark_time = VALIDATE_BENCHMARK_MODE ? { time: 0.01, warmup: 0 } : { time: 30, warmup: 2 }
x.config(
**benchmark_time,
)
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/tracing_trace.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def open
# benchmarking platform to calculate helpful aggregate stats. Because benchmark-ips tries to run one iteration
# per 100ms, this means we'll have around 120 samples (give or take a small margin of error).
# @param [Integer] warmup in seconds. The default is 2 seconds.
def benchmark_time(time: 12, warmup: 2)
def benchmark_time(time: 30, warmup: 2)
VALIDATE_BENCHMARK_MODE ? { time: 0.001, warmup: 0 } : { time: time, warmup: warmup }
end

Expand Down

0 comments on commit ec4fe81

Please sign in to comment.