You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add an API to collect performance benchmark data across Coverband CI runs.
We want to know that each point release is equal or faster than previous releases. We also want to be able to see the performance impact (if any) of PRs before merging them. The first step is collect all the data in one place then add comparisons graphs and other features over time.
Assume a client would post data on each benchmark run, with data following the general benchmark_ips formats, for example, the current Coverband benchmark output looks like so
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
store_reports 3.000 i/100ms
Calculating -------------------------------------
store_reports 36.764 (± 8.2%) i/s - 549.000 in 15.023371s
comparing Coverage loaded/not, this takes some time for output...
coverage loaded
Coverband configured with file store
Warming up --------------------------------------
coverband 1.000 i/100ms
Calculating -------------------------------------
coverband 16.063 (± 6.2%) i/s - 191.000 in 12.034017s
Pausing here -- run Ruby again to measure the next benchmark...
just the work
Coverband configured with file store
Warming up --------------------------------------
no coverband 1.000 i/100ms
Calculating -------------------------------------
no coverband 17.157 (± 5.8%) i/s - 206.000 in 12.043687s
Comparison:
no coverband: 17.2 i/s
coverband: 16.1 i/s - same-ish: difference falls within error
comparing Coverage loaded/not, this takes some time for output...
coverage loaded
Coverband configured with default Redis store
Warming up --------------------------------------
coverband 1.000 i/100ms
Calculating -------------------------------------
coverband 13.735 (± 7.3%) i/s - 165.000 in 12.041785s
Pausing here -- run Ruby again to measure the next benchmark...
just the work
Coverband configured with default Redis store
Warming up --------------------------------------
no coverband 1.000 i/100ms
Calculating -------------------------------------
no coverband 17.176 (± 5.8%) i/s - 206.000 in 12.028462s
Comparison:
no coverband: 17.2 i/s
coverband: 13.7 i/s - 1.25x slower
The API endpoint should collect and store in the AR database the following information along with a basic index and show view page displaying all the collected data.
Below is a brief idea of what data probably is needed, but some thought should be put into the data format. as a starting point for considering data / API design.
Add an API to collect performance benchmark data across Coverband CI runs.
We want to know that each point release is equal or faster than previous releases. We also want to be able to see the performance impact (if any) of PRs before merging them. The first step is collect all the data in one place then add comparisons graphs and other features over time.
Assume a client would post data on each benchmark run, with data following the general
benchmark_ips
formats, for example, the current Coverband benchmark output looks like soThe API endpoint should collect and store in the AR database the following information along with a basic index and show view page displaying all the collected data.
Below is a brief idea of what data probably is needed, but some thought should be put into the data format. as a starting point for considering data / API design.
The text was updated successfully, but these errors were encountered: