Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Benchmark Data Collection Endpoint #1

Open
danmayer opened this issue Nov 4, 2018 · 0 comments
Open

Add Benchmark Data Collection Endpoint #1

danmayer opened this issue Nov 4, 2018 · 0 comments

Comments

@danmayer
Copy link
Collaborator

danmayer commented Nov 4, 2018

Add an API to collect performance benchmark data across Coverband CI runs.

We want to know that each point release is equal or faster than previous releases. We also want to be able to see the performance impact (if any) of PRs before merging them. The first step is collect all the data in one place then add comparisons graphs and other features over time.

Assume a client would post data on each benchmark run, with data following the general benchmark_ips formats, for example, the current Coverband benchmark output looks like so

runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
       store_reports     3.000  i/100ms
Calculating -------------------------------------
       store_reports     36.764  (± 8.2%) i/s -    549.000  in  15.023371s
comparing Coverage loaded/not, this takes some time for output...
coverage loaded
Coverband configured with file store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     16.063  (± 6.2%) i/s -    191.000  in  12.034017s
Pausing here -- run Ruby again to measure the next benchmark...
just the work
Coverband configured with file store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     17.157  (± 5.8%) i/s -    206.000  in  12.043687s
Comparison:
        no coverband:       17.2 i/s
           coverband:       16.1 i/s - same-ish: difference falls within error
comparing Coverage loaded/not, this takes some time for output...
coverage loaded
Coverband configured with default Redis store
Warming up --------------------------------------
           coverband     1.000  i/100ms
Calculating -------------------------------------
           coverband     13.735  (± 7.3%) i/s -    165.000  in  12.041785s
Pausing here -- run Ruby again to measure the next benchmark...
just the work
Coverband configured with default Redis store
Warming up --------------------------------------
        no coverband     1.000  i/100ms
Calculating -------------------------------------
        no coverband     17.176  (± 5.8%) i/s -    206.000  in  12.028462s
Comparison:
        no coverband:       17.2 i/s
           coverband:       13.7 i/s - 1.25x  slower

The API endpoint should collect and store in the AR database the following information along with a basic index and show view page displaying all the collected data.

Below is a brief idea of what data probably is needed, but some thought should be put into the data format. as a starting point for considering data / API design.

{'benchmark_results' =>
    {'ruby_version' => 'xyz',
      'git hash,branch, or PR name' => 'xyz',
      'benchmarks' => [
           {'benchmark_name' => 'xyz',
             'calculations' => [{data_point_name, i/s, total iterations, total time}],
             'comparison' => [{data_point_name, i/s, diff calculation, note}]
            }]
     }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant