Add std-libs benchmarks GH workflow #7597

Akirathan · 2023-08-17T10:39:58Z

Closes #7490

Pull Request Description

Adds "Benchmark Standard Libraries" workflow that can be run with sbt std-benchmarks/bench.

The documentation for the workflow is in benchmarks.md

Important Notes

Tweak some warmup / measurement configurations for some benchmark such that they have enough time for warmup.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:
- Unit tests have been written where possible.
- If GUI codebase was changed, the GUI was tested when built using ./run ide build.

Akirathan · 2023-08-17T10:44:20Z

.github/workflows/std-libs-benchmark.yml

This is essentially a clone of engine-benchmark.yml (which used to be named benchmark.yml). With the only exception that instead of ./run backend benchmark runtime, it invokes ./run backend benchmark enso-jmh

Akirathan · 2023-08-17T10:45:24Z

build/build/paths.yaml

@@ -100,6 +101,9 @@
        docker-entrypoint.sh:
        Dockerfile:
    simple-library-server/:
+  std-bits/:
+    benchmarks/:
+      bench-report.xml:


This is where sbt std-benchmarks/bench saves its output. In the same format as sbt runtime/bench. After the benchmarks are run, I would like to upload this file as an artifact to the GH Run.

Can you add this to the docs?

Which docs do you mean? This is a CI internal stuff that just instructs the CI job where to find the files that should be uploaded as artifacts.

Akirathan · 2023-08-17T10:51:05Z

@mwu-tow I would like to try manually running the new "Benchmark Standard libraries" action to ensure that it does what it should do - run benchmarks and upload an XML report artifact. For that, we should probably merge this PR first, right?

mwu-tow · 2023-08-17T13:14:02Z

@mwu-tow I would like to try manually running the new "Benchmark Standard libraries" action to ensure that it does what it should do - run benchmarks and upload an XML report artifact. For that, we should probably merge this PR first, right?

I think it should be possible to manually dispatch a workflow using the API or their CLI wrapper: https://cli.github.com/manual/gh_workflow_run

GitHub CLI
GitHub CLI
Take GitHub to the command line

…registered by GH API

Akirathan · 2023-08-18T17:59:41Z

According to https://stackoverflow.com/a/67840292/4816269, one has to first, at least temporarily, add on: pull-request to the new Workflow definition, so that the Workflow is run at least once. Then gh workflow list will recognize the new workflow.

Stack Overflow
Github Actions: how to run a workflow created on a non-master branch from the workflow_dispatch event?
Can someone help me understand the behaviour of the Github Actions tab? As somebody new to Actions working on a third party repo I would like to be able to create an action on a branch and execute ...

…flow is registered by GH API" This reverts commit 7effbc1.

…registered by GH API

…-for-std-benchmarks

JaroslavTulach

Looks like everything is working and the results of workflow run seem fine too.

Akirathan · 2023-08-22T14:12:41Z

@mwu-tow I believe this PR is ready to merge. Since the last "Benchmark Standard Libraries" workflow run was successful and its output seems reasonable. Please, look into the build part modifications.

radeusgd · 2023-08-22T14:58:49Z

.github/workflows/std-libs-benchmark.yml

+      - if: failure() && runner.os == 'Windows'
+        name: List files if failed (Windows)
+        run: Get-ChildItem -Force -Recurse
+      - if: failure() && runner.os != 'Windows'
+        name: List files if failed (non-Windows)
+        run: ls -lAR


Btw. @mwu-tow are we ever using results of this?

Good point. If you accidentally click on the output of this step, the UI freezes for a while, that's all I know. But I guess it might be useful sometimes when you debug CI jobs.

radeusgd · 2023-08-22T15:00:05Z

test/Benchmarks/src/Vector/Array_Proxy_Bench.enso

 collect_benches = Bench.build builder->
-    vector_size = 10000
+    vector_size = 1000 * 1000


What was dictating these changes?

From the PR description:

Tweak some warmup / measurement configurations for some benchmark such that they have enough time for warmup.

Optimally, we want to have roughly 5 seconds for warm up and 5 seconds for the measurement phase for each benchmark because we want the whole job to run no longer than 90 minutes and at the same time, we would like to have a reasonable number of benchmarks. From the perspective of performance measurement, 5 seconds for warmup is far too low. But it is sufficient for our purposes - we do not strive to publish a paper about the Engine performance after all.

I was driven by a rule of thumb: The score of each benchmark should be roughly between 0.5 ms/op and 500 ms/op. That way, it is ensured that the benchmark is roughly invoked at least 20 times in the warmup phase.

In this particular case, benchmarks in Array_Proxy_Bench were too fast - the score was lower than 0.005 ms/op, so I increased the data size.

Add std-libs benchmarks GH workflow

76e848b

Akirathan self-assigned this Aug 17, 2023

Akirathan requested a review from mwu-tow as a code owner August 17, 2023 10:39

Remove unused benchmark.yml workflow

1132ec7

Akirathan commented Aug 17, 2023

View reviewed changes

Fix run backend benchmark command name

b9ab659

This was referenced Aug 17, 2023

Ability to invoke all std benchmarks via jmh #7519

Merged

Ensure that stdlib benchmark reports are collected #7598

Closed

Akirathan added the -ci label Aug 17, 2023

[WIP] Add on pull_request so that the std-libs-benchmark workflow is …

7effbc1

…registered by GH API

Akirathan added 3 commits August 18, 2023 20:00

Revert "[WIP] Add on pull_request so that the std-libs-benchmark work…

1108417

…flow is registered by GH API" This reverts commit 7effbc1.

Fix sbt command in the workflow

c749cf4

[WIP] Add on pull_request so that the std-libs-benchmark workflow is …

78cef4f

…registered by GH API

Akirathan added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Aug 21, 2023

Akirathan added 2 commits August 21, 2023 15:16

Merge branch 'develop' into wip/akirathan/7490-Create-a-GitHub-action…

c849567

…-for-std-benchmarks

Tweak warmup/measure configs of some benchmarks.

e358c4b

Akirathan requested review from jdunkerley, radeusgd and GregoryTravis as code owners August 21, 2023 13:58

GregoryTravis approved these changes Aug 21, 2023

View reviewed changes

std-benchmarks/bench is dependent on buildEngineDistribution task

e8cc78a

Akirathan requested review from 4e6, JaroslavTulach and hubertp as code owners August 21, 2023 16:36

Akirathan added the CI: No changelog needed Do not require a changelog entry for this PR. label Aug 21, 2023

JaroslavTulach approved these changes Aug 22, 2023

View reviewed changes

Benchmark Standard Libraries job is not run after pull request

93fd353

Akirathan removed the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Aug 22, 2023

Workaround for compilation failures on Windows.

e3ca5c9

Akirathan mentioned this pull request Aug 22, 2023

Improve Engine benchmark reporting #5714

Open

radeusgd reviewed Aug 22, 2023

View reviewed changes

radeusgd approved these changes Aug 22, 2023

View reviewed changes

4e6 approved these changes Aug 22, 2023

View reviewed changes

jdunkerley approved these changes Aug 22, 2023

View reviewed changes

Akirathan force-pushed the wip/akirathan/7490-Create-a-GitHub-action-for-std-benchmarks branch from d1e7550 to e3ca5c9 Compare August 22, 2023 17:02

hubertp approved these changes Aug 23, 2023

View reviewed changes

mwu-tow approved these changes Aug 23, 2023

View reviewed changes

Akirathan merged commit c32bfad into develop Aug 23, 2023

Akirathan deleted the wip/akirathan/7490-Create-a-GitHub-action-for-std-benchmarks branch August 23, 2023 10:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add std-libs benchmarks GH workflow #7597

Add std-libs benchmarks GH workflow #7597

Akirathan commented Aug 17, 2023 •

edited

Loading

Akirathan Aug 17, 2023

Akirathan Aug 17, 2023

GregoryTravis Aug 21, 2023

Akirathan Aug 21, 2023

Akirathan commented Aug 17, 2023

mwu-tow commented Aug 17, 2023 •

edited by unfurl-links bot

Loading

Akirathan commented Aug 18, 2023 •

edited by unfurl-links bot

Loading

JaroslavTulach left a comment

Akirathan commented Aug 22, 2023

radeusgd Aug 22, 2023

Akirathan Aug 22, 2023

radeusgd Aug 22, 2023

Akirathan Aug 22, 2023

Add std-libs benchmarks GH workflow #7597

Add std-libs benchmarks GH workflow #7597

Conversation

Akirathan commented Aug 17, 2023 • edited Loading

Pull Request Description

Important Notes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Akirathan commented Aug 17, 2023

mwu-tow commented Aug 17, 2023 • edited by unfurl-links bot Loading

Akirathan commented Aug 18, 2023 • edited by unfurl-links bot Loading

JaroslavTulach left a comment

Choose a reason for hiding this comment

Akirathan commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Akirathan commented Aug 17, 2023 •

edited

Loading

mwu-tow commented Aug 17, 2023 •

edited by unfurl-links bot

Loading

Akirathan commented Aug 18, 2023 •

edited by unfurl-links bot

Loading