Run benchmark automatically for performance pull requests #1297
Conversation
.github/workflows/benchmark.yaml
Outdated
| search="{ git = \"https://github.com/jstarry/yew\", branch = \"keyed-list\" }" | ||
| replace="{ git = \"${{ github.event.pull_request.head.html_url }}\", branch = \"${{ github.event.pull_request.head.ref }}\" }" | ||
| input=$(cat frameworks/keyed/yew/Cargo.toml) | ||
| output=${input//"$search"/"$replace"} | ||
| if [[ "$input" == "$output" ]]; then | ||
| echo "ERROR: failed to configure Cargo.toml" | ||
| exit 1 | ||
| fi | ||
| echo $output > frameworks/keyed/yew/Cargo.toml |
There was a problem hiding this comment.
This is very hacky but it does the trick. If we want to make this cleaner I think we should tackle it over at yewstack/js-framework-benchmark.
| (cd webdriver-ts && npm run build-prod) | ||
|
|
||
| - name: Benchmark | ||
| run: npm run bench -- --headless |
There was a problem hiding this comment.
I'm unsure about how reliable the benchmarks will be reliable/accurate if run on Github actions – are all containers in which workflows are run guaranteeed to have the same performance?
There was a problem hiding this comment.
Ah. I think Github Actions spins stuff up in containers, and it may be that even during a run the resources available to a container change to manage supply/demand amongst all jobs? I'll look at the documentation and see what they say.
There was a problem hiding this comment.
I don't think this is valid criticism of this PR since the current process already asks users to open a PR over at the js-framework-benchmark which also runs the benchmark on GitHub Actions.
You still make a good point, but I don't think it's that much of a problem. The workflow isn't running in a docker container, it's a virtual machine and if the documentation is to be trusted, these have consistent hardware resources.
There was a problem hiding this comment.
That's cool then; thanks!
There was a problem hiding this comment.
Continuous Benchmark has a paragraph about "Stability of Virtual Environment" which says:
As far as watching the benchmark results of examples in this repository, the amplitude of the benchmarks is about +- 10~20%.
|
Sweet, this looks good for now, we'll likely need to tweak it over time but that's fine with me |
Description
Right now the process of benchmarking changes is rather tedious and possibly even intimidating for newcomers.
This PR aims to help with this by automatically generating a benchmark report for pull requests labelled with "performance".
You can see it in action here: siku2#2
(Please ignore all of the deleted messages and force pushes. I was using the PR during development :D)
I wanted to use a
/benchmarkcommand to run the benchmarks manually but GitHub actions doesn't really support that right now...