Existing sample/recipe for integrating asv with github

[yarikoptic]

Backstory aside - is anything aware of an existing use of asv integrated with GitHub?

What I envision is having a container image with eg buildbot and asv installed, to be deployed on a dedicated box, configured to monitor/report back on PRs to GitHub about impact on benchmarks.

May be there is a service which already does something like that?

I will look into setting up such one, but wanted to make sure I am not reinventing the wheel or making it square.

[saraedum]

@yarikoptic did you manage to build such a service?

[yarikoptic]

Nope, but my colleagues set it up to just run asv to get an estimate for performance impact comparing against master in PRs. See e.g. our datalad github actions workflow: https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml and the actual helper https://github.com/datalad/datalad/blob/master/tools/ci/benchmark-travis-pr.sh . Eventually we just might look into establishing self-hosted runner for actions: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners so we get more stable estimates

[yarikoptic]

note - that we nohow report back overall effect, this action is merely for manual investigation if there is a suspicion that PR might affect performance. Would be nice to report back to action state if e.g. any benchmark became significantly slower. we haven't looked into that yet

[saulshanabrook]

In case it helps anyone, I just set up asv to run a job on each PR which compares it against the branch it is being merged into. It doesn't save the results anywhere, but anyone can look at the logs there to see if affects the performance.

My github action looked like this:

  benchmark:
    name: benchmark
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v2
        with:
          python-version: "3.10"
      - uses: actions/checkout@v3
      - run: git fetch origin $GITHUB_BASE_REF:base $GITHUB_REF:pr
      - run: pip install asv virtualenv
      - run: asv machine --yes
      - run: asv continuous base pr --strict -e

[yarikoptic]

in https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml we also fail the workflow if any benchmarks gets significantly worse:

    - name: Fail if any benchmarks have slowed down too much
      run: |
        ! asv compare --factor 1.2 --split refs/bm/merge-target refs/bm/pr | grep -q "got worse"