Action is not creating any comments on PR

[nilsfriess]

I was trying to setup a Github action that runs when I open a PR and add a comment with the content /benchmark.
Here's my workflow file:

name: Benchmark test

on:
  issue_comment:
    types: [created, edited, deleted]

jobs:
  run-benchmark:
    if: contains(github.event.comment.html_url, '/pull/') && contains(github.event.comment.body, '/benchmark')
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4      
    - name: Run benchmark
      run: |
        python3 gen_random_result.py | tee ${GITHUB_WORKSPACE}/output.json
    - name: Download previous benchmark data
      uses: actions/cache@v4
      with:
        path: ./cache
        key: ${{ runner.os }}-benchmark

    - name: Store new result
      uses: benchmark-action/github-action-benchmark@v1
      with:
        tool: customSmallerIsBetter
        output-file-path: output.json
        external-data-json-path: ./cache/benchmark-data.json
        github-token: ${{ secrets.GITHUB_TOKEN }}
        summary-always: true
        comment-always: true

The gen_random_result.py script does exactly what its name says, it just generates some random data of the form

[
 {
    "name": "Test benchmark",
    "unit": "Seconds",
    "value": "0.4937825918725717"
 }
]

The action runs as expected but it doesn't generate a comment on the PR with the results.
Is there something I'm missing?

[nilsfriess]

I think I found out what the problem is: Since I trigger the workflow on issue_comment, the "parent context" (don't know what to call it) is not a PR, so in this section here

github-action-benchmark/src/write.ts

Lines 247 to 249 in 55f6195

	return await (pr?.number
	? leavePRComment(repoMetadata.owner.login, repoMetadata.name, pr.number, body, commentId, token)
	: leaveCommitComment(repoMetadata.owner.login, repoMetadata.name, commitId, body, commentId, token));

the function leaveCommitComment will be called. Don't know how to fix it yet, but it shouldn't be too hard to get the parent PR from a comment.

[WorldSEnder]

Over at yew, we also have a setup which currently only comments on the commits themselves, not the PR.

The benchmarking is run on pull_request in this workflow. Important for this discussion is that we then use the generated artifacts in a workflow_run dependent workflow. This avoids giving out permissions and running the benchmarking workflow in the context of the pull request itself, instead using the second benchmark from the main branch to perform the tasks of reacting to and posting the results.

We thus have a similar issue. While it's possible to guide benchmark-action/github-action-benchmark to the correct commit with the ref option, there is no way to guide it to the correct issue number (although we already recover that as part of the workflow).

[gaby]

@nilsfriess You're missing permissions in your workflow:

permissions:
  # deployments permission to deploy GitHub pages website
  deployments: write
  # contents permission to update benchmark contents in gh-pages branch
  contents: write
  # allow posting comments to pull request
  pull-requests: write