Comment doesn't get posted when PR is opened by an external contributor

[AlessandroMiola]

Hi @ewjoachim, everybody,
first off, thank you for your amazing work on this Github action!

When trying to add this action to the CI pipeline of an open-source library, I did encounter an issue, namely the fact that me, as an external contributor (but this seems to happen to all people, but the maintainers) opening a PR, I am able to generate the coverage comment artifacts, but I am not able to get the comment posted on the PR. These kinds of annotations do appear within the summary of
coverage-pr-comment job (as per the name I gave to it) pertaining to the CI workflow:

Notice: Starting action
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic "HTTP/1.1 200 OK"
Notice: Generating comment for PR
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic/contents/data.json?ref=python-coverage-comment-action-data "HTTP/1.1 404 Not Found"
Notice: HTTP Request: GET https://api.github.com/user "HTTP/1.1 403 Forbidden"
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic/issues/9770/comments "HTTP/1.1 200 OK"
Notice: Adding new comment
Notice: HTTP Request: POST https://api.github.com/repos/pydantic/pydantic/issues/9770/comments "HTTP/1.1 403 Forbidden"
Notice: Cannot post comment. This is probably because this is an external PR, so it's expected. Ensure you have an additional `workflow_run` step configured as explained in the documentation (or alternatively, give up on PR comments for external PRs).
Notice: Ending action

I tried to follow the indications in https://github.com/marketplace/actions/python-coverage-comment#basic-usage and the workflow appears to be as such: ci.yml

  name: CI
  
  on:
    push:
      branches:
        - main
      tags:
        - '**'
    pull_request: {}

  [...]

    coverage-combine:
        needs: [test, test-mypy]
        runs-on: ubuntu-latest
    
        steps:
          - uses: actions/checkout@v4
    
          - uses: actions/setup-python@v5
            with:
              python-version: '3.12'
    
          - name: get coverage files
            uses: actions/download-artifact@v4
            with:
              merge-multiple: true
              pattern: coverage-*
              path: coverage
    
          - run: pip install coverage[toml]
    
          - run: ls -la coverage
          - run: coverage combine coverage
          - run: coverage report
          - run: coverage html --show-contexts --title "pydantic coverage for ${{ github.sha }}"
    
          
          - name: Store coverage data
            uses: actions/upload-artifact@v4
            with:
              name: coverage-data
              path: .coverage
          

          - name: Store coverage html
            uses: actions/upload-artifact@v4
            with:
              name: coverage-html
              path: htmlcov
      
      coverage-pr-comment:
        needs: coverage-combine
        runs-on: ubuntu-latest
        if: github.event_name == 'pull_request'
        permissions:
          pull-requests: write
          contents: write
    
        steps:
          - uses: actions/checkout@v4
    
          - name: Download coverage data
            uses: actions/download-artifact@v4
            with:
              name: coverage-data
    
          - name: Generate coverage comment
            id: coverage-comment
            uses: py-cov-action/python-coverage-comment-action@v3
            with:
              GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    
          - name: Store coverage comment
            uses: actions/upload-artifact@v4
            if: steps.coverage-comment.outputs.COMMENT_FILE_WRITTEN == 'true'
            with:
              name: python-coverage-comment-action
              path: python-coverage-comment-action.txt

coverage.yml is

name: Post coverage comment

on:
  workflow_run:
    workflows: ["CI"]
    types:
      - completed

jobs:
  post-coverage-comment:
    name: Push coverage comment
    runs-on: ubuntu-latest
    if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
    permissions:
      pull-requests: write
      contents: write
      actions: read
    steps:
      # DO NOT run actions/checkout here, for security reasons
      # For details, refer to https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
      - name: Post comment
        uses: py-cov-action/python-coverage-comment-action@v3
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITHUB_PR_RUN_ID: ${{ github.event.workflow_run.id }}

Can you please point me to where the issue might lie on in your opinion?
Thank you! :)

EDIT

Hi Joachim, I took some time to revise a couple of things and I noticed that what I previously posted IMO could not be sufficient to give a complete picture. So here are a couple of more details.

Besides the annotations reported above, which do relate to the PR comment component of the CI workflow, here are the annotations pertaining to the PR comment component of the coverage workflow.

Notice: Starting action
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic "HTTP/1.1 200 OK"
Notice: Posting comment to PR
Notice: HTTP Request: GET https://api.github.com/user "HTTP/1.1 403 Forbidden"
Notice: Search for PR associated with run id 9815232181
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic/actions/runs/9815232181 "HTTP/1.1 200 OK"
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic/pulls?state=open&head=d8e-ai%3Avalidate-base64&sort=updated&direction=desc "HTTP/1.1 200 OK"
Notice: PR number: 9770
Notice: Download associated artifacts
Notice: HTTP Request: GET https://api.github.com/repos/pydantic/pydantic/actions/runs/9815232181/artifacts "HTTP/1.1 200 OK"
Notice: Artifact was not found, which is probably because it was probably already posted by a previous step.
Traceback (most recent call last):
  File "/workdir/coverage_comment/github.py", line 59, in download_artifact
    artifact = next(
               ^^^^^
StopIteration


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/workdir/coverage_comment/main.py", line 302, in post_comment
    comment = github.download_artifact(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workdir/coverage_comment/github.py", line 63, in download_artifact
    raise NoArtifact(
coverage_comment.github.NoArtifact: Not artifact found with name python-coverage-comment-action in run 9815232181
Notice: Ending action

What caught my attention is the following (kudos to @fabiogallotti for the catch): https://api.github.com/repos/pydantic/pydantic/actions/runs/9815232181/artifacts does contain 30 elements only (GH's default, I guess) among which you won't effectively find the python-coverage-comment-action, but as soon as you access the next page at https://api.github.com/repos/pydantic/pydantic/actions/runs/9815232181/artifacts?page=2, the relevant artifact does appear!

Could this be the issue? Namely,

python-coverage-comment-action/coverage_comment/github.py

Line 57 in ef2cb6c

artifacts = repo_path.actions.runs(run_id).artifacts.get().artifacts

not handling pagination correctly? (If I'm able) I'll try to add a test (^^^) to verify such hypothesis while waiting for your answer.

Thank you again! :)

EDIT 2 (^^^)

As for the (existing) tests, some of them do fail on my Windows (😢) machine; most of them fail because of paths slashes within strings and I could find easy OS-agnostic solutions to make them pass, others are trickier (eg some tests in unit\test_subprocess.py) and made me stop for the moment.

[ewjoachim]

(I'll do my best to answer but I'm a little busy for the next few days. If I don't answer within 10 days, feel free to bump :) )

[AlessandroMiola]

Gently pinging @ewjoachim :)
Thank you!

[ewjoachim]

Seeing your edits: I think you nailed it :) it's probably a pagination issue, that would be the error you'd get.

I'm not going to be very helpful for windows debugging, apart from suggesting WSL or Docker ^^'
I have no idea how subprocesses work on Windows and cannot easily access a windows machine to test, sorry. That said, GitHub actions will probably be happy to run the tests for you.
I'm really sorry that the contribution process for windoS folks is subpar :/

[ewjoachim]

Ah: feel free to open an issue or a draft PR so we can continue discussing over there.

[AlessandroMiola]

Seeing your edits: I think you nailed it :) it's probably a pagination issue, that would be the error you'd get.

I'm not going to be very helpful for windows debugging, apart from suggesting WSL or Docker ^^' I have no idea how subprocesses work on Windows and cannot easily access a windows machine to test, sorry. That said, GitHub actions will probably be happy to run the tests for you. I'm really sorry that the contribution process for windoS folks is subpar :/

Indeed, with WSL everything works smoothly:). I'll try to start working on this in the next days and let you know.

[ewjoachim]

Closed by #453