Skip to content

Commit

Permalink
ci: use pagination to retrieve complete list of changed files [backport
Browse files Browse the repository at this point in the history
#6818 to 1.18] (#6829)

Backport of #6818 to 1.18

We make sure to fetch all the pages from the call to the pull request
GitHub REST API so that we can get the full list of changed files,
instead of just the first 30 entries.

## Checklist

- [x] Change(s) are motivated and described in the PR description.
- [x] Testing strategy is described if automated tests are not included
in the PR.
- [x] Risk is outlined (performance impact, potential for breakage,
maintainability, etc).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] [Library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
are followed. If no release note is required, add label
`changelog/no-changelog`.
- [x] Documentation is included (in-code, generated user docs, [public
corp docs](https://github.com/DataDog/documentation/)).
- [x] Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist

- [x] Title is accurate.
- [x] No unnecessary changes are introduced.
- [x] Description motivates each change.
- [x] Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes unless absolutely necessary.
- [x] Testing strategy adequately addresses listed risk(s).
- [x] Change is maintainable (easy to change, telemetry, documentation).
- [x] Release note makes sense to a user of the library.
- [x] Reviewer has explicitly acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment.
- [x] Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
- [x] If this PR touches code that signs or publishes builds or
packages, or handles credentials of any kind, I've requested a review
from `@DataDog/security-design-and-guidance`.
- [x] This PR doesn't touch any of that.
  • Loading branch information
P403n1x87 authored Sep 8, 2023
1 parent 7bca241 commit 1ab6977
Showing 1 changed file with 13 additions and 5 deletions.
18 changes: 13 additions & 5 deletions scripts/needs_testrun.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from argparse import ArgumentParser
import fnmatch
from functools import cache
from itertools import count
import json
import logging
import os
Expand Down Expand Up @@ -72,13 +73,18 @@ def get_changed_files(pr_number: int, sha: t.Optional[str] = None) -> t.Set[str]
"""
rest_check_failed = False
if sha is None:
files = set()
try:
url = f"https://api.github.com/repos/datadog/dd-trace-py/pulls/{pr_number}/files"
headers = {"Accept": "application/vnd.github+json"}
return {_["filename"] for _ in json.load(urlopen(Request(url, headers=headers)))}
for page in count(1):
url = f"https://api.github.com/repos/datadog/dd-trace-py/pulls/{pr_number}/files?page={page}"
headers = {"Accept": "application/vnd.github+json"}
result = {_["filename"] for _ in json.load(urlopen(Request(url, headers=headers)))}
if not result:
return files
files |= result
except Exception:
rest_check_failed = True
LOGGER.warning("Failed to get changed files from GitHub API")
LOGGER.warning("Failed to get changed files from GitHub API", exc_info=True)

if sha is not None or rest_check_failed:
diff_base = sha or get_merge_base(pr_number)
Expand Down Expand Up @@ -175,7 +181,9 @@ def main() -> bool:

argp.add_argument("suite", help="The suite to use", type=str)
argp.add_argument("--pr", help="The PR number", type=int, default=_get_pr_number())
argp.add_argument("--sha", help="Commit hash to use as diff base (defaults to PR merge root)", type=lambda v: v or None)
argp.add_argument(
"--sha", help="Commit hash to use as diff base (defaults to PR merge root)", type=lambda v: v or None
)
argp.add_argument("--verbose", "-v", action="store_true", help="Verbose output")

args = argp.parse_args()
Expand Down

0 comments on commit 1ab6977

Please sign in to comment.