Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include diff file in test-scraper.yml Action artifact #93

Open
dphoria opened this issue Apr 23, 2022 · 1 comment
Open

Include diff file in test-scraper.yml Action artifact #93

dphoria opened this issue Apr 23, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@dphoria
Copy link
Collaborator

dphoria commented Apr 23, 2022

Feature Description

A comment in the form /test-scraper --from_dt=<iso-formatted-datetime> --to_dt=<iso-formatted-datetime> get_{municipality_slug}_events in a pull request or an issue triggers the GitHub Action .github/workflows/test-scraper.yml. The Action uploads a ZIP file as an artifact. The ZIP contains a JSON file for the scraped events returned by cdp_scrapers.instances.get_{municipality_slug}_events().

If the Action was triggered by a pull request comment, it would be helpful to also include a diff file, comparing the results from the same call between the pull target branch (CouncilDataProject/cdp-scrapers main) and the pull merge branch.

Change summary:

  1. Do something similar to what test-scraper.yml does currently: save scraped events in JSON file.
  2. Repeat, using CouncilDataProject/cdp-scrapers main.
  3. Save output from diff between the two JSON files.
  4. Add the diff file in the ZIP file artifact for the Action.

Use Case

Often we find ourselves manually creating this kind of diff files to study changes (if any) some new code would yield in the scraped IngestionModels. Doing so automatically for a pull request /test-scraper run would reduce manual effort.

Solution

File(s) in .github/workflows/ (and subdirectories) updated as necessary to include diff files, described above, in the artifact ZIP.

@dphoria dphoria added the enhancement New feature or request label Apr 23, 2022
@evamaxfield
Copy link
Member

YES SUNG. This would be amazing. ♥️♥️♥️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants