#145: bug/get_legistar_content_uris - adjusting inital media URL extraction #785
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Reference | |
| # https://github.com/CouncilDataProject/cookiecutter-cdp-deployment | |
| name: Test Scraper | |
| on: | |
| issue_comment: | |
| types: [created, edited] | |
| jobs: | |
| test-scraper: | |
| if: ${{ startsWith(github.event.comment.body, '/test-scraper') }} | |
| runs-on: ubuntu-latest | |
| permissions: | |
| issues: write | |
| pull-requests: write | |
| steps: | |
| - name: Show Acknowledgement | |
| uses: peter-evans/create-or-update-comment@v3 | |
| with: | |
| issue-number: ${{ github.event.issue.number }} | |
| body: | | |
| Begin `/test-scraper`. This may take several minutes. | |
| ######################################################################### | |
| # Check initiator is a member of CDP | |
| - name: Get CDP Organization Members | |
| id: cdp-members | |
| env: | |
| GITHUB_TOKEN: ${{ secrets.ACCESS_TOKEN }} | |
| run: | | |
| members="$(gh api -X GET 'orgs/CouncilDataProject/members' -F per_page=100 --paginate --cache 1h --jq '[.[].login] | join("---")')" | |
| echo "::set-output name=members::$members" | |
| - name: Generate Safe Username Check | |
| id: safe-username | |
| run: | | |
| username=${{ github.event.comment.user.login }} | |
| username="---$username---" | |
| echo "::set-output name=username::$username" | |
| - name: Check Job Initiator - Message | |
| if: | | |
| !contains( | |
| steps.cdp-members.outputs.members, | |
| steps.safe-username.outputs.username | |
| ) | |
| uses: peter-evans/create-or-update-comment@v3 | |
| with: | |
| issue-number: ${{ github.event.issue.number }} | |
| body: | | |
| ❌ ❌ **Rejected** ❌ ❌ | |
| User (${{ github.event.comment.user.login }}) attempted to run `/test-scraper` without permissions. | |
| _Only users which are members of the CouncilDataProject organization may run `/test-scraper` bot._ | |
| **Stopping Scraper Test** | |
| - name: Check Job Initiator - Exit | |
| if: | | |
| !contains( | |
| steps.cdp-members.outputs.members, | |
| steps.safe-username.outputs.username | |
| ) | |
| run: | | |
| exit 1 | |
| ######################################################################### | |
| # Workflow Setup | |
| # unlike pull_request and issue triggers | |
| # must specify merge HEAD as ref for action/checkout | |
| # when triggered by comment in pull request | |
| - name: Check Out Merge Repository | |
| if: ${{ github.event.issue.pull_request }} | |
| uses: actions/checkout@v4 | |
| with: | |
| ref: refs/pull/${{ github.event.issue.number }}/merge | |
| - name: Check Out Reference Repository | |
| if: ${{ ! github.event.issue.pull_request }} | |
| uses: actions/checkout@v4 | |
| - name: Set Up Python | |
| uses: actions/setup-python@v2 | |
| with: | |
| python-version: 3.9 | |
| - name: Install Dependencies | |
| run: | | |
| python -m pip install --upgrade pip | |
| pip install .[test] | |
| - name: Run Scraper | |
| run: | | |
| python .github/workflows/scripts/run-scraper.py ${{ github.event.comment.body }} | |
| - name: Upload Results | |
| uses: actions/upload-artifact@v2 | |
| with: | |
| name: scraper-test-results | |
| # default 90 seems unnecessary for these test results | |
| retention-days: 7 | |
| path: | | |
| scraper-params.txt | |
| scraper-results.json | |
| - name: Show Test Result | |
| uses: peter-evans/create-or-update-comment@v3 | |
| with: | |
| issue-number: ${{ github.event.issue.number }} | |
| body: | | |
| `/test-scraper` is finished. | |
| Please go to the [Actions page](${{ github.event.repository.html_url }}/actions) then click on the latest workflow titled "${{ github.event.issue.title }}". | |
| The results are saved in `scraper-test-results` under Artifacts on the workflow summary page. |