Integrate Snyk properly with python module management

[jbrown-xentity]

User Story

In order to simplify python upgrades of dependencies that are at risk, data.gov sysadmin wants a snyk process that opens PR's that are reviewable and mergeable as is, without user management.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN a security concern is found in a python dependency for catalog or inventory
  WHEN a PR is created to correct the dependency
  THEN the dependencies input list is updated and a lock file is created. \

Background

Should consider taking on #3415 first, as snyk integration may be easier or harder with pipenv.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

[nickumia-reisys]

While this is not the solution we had hoped for, it is the best solution that we can come up with for now. Both catalog and inventory contain a cusotm script that runs a Snyk scan and creates a PR if there are any "fixable" vulnerabilities that exist. I turned off the "automatic PR" feature of Snyk for these two repos.

[nickumia-reisys]

So, the solution to this was writing a custom script that analyzes the snyk scan output and updates the requirements accordingly. If there is an update, it creates a PR:

• Custom script:
  • https://github.com/GSA/catalog.data.gov/blob/main/tools/snyk-update.py
  • https://github.com/GSA/inventory-app/blob/main/bin/snyk-update.py
• Daily snyk scan:
  • https://github.com/GSA/catalog.data.gov/blob/main/.github/workflows/snyk.yml
  • https://github.com/GSA/inventory-app/blob/main/.github/workflows/snyk.yml

Key points:

• Snyk scan run daily from github action
• Handle Snyk Scans with no Upgrade/Patch-ability #4125 (comment)
• If a fix is made, the requirements.in is updated with sed and then make update-requirements is run which then updates requirements.txt