Feature: Scorecard Dependency-diff API, CLI and Action

[aidenwang9867]

Is your feature request related to a problem? Please describe.
The scorecard Action only reports the Scorecard results for the repository under analysis. The project will surface scorecard results for the dependencies for new pull requests.

Describe the solution you'd like

1. Add an API design that can fetch direct dependency changes (diffs) from a pull request using the GitHub Dependency Review API, surface Scorecard checks and scores for those dependencies and return the results.

2. Add a new Scorecard sub-command CLI, scorecard dependency-diff, which uses the above API and outputs the result dependencies as JSON.

3. Add a new Scorecard Action running on a pull request, use the API to visualize the dependency-diff and their Scorecard checking results in the PR comments as markdown.

Additional context
Assignee: @aidenwang9867

[naveensrinivasan]

Is your feature request related to a problem? Please describe. The scorecard Action only reports the Scorecard results for the repository under analysis. The project will surface scorecard results for the dependencies for new pull requests.

Describe the solution you'd like

1. Add a design that can fetch direct dependency changes (diffs) from a pull request using the GitHub Dependency Review API, get dependencies of these direct dependencies (indirect ones) and vulnerabilities of both direct and indirect dependencies.
2. Information regarding Dependencies and vulnerabilities can be retrieved from the BigQuery dataset Open Source Insights.
3. Visualize the result to the PR comments like what CodeCov does, with information showing if a dependency is removed or added. Also, give vulnerability alerts (VulnID, VulnSource, CVSS3Score, Severity, etc.) either together in the comments, or in the repo Security tag.
4. Blocks the PR if vulnerabilities found.

Describe alternatives you've considered

Additional context This design is Scorecard-focused and is expected to be used in a pull request as a part of the Scorecard Action pre-submit check automation.

With this one in place, the Scorecard Action will be further empowered to check changes of dependencies between a base commit and a head commit in GitHub pull requests, giving detailed alerts to repository maintainers if vulnerabilities (CVEs, GHSAs, OSVs, etc.) are found in the added/updated dependencies.

Assignee: @aidenwang9867

This is cool. BigQuery needs authentication to run these checks. Would we require the user to provide those credentials?

[laurentsimon]

I also wonder whether it'd be useful to have a

1. command line option to get the results for a dependencies, rather than run on a project itself
2. a --from-bq option in Scorecard to fetch a result from BQ instead of running scorecard on a project itself

Note that the project is focused on exposing the functionality in the Action first. I'm asking the questions above out of curiosity :-)

[naveensrinivasan]

Note that the project is focused on exposing the functionality in the Action first. I'm asking the questions above out of curiosity :-)

Why not build that capability within Scorecard and then build it in the action? This will help with the larger customers who aren't just focused on Action.

[aidenwang9867]

Yeah I'm preparing for my first pull request and get confused now about where I should put my code to. Currently, I'm testing my code locally in the main function and visualizing the results to the StdOut.

also cc @azeemshaikh38

[aidenwang9867]

Is your feature request related to a problem? Please describe. The scorecard Action only reports the Scorecard results for the repository under analysis. The project will surface scorecard results for the dependencies for new pull requests.
Describe the solution you'd like

1. Add a design that can fetch direct dependency changes (diffs) from a pull request using the GitHub Dependency Review API, get dependencies of these direct dependencies (indirect ones) and vulnerabilities of both direct and indirect dependencies.
2. Information regarding Dependencies and vulnerabilities can be retrieved from the BigQuery dataset Open Source Insights.
3. Visualize the result to the PR comments like what CodeCov does, with information showing if a dependency is removed or added. Also, give vulnerability alerts (VulnID, VulnSource, CVSS3Score, Severity, etc.) either together in the comments, or in the repo Security tag.
4. Blocks the PR if vulnerabilities found.

Describe alternatives you've considered
Additional context This design is Scorecard-focused and is expected to be used in a pull request as a part of the Scorecard Action pre-submit check automation.
With this one in place, the Scorecard Action will be further empowered to check changes of dependencies between a base commit and a head commit in GitHub pull requests, giving detailed alerts to repository maintainers if vulnerabilities (CVEs, GHSAs, OSVs, etc.) are found in the added/updated dependencies.
Assignee: @aidenwang9867

This is cool. BigQuery needs authentication to run these checks. Would we require the user to provide those credentials?

Thanks Naveen and, great question :D
My answer is no - we won't ask the users to provide credentials.

Instead, I am actively building a REST API service on a GKE with an external IP, putting the authentication & authorization process on the service so that the Scorecard project can connect to my REST APIs to query the BigQuery dataset (without additional credentials needed).

[naveensrinivasan]

Instead, I am actively building a REST API service on a GKE with an external IP, putting the authentication & authorization process on the service so that the Scorecard project can connect to my REST APIs to query the BigQuery dataset (without additional credentials needed).

OK, Probably that should have been called out. Is there a design for this? So that we can understand and have a consensus.

[aidenwang9867]

Instead, I am actively building a REST API service on a GKE with an external IP, putting the authentication & authorization process on the service so that the Scorecard project can connect to my REST APIs to query the BigQuery dataset (without additional credentials needed).

OK, Probably that should have been called out. Is there a design for this? So that we can understand and have a consensus.

I am doing some modifications on my design doc, and then i'll share it with you via email by the end of tomorrow, so that we can discuss it in Thursday's meeting.

[naveensrinivasan]

Cool @aidenwang9867! Can you please post it in the google groups? Thanks

[aidenwang9867]

Cool @aidenwang9867! Can you please post it in the google groups? Thanks

np

[naveensrinivasan]

Based on what was presented in the Bi-Weekly scorecard meeting and the design document.

Here are the things that we have discussed.

1. v0 was supposed to be only running scorecard against dependencies for the repository and not build the REST endpoint for scorecard.
2. It was discussed that GitHub API for figuring out the dependencies doesn't support transitive dependencies for ecosystems like python, npm, and a few others.
3. deps.dev provides transitive dependencies for a given repository
4. It was discussed as part of v0 utilizing bot the GitHub API and deps.dev to get all the dependencies. But if a repository adds a new dependency in a PR then deps.dev wouldn't be aware of that and so this new check would miss those transitive dependencies.
5. deps.dev link may be out of sync with a real-time run of scorecard. We need to be careful about how to present the data

[aidenwang9867]

Based on what was presented in the Bi-Weekly scorecard meeting and the design document.

Here are the things that we have discussed.

1. v0 was supposed to be only running scorecard against dependencies for the repository and not build the REST endpoint for scorecard.
2. It was discussed that GitHub API for figuring out the dependencies doesn't support transitive dependencies for ecosystems like python, npm, and a few others.
3. deps.dev provides transitive dependencies for a given repository
4. It was discussed as part of v0 utilizing bot the GitHub API and deps.dev to get all the dependencies. But if a repository adds a new dependency in a PR then deps.dev wouldn't be aware of that and so this new check would miss those transitive dependencies.

Thank you so much Naveen, for summarizing the things we discussed!
cc @laurentsimon @azeemshaikh38

[laurentsimon]

4. It was discussed as part of v0 utilizing bot the GitHub API and deps.dev to get all the dependencies. But if a repository adds a new dependency in a PR then deps.dev wouldn't be aware of that and so this new check would miss those transitive dependencies.

by "repository" you mean the repository with scorecard Action installed?

Or do you mean:

1. a dependency P1 of the repository adds a new dependency P2
2. repository adds P1 in a PR (P2 is therefore a transitive dependency)
3. deps.dev has stale data because they only refresh their dataset every X days.

Is this what you meant?

[aidenwang9867]

4. It was discussed as part of v0 utilizing bot the GitHub API and deps.dev to get all the dependencies. But if a repository adds a new dependency in a PR then deps.dev wouldn't be aware of that and so this new check would miss those transitive dependencies.

by "repository" you mean the repository with scorecard Action installed?

Or do you mean:

1. a dependency P1 of the repository adds a new dependency P2
2. repository adds P1 in a PR (P2 is therefore a transitive dependency)
3. deps.dev has stale data because they only refresh their dataset every X days.

Is this what you meant?

@naveensrinivasan in case u missed this message from Laurent :D

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[github-actions]

This issue has been marked stale because it has been open for 60 days with no activity.

[justaugustus]

Recent release of dependency-review-action supports this, but there may be edge case for projects not in our API dataset.

@jeffmendoza (paraphrased):

It should be seen as a success when GitHub adds a feature that Scorecard/Allstar previously developed and added to the release notes (when we deprecate this code path).

@spencerschrock proposed the deprecation, so I'm assigning this to him.

Note that we also need to check on whether the https://github.com/ossf/scorecard-dependencyanalysis should be archived.