DISCUSSION: v4 milestone

[laurentsimon]

To start thinking of our next step towards v4 release, let's write some ideas in this issue. We're thinking of v4 release for EOY 2021.

We can talk about them during next scorecard meeting, create issues and assign them to contributors, and then have them as milestones. Here a list to start with:

• Scorecard scaled to 1M+ repos (P1)
• Scorecard adoption (could be showcasing how to use scorecard to vet dependencies @naveensrinivasan)
• Scorecard CI/CD action finished (market place and verified?)
• Allstar policy for scorecard (joint with allstar)
• Scorecard badges
• Scorecard E2E tests
• Stackdriver cost optimization
• GitHub API rate limits
• Scorecard contributing + communication process
• SLSA compliance checks, verification (doc, 1-2 checks)

Please add what you think is worth discussing about. This will help for selection and prioritization.

Thanks everyone!

[asraa]

Random thought that might be nice to do for another release (not sure when v4 is on the timeline): scorecard as in-toto attestation? would mean defining a scorecard predicate and having an output format for that.

Higher levels could sign the envelope (e.g. a github-runner could run scorecard and then sign the in-toto proving that it ran scorecard.

[laurentsimon]

Random thought that might be nice to do for another release (not sure when v4 is on the timeline): scorecard as in-toto attestation? would mean defining a scorecard predicate and having an output format for that.

Higher levels could sign the envelope (e.g. a github-runner could run scorecard and then sign the in-toto proving that it ran scorecard.

Thanks for the suggestion, @asraa!
We were thinking v4 release EOY.

[naveensrinivasan]

@laurentsimon Thanks, Probably add this as a milestone once we have consensus?

[azeemshaikh38]

To expand on Laurent's comment, we are looking for contributors interested in owning some these KRs end-to-end.

• Scorecard E2E tests

(i) enable e2e tests on ossf-test repos #861
(ii) generalize e2e tests to run on RepoClient interface - #1113 (comment)

Also, the below KRs either require community inputs or a general helping hand:

• Scorecard adoption (could be showcasing how to use scorecard to vet dependencies @naveensrinivasan)
• Scorecard contributing + communication process

• Scorecard config design doc - https://docs.google.com/document/d/19PzTxjVL9iR6tEkCz_wPX2-xLJQtWzPSDxG8GZPu0Ao/edit?usp=sharing&resourcekey=0-y_cKeath-Yq9TBTtKy7pZA

Finally, items to help reduce technical debt. Not part of Milestone V4, more like ongoing KRs which help improve code quality:

• Automated SemVer releases - Feature: automated semver releases #982
• Reduce number of nolint - Reduce number of nolint comments in code  #962
• Add more unit tests - Feature: improve unit tests and e2e test for better coverage #435
• File parsing errors - BUG: Parsing errors #839

@naveensrinivasan @chrismcgehee @david-a-wheeler FYI. Let us know if you would like to see anything else added here.

[laurentsimon]

@naveensrinivasan also proposed doing a scorecard demo/blog post on using scorecard to vet dependencies automatically.

[naveensrinivasan]

@naveensrinivasan also proposed doing a scorecard demo/blog post on using scorecard to vet dependencies automatically.

More understand the state of dependencies with scorecard data

[laurentsimon]

automatic documentation generation #898

[laurentsimon]

GitHub action issue #193. v4 milestone added.

[laurentsimon]

This issue #426 is an important one, especially the pull_request_target trigger.

[laurentsimon]

Adding lines/filenames to our results #725 is an important issue we should tackle for v4 since it improves the UX experience in the GitHub scanning dashboard

[laurentsimon]

@oliverchang will tackle #1148. Thanks Oliver!

[laurentsimon]

FYI, Asra @asraa will tackle part of #426. Thanks you Asra!

[laurentsimon]

This is also useful #435 (comment)

[laurentsimon]

Would love to have dangerous workflows in v4 #1168, if possible
Long-term, I think we'll merge Token-Permissions into it, as well as the GH workflow pinning that currently lives under pinned dependencies.

[azeemshaikh38]

We seem to have a lot more v4 issues than initially discussed. Do we have the time commitment to complete all these extra items? Please note that we are aiming for a mid-Jan timeframe for a v4 release. And with a winter break, it does not give us a lot of time. @laurentsimon @naveensrinivasan

Extra issues I noted: #1174, #1196, #1038, #1260, #1270, #1275, #1238.

[laurentsimon]

I've removed the first one. The others are best effort. Many are simple enough that it's doable, and they improve the checks: I think it's god if we can fix those small issues before releasing. The ignore list for binary artifacts would be great, so I added in case I have time.

[laurentsimon]

Aiming for a release mid-January 2022. What's left:

1. verify GH action for market place (WIP)
2. documentation of the action
3. submit starter workflows (WIP)
4. Test end to end flow of installation
5. Install action on a few repos OSSF owns and others
6. Release notes
7. Blog post prep

We have a v1 milestone on the action repo https://github.com/ossf/scorecard-action/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1

[georgettica]

hey! I found this v4 when testing locally

docker run -v ${PWD}:/app -e SCORECARD_V4=y -e GITHUB_AUTH_TOKEN=token gcr.io/openssf/scorecard:stable --show-details --local=/app

makes it much faster to make my codebase comply with scorecard

[laurentsimon]

note that when running on a local repo --local=, not all the checks are run. Checks that are run are those that do not use GitHub APIs. Supported checks are indicated thru a repos: local in this file https://github.com/ossf/scorecard/blob/main/docs/checks/internal/checks.yaml#L47

FYI, this https://github.com/ossf/scorecard/pull/1405/files will remove the need for SCORECARD_V4 once merged.

[laurentsimon]

Closing since v4 is out.