-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Branch-Protection: Review/remove scoring based on Tiers #3123
Comments
It seemed like there was a reason when this leveled scoring was introduced in #1287. Let me check with @laurentsimon regarding any consumers of these tiers. |
the tiers were introduced because without them, scores were hard to disambiguate. A score of X could mean you do no code review and use pre-submit; or the other way around, which is different from a security's standpoint. The tiers were designed from a security point of view.
They have a title pre-submit https://github.com/firebase/flutterfire/blob/master/.github/workflows/pr_title.yaml. They can make this one a status check and get the points for it. They don't need all the pre-submit to be present. One should suffice to pass
This one is a bit more unfortunate. Note that |
So after structured results land, removing the tiers could be a possibility? |
I suppose it could. Depends if some users still want to use scores for project comparisons. |
Yes that makes sense, it should be easy for them to get the whole score for Tier 2. But I don't like the idea of requiring a exact predetermined posture or behaviour in order to give a good Scorecard score. I'd rather see Scorecard as a tool that is able to evaluate the security of a project independently of some specific choices. In other words, I don't think a maintainer should need to know the Scorecard rules to get a good score -- if their project follows good security measures, it should receive a good score already. Requiring 2 reviewers is a very strong security measure per se (and also a huge time and resource investment), I believe Scorecard should value and recognize this effort independent of other requirements.
Yeah... Unfortunately I couldn't find cases like this 😢 , even though I'd love to believe this would be the case of solo-maintainers with huge care on security and/or best practices. EDIT: I think it's may be the case of github.com/nghttp2/nghttp2. Although I couldn't confirm 100%, the maintainer is the only committer and he keeps opening PRs to merge their commits, without approval of anyone else. |
With #3354 merged, I think the tier system (or having admin enforced in tier 1) is going to limit most repo rule scenarios to 2 points for tier 1. |
This issue is stale because it has been open for 60 days with no activity. |
Is your feature request related to a problem? Please describe.
Currently, the Branch-Protection check calculates the scores based on Tiers. That can end up causing frustration, because if a repo is placed in an inferior Tier -- limited by one specific requirement of its next tier --, it gets no score rewards if it completes a requirement of a higher tier. I can give two practical examples:
Status checks defined
rule. However, the repository uses the Branch Protection rule ofrequired reviewers >= 2
, which is a rule with strong security impact, and gets no score reward for that rule, because the rule is on Tier 4. See below the Scorecard Branch-Protection evaluation for this example:required reviewers >= 1
(a perfectly understandable use-case is a project of a solo-maintainer), it will be locked on the Tier 1 -- earning 3/10 -- and will not receive any score reward if it implements theStatus checks defined
, which is also a rule with strong security impact and it's the best a solo-maintainer can do as Branch-Protection effort.Describe the solution you'd like
Further discussion might be required for the best solution to this, but an initial purpose is to remove the idea of Tiers and define that each Branch Protection rule have an independent value, and your score will be the sum of the scores of the rules you comply.
As an example, for an analysis without using the administrative access token, we could evaluate the score according to a table as this:
With this configurations, the examples I gave above would have the following score changes:
Required reviewers >= 2
rule would be considered.Status checks defined
rule would be considered.The text was updated successfully, but these errors were encountered: