Suggestion: Change Branch-Protection check to consider rule of "change only through PRs"

[diogoteles08]

Is your feature request related to a problem? Please describe.

The current Branch Protection check does not consider the rule named Require a pull request before merging (you can check it on the GitHub page). This rule is actually very important because it forces any change on the branch to be made through PRs, and would therefore prevent malicious pushes -- that could even be made through actions with write permissions.

The closest to this we currently have is the requirement of Reviewers >= 1 to get a 6/10 score. But this might be specially difficult to solo-devs, while the Require a pull request before merging is possible even for a single maintainer.

Describe the solution you'd like

The purpose is to make this rule enhance the Branch-Protection rule somehow. My suggestion would be to add this rule as a new tier with a 5/10 punctuation. This would give a score higher than the 3/10, which basically requires prevention of force pushes and branch deletion, and slightly lower than 6/10, which requires a reviewer other than the commiter.

Describe alternatives you've considered

• The rule could be added as a new requirement to get a 3/10 grade. But this might be frustrating to the projects that already had the effort to get this grade, as they could be led to 0/10.

[diogoteles08]

Is there any opposition to this? If not, feel free to assign me to it

[spencerschrock]

Overall, I could see this being some sort of partial credit. I've left some comments inline.

Is your feature request related to a problem? Please describe.

The current Branch Protection check does not consider the rule named Require a pull request before merging (you can check it on the GitHub page). This rule is actually very important because it forces any change on the branch to be made through PRs, and would therefore prevent malicious pushes -- that could even be made through actions with write permissions.

The closest to this we currently have is the requirement of Reviewers >= 1 to get a 6/10 score. But this might be specially difficult to solo-devs, while the Require a pull request before merging is possible even for a single maintainer.

I think "Require branch to pass at least 1 status check before merging" also captures this to some extent. Of course this has the issue of tiers (which you raised in #3123).
Also some of the admin level tier 2 points indirectly hit on this, which might become more visible after repo rules are supported (#3326).

Describe the solution you'd like

The purpose is to make this rule enhance the Branch-Protection rule somehow. My suggestion would be to add this rule as a new tier with a 5/10 punctuation. This would give a score higher than the 3/10, which basically requires prevention of force pushes and branch deletion, and slightly lower than 6/10, which requires a reviewer other than the commiter.

I think the code review is a much more important bit, but I could see requiring a PR to possibly be worth a single point.

Describe alternatives you've considered

• The rule could be added as a new requirement to get a 3/10 grade. But this might be frustrating to the projects that already had the effort to get this grade, as they could be led to 0/10.

This could also be part of the tier 1 if #3123 is changed.

[diogoteles08]

One important detail that I discovered:
By my testings, the Require a pull request before merging seems only to be retrievable when using an admin token :/
I concluded this because running Scorecard in a repository with this option disable, using an Admin token, I get

BranchProtectionRule.RequiredPullRequestReviews = {
    RequiredApprovingReviewCount: *int32 nil,
    DismissStaleReviews: *false,
    RequireCodeOwnerReviews: *false
}

, but if I run without the admin token, for the exact same repository without changing the configs, I get

BranchProtectionRule.RequiredPullRequestReviews = {
    RequiredApprovingReviewCount: *int32 0,
    DismissStaleReviews: *false,
    RequireCodeOwnerReviews: *false
}

, which is the exact same result that we get when the repository requires PRs, but doesn't require a reviewer. This way, we could not differentiate a repository that does not requires PRs to a repository that requires PRs but doesn't require another reviewer.

Obs: those values were gotten inside the run of Scorecard as values available on the evaluation part of the run. This could be fact-checked by looking into the complete response of Github's API.

EDIT: I was able to verify directly using the GitHub's API and the problem still applies. To test, I used the GraphQL explorer and ran the following query:

{
  repository(owner: "diogoteles08", name: "testing-branch-protections") {
    defaultBranchRef {
      name
      refUpdateRule {
        requiredApprovingReviewCount
      }
    }
  }
}

In any ways, I think this feature is still important, especially thinking that after repo rules support (#3326) all branch protection configuration should be public.

[diogoteles08]

After private discussions with @spencerschrock , I've came up with this idea on how to add this rule on the current Branch Protection score:

We'd add this as another Tier-2 requirement, but as the "Require 1 reviewer" has a stronger security impact, it'll have a bigger weight over the other Tier-2 requirements.

On the current implementation of this check we compute the score of each tier using the concept of a total point per tier (which currently is the number of requirements for the tier). Then we keep adding points as the branch matches the rules (i.e. + 1 for having 1 reviewer and + 1 for requiring review of most recent push) and then make the score for the tier as the summed points / total point per tier * maximum score for this tier. The latter is always 3 for the Tier 2, as it's the score added on the Branch Protection check when you fully comply with the Tier-2 requirements

Currently the Tier 2 requirements are like this:

"total point per tier" = 3 if run with admin, otherwise "total point per tier" = 1

+1 point for Require at least 1 reviewer for approval before merging
+1 point for Require branch to be up to date before merging (admin only)
+1 point for Require approval of the most recent reviewable push (admin only)

My idea of implementation would change this to:

"total point per tier" = 5 if run with admin, otherwise "total point per tier" = 2

+1 point for Require a pull request before merging (admin only)
+2 point for Require at least 1 reviewer for approval before merging
+1 point for Require branch to be up to date before merging (admin only)
+1 point for Require approval of the most recent reviewable push (admin only)

Analysis on use cases

Running without admin permissions:

• Someone using scorecard without admin permissions would only need "Require at least 1 reviewer for approval before merging" to complete Tier 2 requirements. The same occurred before this PR.

Running with admin permissions (or after Scorecard support for Rule Set #3326):

• A solo-maintainer could match all Tier 2 requirements except "Require at least 1 reviewer for approval before merging" and he would get 3/5 * 3 = 1.8 as the Tier 2 score. Which means that he could get a maximum score of 4.8/10 on Branch protection
• A non-solo-maintainer that complies "Require at least 1 reviewer for approval before merging", would automatically comply with "Require a pull request before merging". So if they for any reason don't want to comply with the other Tier 2 requirements, they'd already have a 3/5 * 3 = 1.8 as the Tier 2 score -- leading to a 4.8/10 score on Branch Protection

[github-actions]

This issue is stale because it has been open for 60 days with no activity.