BUG Signed-Releases: internal error: too many releases, please report this

[cpswan]

Describe the bug

Running v5.0.0-rc1

Signed-Releases check reports internal error: too many releases, please report this

Reproduction steps
Steps to reproduce the behavior:

1. Downloaded scorecard_5.0.0-rc1_linux_amd64.tar.gz
2. Untarred the binary and copied it to ~/.local/bin/scorecard so it's on my path
3. Ran scorecard --repo github.com/cpswan/release_automation

Expected behavior

I get a score for my signed releases.

Additional context

The repo I was testing against presently has 31 releases. I was previously signing with sigstore, but I just added a workflow to add SLSA provenance instead, so the releases now have a mixture of .sigstore and multiple.intoto.jsonl. My first guess is that this might cause the problem.

Testing against a repo that only has (some) .sigstore releases seems to work fine (e.g. scorecard --repo github.com/atsign-foundation/noports) NB that repo has more releases than the one that's failing, so it's not simply that 31 is too many.

Testing against this repo, which has just multiple.intoto.jsonl is also fine

[spencerschrock]

Thanks for trying out v5.0.0-rc1 and reporting this issue!

It didn't have to do with the specific artifacts, but rather which release had artifacts and which didn't. We limit our analysis to the last 5 releases, but there was an order of operation bug which was skipping this, and your test repo met the bug criteria.

The 6th most recent release had no release artifacts, so we went on to the 7th (and beyond)