BUG: Running checks with --local option

[gk4delltech]

Describe the bug
The "--local" option does not work. I must explicitly name very few checks to get it to run, then I get a score of 10 because nothing is checked.

Reproduction steps
Steps to reproduce the behavior:

1. See the attached scorecard2.sh shell script and error log.

Expected behavior
Expecting "--local" to automatically exclude checks that are GitHub specific. Expecting scorecard to run on an extracted source code tarball like those referenced in the Yocto Project which builds embedded Linux.

Additional context
Add any other context about the problem here.
scorecard2.sh.txt
scorecard3.sh.txt
hello-tgz-scorecard.txt
hello-github-scorecard.txt

[spencerschrock]

Expecting "--local" to automatically exclude checks that are GitHub specific.

So it does this already if you just do --local without specifying --checks, it will run all of the checks that support local, which I think is what you want.

[gk4delltech]

It bombs out for me with "internal error".

• scorecard --local hello-2.12
  Starting [Token-Permissions]
  Starting [Dangerous-Workflow]
  Starting [Vulnerabilities]
  Starting [Pinned-Dependencies]
  Starting [Binary-Artifacts]
  Starting [Dependency-Update-Tool]
  Finished [Binary-Artifacts]
  Finished [Dependency-Update-Tool]
  Finished [Token-Permissions]
  Finished [Dangerous-Workflow]
  Finished [Vulnerabilities]
  Finished [Pinned-Dependencies]

RESULTS

Aggregate score: 10.0 / 10

Check scores:
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| SCORE | NAME | REASON | DOCUMENTATION/REMEDIATION
|
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| 10 / 10 | Binary-Artifacts | no binaries found in the repo | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#binary-artifacts |
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| ? | Dangerous-Workflow | no workflows found | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#dangerous-workflow |
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| ? | Dependency-Update-Tool | internal error: Search: | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#dependency-update-tool |
| | | unsupported feature |
|
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| ? | Pinned-Dependencies | no dependencies found | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#pinned-dependencies |
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| ? | Token-Permissions | no github tokens found | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#token-permissions |
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| ? | Vulnerabilities | internal error: | https://github.com/ossf/scorecard/blob/49c0eed3a423f00c872b5c3c9f1bbca9e8aae799/docs/checks.md#vulnerabilities |
| | | vulnerabilitiesClient.ListUnfixedVulnerabilities: |
|
| | | osvscanner.DoScan: no packages found in scan |
|
|---------|------------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
Error: check runtime error: Dependency-Update-Tool: internal error: Search: unsupported feature
2024/01/17 17:55:38 error during command execution: check runtime error: Dependency-Update-Tool: internal error: Search: unsupported feature

[spencerschrock]

It bombs out for me with "internal error".

This was fixed by #3756, but we don't have a release yet with that fix sorry. You can always install from source.

go install github.com/ossf/scorecard/v4@latest

[gk4delltech]

Okay, you say that #3756 is not in the v4.13.1 release, but in the latest via "go install". I've just tried that and I still get an "internal error" . Please see the attached.

scorecard4.log.txt
scorecard4.sh.txt

[spencerschrock]

Okay, you say that #3756 is not in the v4.13.1 release, but in the latest via "go install". I've just tried that and I still get an "internal error" . Please see the attached.

scorecard4.log.txt scorecard4.sh.txt

My mistake, it appears @latest grabs the last release, which is still v4.13.1.

Apparently @HEAD is what you want, although it may be cutting edge and change day to day. So you could pick the current commit (4a2dfa9), and pin it to that.

# HEAD may be unstable
# go install github.com/ossf/scorecard/v4@HEAD
go install github.com/ossf/scorecard/v4@4a2dfa96be97d63da84b230c44610ca2f7178e27

[gk4delltech]

I tried with commit @4a2dfa9... and @Head, but I still get an "internal error". Please see the attached scorecard5 files.

scorecard5.log.txt
scorecard5.sh.txt

[spencerschrock]

I tried with commit @4a2dfa9... and @Head, but I still get an "internal error". Please see the attached scorecard5 files.

scorecard5.log.txt scorecard5.sh.txt

This is a different error actually. This time it's in the Vulnerabilities check, not the Dependency-Update-Tool check.

[spencerschrock]

This new error is something we need to fix on our side. And once that gets patched, you should be able to update the hash and the error would be gone.

[spencerschrock]

These should both be fixed if you update your pin to the latest commit b556d93:

go install github.com/ossf/scorecard/v4@b556d932a49cc4004ed15c0e2333b8ac2fac252e

Feel free to re-open if that's not the case.

[gk4delltech]

Okay, no errors with b556d93. When will this be added to a release?

[spencerschrock]

When will this be added to a release?

We are in the middle of prepping for a major release, so I would need to see which changes have been merged already. And see if its something we can put out a patch release for without breaking things.