✨ Added additional github repositories in projects.csv

[Parth59]

What kind of change does this PR introduce?

(Is it a bug fix, feature, docs update, something else?)
This can be considered a feature. This PR will allow the generation of scorecard metrics for additional repositories that we consider vital to the PyPi and NPM ecosystem.

What is the current behavior?

Currently, there are around 1M GitHub repos. After this PR is approved additional 250k repos will be added.

What is the new behavior (if this is a feature change)?**

Added additional repositories. The additional repositories are from packages from the NPM and PyPI ecosystem.

• Tests for the changes have been added (for bug fixes/features)
  No need for tests as no code change is involved.

Which issue(s) this PR fixes

A Request to bump up the repositories analyzed. No issue was created.

Special notes for your reviewer

Please note that some of the Github repos that are added might encounter 404 due to the repository being deleted by the developer/maintainer. While we have taken utmost care to filter them out, there might still be some that might have gone under the radar.

Does this PR introduce a user-facing change?

NONE

[laurentsimon]

qq: which additional repos did you add? Are the entires sorted?

[azeemshaikh38]

2 changes:

1. Run make add-projects which will de-duplicate the projects and sort them alphabetically.
2. Update the PR title to add an appropriate prefix - https://github.com/ossf/scorecard/blob/main/CONTRIBUTING.md#pr-process

[azeemshaikh38]

qq: which additional repos did you add? Are the entires sorted?

Yeah GH UI does not show the diff since it's too large. I did a local diff to see it for myself.

[github-actions]

Integration tests success for
[8e4d3f4]
(https://github.com/ossf/scorecard/actions/runs/2272396272)

[codecov]

Codecov Report

Merging #1886 (376dfcf) into main (72086c9) will increase coverage by 3.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1886      +/-   ##
==========================================
+ Coverage   51.35%   54.40%   +3.04%     
==========================================
  Files          79       79              
  Lines        6729     6729              
==========================================
+ Hits         3456     3661     +205     
+ Misses       3043     2832     -211     
- Partials      230      236       +6     

[azeemshaikh38]

Also, I noticed that these dependencies have num_dependencies_deps.dev metadata field. Please remove that. We get that metadata from deps.dev API directly.

[Parth59]

Hi Team,
I am facing errors while running make add-projects. Can anyone let me know the process along with commands or maybe if it is already documented can someone please redirect me to the corresponding location?

Error log when running make add-projects

# Add new projects to ./cron/data/projects.csv
./cron/data/add/add ./cron/data/projects.csv ./cron/data/projects.new.csv
panic: iter.Next: invalid GitHub URL: error during parse: invalid repo flag: . Exepted full repository url

goroutine 1 [running]:
main.main()
github.com/ossf/scorecard/v4/cron/data/add/main.go:49 +0x190
make: *** [add-projects] Error 2

It occurs even when I add the following GitHub repo list - https://drive.google.com/file/d/1vF2r6onpc4lUKVJobgBNixW0yuuE-BZ3/view?usp=sharing

[azeemshaikh38]

Can anyone let me know the process along with commands or maybe if it is already documented can someone please redirect me to the corresponding location?

So the error will tell you what's going on. For example, when I run this command on your PR locally, I get the error: invalid repo flag: github.comments. Which means there is a repository with URL github.comments in your data. This is an invalid URL which you either need to delete or fix.

Note that for a GitHub URL to be considered valid, it should be of the form: github.com/<org>/<repo>. Anything else will fail.

[Parth59]

Hi Team,
I have made the changes that were required. Requesting you to kindly review and lemme know if more changes are needed.
Also has the tool a provision for handling cases when a repo has been deleted by the maintainer/developer and GitHub gives a 404 error on the repo page?
Thanks

[github-actions]

Integration tests success for
[cdc9e99]
(https://github.com/ossf/scorecard/actions/runs/2281991206)

[github-actions]

Integration tests success for
[d627b80]
(https://github.com/ossf/scorecard/actions/runs/2281983264)

[github-actions]

Integration tests success for
[376dfcf]
(https://github.com/ossf/scorecard/actions/runs/2282040527)

[github-actions]

Integration tests success for
[2fb39cc]
(https://github.com/ossf/scorecard/actions/runs/2282034481)

[azeemshaikh38]

LGTM, thanks! Excited to see the results from your research.

[azeemshaikh38]

Hi Team, I have made the changes that were required. Requesting you to kindly review and lemme know if more changes are needed. Also has the tool a provision for handling cases when a repo has been deleted by the maintainer/developer and GitHub gives a 404 error on the repo page? Thanks

The tool does not look for 404 errors on the repo page. If there is a 404 error during Scorecard calculation we simply skip it so no data will be populated.