Get and expose npm + GitHub data for dependency packages

[humphd]

In #2797 we are adding a dependency service, and it currently only includes package names. Let's add a way to expose both npm and GitHub data for a package.

For npm, we can query the npmjs registry programmatically. For example, you can get the package info for React using https://registry.npmjs.org/react. This will return everything we need in JSON format.

Here are some docs:

• https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md
• https://www.edoardoscibona.com/exploring-the-npm-registry-api
• https://www.npmjs.com/package/query-registry
• https://www.jsdocs.io/package/query-registry

The npm data for a package will also give us GitHub URLs that we can use to query the GitHub API via Octokit.

I don't know if there is any point in doing this for every package, or only on-demand when requested. Maybe at first, we can expose this data via a REST endpoint in the dependency service: GET /projects/:name.

We should cache this data in Redis (available via Satellite), and then we can re-use it for future requests for the same data.

Down the road, once this working, we can decide how to use this data (e.g., find issues to work on).

[JerryHue]

I think we can go with a mix of both.

GitHub data is going to be very dynamic. There are some projects that constantly get new issues, as well as close some issues, and so on.

On the other hand, the npm data is not that dynamic. Unless there are some extreme circumstances (the package gets completely removed from npm), the npm package metadata is going to be the same for a long time.

So I would plan it out like this:

During initialization stage, we get the package names from deps.txt, pull the data from the npm on mass (all async to not block the service). Hook some callbacks to pull npm data or GitHub data (usually on a request for a package).

These callbacks will get in charge of fetching the data and cache it, as well as make sure we are not requesting too many times to npm or GitHub (gonna have to look deeper into the GitHub API if we can do something like this).

[humphd]

I agree, the data is both stable and dynamic, and our fetching and data backend can reflect that.

NOTE: you're describing something very similar to how our RSS feed parser works. We have a queue in Redis managed via JS with BullMQ. It lets us add jobs, and process them over a long period of time using worker processes.

You could easily add another queue to process dependencies/github projects. We'd enqueue the project, and over time, process (or re-process) it and cache or store the data somehow.

Some of this could be done in the parser service itself, and you could consume the results in your dependency service.

cc'ing @TueeNguyen, who is working on the parser service.