Build a better picture of our dependency graph

[humphd]

One of the 3.0 goals is to start being better stewards of the ecosystems we depend upon. Telescope has lots of its own code, but if you consider all of the things we use (npm modules, docker images, command line tools, libraries, frameworks, etc.), it's massive. However, we have very little visibility into it. I'd like to change that.

Students often want help finding projects to work on in OSD600/DPS909, and it would be good if we could point at a giant (dynamic) list of Issues marked bug, good first issue, help wanted`, etc.

In addition, I'd like to see us contributing fixes and features to some of these upstream projects in 2022, and helping to maintain the open source we need.

Let's begin to build this graph. At first, it could be a manual thing we create. Later, once we understand it a bit more, we can try to automate it.

Google has a neat tool for helping to find and visualize your dependencies: https://deps.dev. Here is Satellite for example: https://deps.dev/npm/%40senecacdot%2Fsatellite.

[menghif]

The "Insights" tab also provides a list of GitHub links to all the dependencies we use.
https://github.com/Seneca-CDOT/telescope/network/dependencies

[humphd]

I really want to do this in an automated way.

Because we're now using pnpm we can use the list command to help us:

$ pnpm list --recursive --depth 2 --parseable

/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
/Users/humphd/repos/telescope/tools/autodeployment/node_modules/.pnpm/[email protected]
...

This gives us our dependencies, and their dependencies, up to a depth of 2 (we can go deeper if we want). With a bit of regex parsing, we can extract the package names, and then use that to query the GitHub API for bugs we could fix.

We could probably extend https://github.com/mjswensen/shoulders to do this for pnpm, since it already does most of what we need.

[JerryHue]

I am aware that this might be a little bit too early to ask about this question, but I am wondering how the end application is going to be like.

Do we want a CLI like shoulders, which you invoke at the root of your project and extract the information from the package.json file? Or do we want a CLI that accepts a link to a GitHub repository?

[humphd]

I think that since Telescope is a web service and web app, we should think in those terms. Imagine a little microservice that does periodic querying of our dependencies and gets GitHub data, caches it in Redis, then uses that to create some dashboard views that users can look through.

A cool first prototype might be a list of Issues labelled "good first issue" in any of our dependencies, or their dependencies, that you can scroll though, and click to go see them on GitHub.

[JerryHue]

I think that since Telescope is a web service and web app, we should think in those terms. Imagine a little microservice that does periodic querying of our dependencies and gets GitHub data, caches it in Redis, then uses that to create some dashboard views that users can look through.

This sounds promising! I will try to see if I can come up with anything during the weekend. If anybody else is up for this prototype, you may contact me. I'd love to discuss it further.

[TDDR]

Related discussion #2666