🛠️ Neighborhood: Keep track of code coverage as we review and merge code

[anaulin]

• 🥔⛰️Neighborhoods #892

As Convene becomes a More Serious Product, it would be nice to have some automated tooling to help us keep our testing up-to-date.

Some options:

• add SimpleCov and use the SimpleCov+ GitHub Action (https://github.com/marketplace/actions/simplecov-action), to check for a minimum code coverage on PRs
• integrate a 3rd party service like https://codecov.io/ or https://codeclimate.com/ (both these examples are free for open source projects)

Thoughts?

[zspencer]

I think this is something that would be super exciting! I would love to combine it with a piece of furniture that kept the history, rather than requiring us to use an outside service.

[jalcine]

https://github.com/Librecov/librecov could be an option for furniture. It does look a little unmaintained , assuming from Librecov/librecov#203. I have experience with Elixir projects, so I can try seeing if I can integrate this into Convene over the next week.

If not, this does feel like a (big) project in itself that the general developer community interested in self-hosting would be interested in seeing a solution for.

(Originally published at: https://jacky.wtf/2023/3/HhHK)

[zspencer]

Yea, I could see us offering a Hosted Utility for LibreCov , but that seems a lot heavier weight than a tiny piece of Furniture that:

1. Listens for a PUT of SimpleCov results
2. Parses those for high level metrics (i.e. x Lines of Implementation Code, y % covered by z lines of Test Code)
3. Persists the high-level metrics and full results on a commit-sha basis

If we were to go with a Hosted Utility, I'd like to have a few answers to questions like:

• Is the Librecov Hosted Utility a Neighborhood resource? Or Space? If Space, do we require it to be ran on a VM provider maintained by the Space Owner? If Neighborhod, is a librecov installation sufficiently useful to re-inflate our "Backend" VM?
• How do we make sure our Neighborhood-level Librecov installation stays well-patched and doesn't serve as an entrance point for a security vulnerability which could leak Space-owners IP?
• If it's a Space-level installation, what are the hosting costs on a per-month basis for the Space? At present, we are shooting for ~100% markup over server costs; so if it's $5/mo in compute to run a secure LibreCov install, we'll need to charge $10/mo per Space with a LibreCov add-on installed.

I think a project like abra by CoopCloud is well suited for holding the code for provisioning and updating a LibreCov deployment.

TL/DR: Let's start with getting SimpleCov working on Actions, then we can think about how we want to report/store them. Small, safe changes > big jumps