Public CI metaissue

[nomeata]

This issue tracks various refinements we want to do to our CI infrastructure post-open source.

• Post-open-source CI proposal #2544 Pick a public CI system (Github actions + cachix for now)
• Nix: no drun by default, and build with Github Actions #1224 Set it up
• CI: Create Github releases from CI #2566 Automatically create releases and publish build artifacts there
• CI/Buildsystem: drun for all #2567 Allow public access to drun, so that all tests can be run on CI
  This blocks most of the following items:
• CI: Publish reports #2568 Publish reports (profiling, documentation preview) from GitHub Action runs
• CI: Create performance delta comment via Github Action #2740 Report performance changes in comment to PR (maybe using this Github Action)
• CI.md: Mention branch protection #2744 Switch branch protection to Github Action rather than hydra
• CI: Remove hydra files #2771 Stop using the internal CI system.

[nomeata]

Useful article about pull_request and pull_request_target, and the security implications of PRs from other repositories: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

[nomeata]

Discussing nix-based CI options on the Nix Discourse: https://discourse.nixos.org/t/github-actions-and-or-with-hercules-ci/13537

[osa1]

"CI: Publish reports" this is not something we do today, right?

I'm thinking what's left to do before stopping using Hydra. I think once we port posting perf and size changes to GH CI we should be able ditch it?

[nomeata]

"CI: Publish reports" this is not something we do today, right?

Yes, we have direct links to the latest flame graph, documentation, etc. for each PR. Would be a shame to lose that, but maybe ok temporarily until we have a good idea. I should research what kind of services are there out there.

Besides that it's just the perf changes comments, which I can look into this week maybe

[osa1]

Yes, we have direct links to the latest flame graph, documentation, etc. for each PR.

Interesting, I didn't know that. Where can I see these reports?

[nomeata]

You didn't know? Anyways, see #2562 for some of the links, which we removed from the README to keep up appearances.

[nomeata]

Flame graphs at https://hydra.dfinity.systems/job/dfinity-ci-build/motoko/tests.profiling-graphs/latest/download/1/index.html I assume (I can't check, of course). One can construct similar urls for each PR. I hope to preserve that feature, or improve it (easier discoverability in particular of course). But needs not block turning off hydra.

[nomeata]

Maybe we can sign up to coveralls.io (free for open repos) and upload coverage data there. Supports OCaml (https://github.com/aantron/bisect_ppx#Coveralls). We used to have coverage reports, but they have somewhat bitrotted.

[nomeata]

We might very easily be able to speed up our CI system by hooking up to (and paying for) https://nixbuild.net/, with no noticable changes to our existing workflows.

Would it be possible to get signed up for this? I'll then set it up.

Oh, no Mac support it seems. Maybe not then.