create a self-hosted builder variant for running gitea/github actions runners

[bsherman]

For running a Github or Gitea Actions self-hosted runner one needs a bunch of tools used by the runners.

Github has self-hosted runners, though they don't seem to elaborate on what is required on the hosts in their docs... But they do have a public repo which shows how they build their images for runner VMs.

In short, Github runners run workflows on a VM, not in a container.

In the case of Gitea Actions, they use a fork of act which runs your workflows in a container, since for running locally, you need a way to get all the tools and probably don't want to install all the things on your workstation.
Act also provides some good docs regarding options for runner images

Gitea doesn't use act though, they have a soft fork act_runner, and their docs show that while containers are the default for running workflows, they CAN run directly on a host (baremetal or VM).

Why does this matter, runners on container vs host? For the vast majority of use cases, it won't. If you are building node, python, Go, whatever... it's all good either way. In fact, you may prefer using a container for convenience or security reasons.

However, in my personal testing, I used a workflow based on some of our ublue workflows, and these use redhat-actions/buildah-build and redhat-actions/push-to-registry.

I very quickly hit a problem:

Error during unshare(CLONE_NEWUSER): Operation not permitted
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 

Turns out this is not a new issue, and... probably isn't going to change since it's by design: containers/buildah#1901

Options are:

1. do not use buildah (instead using a docker or kaniko action) ... so sad...
2. build a custom ucore-buildah builder/runner image specifically for powering self-hosted Actions runners ... so fun!

[bsherman]

I received some feedback about the potential security issues of this approach.

The concern is that while Github's hosted runners use VMs, they are ephemeral VMs, so a bad actor can't meaningfully compromise the system or poison it for a subsequent workflow run.

This is a very important point, and Gitea Actions, by default, addresses this through the use of containers for their workflow runs.

That said, while Github recommends auto-scaling runners when possible, they also acknowledge the probably need for persistent runners and the inherent security concern:

Warning: We recommend that you only use self-hosted runners with private repositories. This is because forks of your public repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow.

• from Adding self-hosted runners

I'd love to see a future where we can build uCore qcow2, ova, and cloud images, like upstream does, but using our ostree container as the source. This would make it perfectly viable to use what I wish to build here as an ephemeral runner image. We don't have that today, but I hope it's in our future.

So, my thoughts on this concern echo Github's warning: ONLY use a persistent self-hosted runner for private repositories, not public.

With that safeguard, I believe further development of this idea is reasonable.

[castrojo]

Yeah this is one of the reasons I've been looking at actuated.dev, they use firecracker to keep things isolated but it's your bare metal.

[bsherman]

Actuated.dev looks pretty cool, but I don't see any support for Gitea only GitHub.

[bsherman]

Read more about actuated.dev ... and yes, one of their big goals is to provide "self-hosted" (not Github hosted) runners which are secure for use even in public repos.

I still think it's viable to use a persistent ucore based runner VM for private only use.

I definitely like the idea of using firecracker VMs for this, but as I mentioned above, we need the ability to build "cloud images" from an ostree container image before we could use ucore images in a VM like that.
Once that becomes a reality, I would maybe be interested in figuring out how to use Gitea Actions runner to use ephemeral VMs. Or if actuated.dev has some partial open source stuff that can be worked with.

[bsherman]

I've been doing some tinkering on this and ran into an annoying bug:

I have some test OCI images with act_runner running on "host" of a VM and I was testing redhat's buildah-build ...

I kept running into this error:

::error::Error: buildah, and therefore this action, only works on Linux. Please use a Linux runner.

Turns out it's this line:

https://github.com/redhat-actions/buildah-build/blob/b4dc19b4ba891854660ab1f88a097d45aa158f76/src/index.ts#L17

the env var RUNNER_OS in act_runner is "linux" not "Linux"

We should probably PR a fix.

[bsherman]

I ran into several issues with this which seemed a bit silly given other alternatives which exist (github, other available builders, and simply using act runner to run builds in docker), plus the already expressed potential security concerns. I decided not to pursue this further.