Self-hosted runners on GitHub Actions loses connection before job completion

[raisedadead]

We want to build and deploy news builds with GitHub Actions and self-hosted runners. This is so that we can guarantee that runners are available and we do not wait on GitHub to provision free runners, which we use for other things like CI tests, etc.

We have implemented such a setup, with our localized versions of the JAMStack news. That is, we use auto-scalable self-hosted runners on AWS (spot instances), to handle our build workloads for those instances.

However, we have been seeing issues where a self-hosted runner loses connection with GitHub's backend. The setup details are available in our staff-wiki:

🔒 - https://staff-wiki.freecodecamp.org/docs/flight-manuals/self-hosted-runners#runners-on-aws-on-aws

Here is a failed run: https://github.com/freeCodeCamp/news/actions/runs/1641250226

Other implementations are seeing the same issues as us: actions/actions-runner-controller#466

As a troubleshooting measure, we even deployed an always available runner on Azure instead and got the same failures.

As a workaround, we moved to use Azure Pipelines (which we can guarantee to have runners available, because of our low usage on that platform). A self-hosted runner on Azure Pipelines will work fine, but it's a tedious process to set up.

Really looking forward to a sane way of using self-hosted runners with English builds.

[raisedadead]

Seems to be fixed. Only took 2 years and we dont know what fixed it.