core-github-actions-runner

Configuration for generating the OS images used by Elvia's self-hosted GitHub Actions runners.

The image is generated in the GitHub workflow generate-image.yml using Packer. Packer will also publish the image to Azure.

The GitHub Actions runners used by Elvia (elvia-runner) are created dynamically by Actions Orchestrator and are based on the image generated by this repository. The Terraform configuration for the all the infrastructure not created by Actions Orchestrator (e.g. networking) is located in core-terraform. The credentials for authenticating to Azure are stored in GitHub environment variables/secrets and are also configured in core-terraform.

⚡ Quick-start

Change your GitHub Actions workflows from ubuntu-latest to elvia-runner:

runs-on: elvia-runner

Updating the image

Syncing with upstream

The configuration for the image is based on the runner-images repository (aka 'upstream'). We use a Go script to apply our own modifications to the upstream configuration. All these changes are reproducible, meaning no manual edits should be made to the configuration located at runner-images/! This is because any manual changes would be overwritten later when we run the Go script to sync with the upstream repository.

Syncing this repository with runner-images should be handled automatically by the GitHub workflow sync-upstream.yml. This workflow will run on a schedule and check for changes in the upstream repository, and create a pull request if necessary. The pull request should include a short guide on how to proceed with merging the changes.

If you want to manually sync with the upstream repository, you can run the following command:

go run main.go

Go, Packer and git must be installed on your machine for this to work.

Removing software

We remove some software from the image to reduce build times. To remove software from the image, edit the removeSoftwareList variable in main.go. After editing the file, run the following command:

go run main.go --apply

This should remove the required configuration from Packer and also remove the installation script. You can double check by checking the git diff.

Adding software

To add software to the image, edit the addSoftwareList variable in main.go.

You will also need to supply an installation script in the scripts directory. This script should contain the necessary commands to install the software. See scripts/install-trivy.sh for an example. Your script MUST follow the same naming convention as this example, i.e.: install-<software>.sh.

As with removing software, run the following command:

go run main.go --apply

This should add the required configuration to Packer and copy the installation script to the correct location. You can double check by checking the git diff.

Unremoving or unadding software

If you have previously removed or added software and want to revert the changes, you can run the following command:

go run main.go

This will clone a fresh copy of the upstream repository and try to reapply the changes. If you are out of date with the upstream repository, this will also include any new (unrelated) changes. It is recommended to sync with upstream using the above command if you are out of date, before trying to edit the removeSoftwareList or addSoftwareList variables.

Remove installations script when unadding software

If you have supplied an installation script for software that you no longer want to add, you should remove the script from the scripts directory first. Then, follow the steps above to unadd the software. However, the script will not be remove from the runner-images repository. This must be done manually. You can do this by running the following command:

rm runner-images/images/ubuntu/scripts/build/install-<software>.sh

Development

We use trunk-based development and two environments prod and dev. Any pull request to the trunk branch will generate and push an image to the dev environment. After merging to the trunk branch, the image will be pushed to the prod environment. Any new runners created by Actions Orchestrator in dev will use the dev image, and the same for prod.

The prod runners are available with the label elvia-runner and the dev runners are available with the label elvia-runner-dev:

runs-on: elvia-runner # prod runner
runs-on: elvia-runner-dev # dev runner

When testing, open a pull request to trunk and generate your image, which will be deployed to the dev environment. SSH access is enabled for the dev environment, so you can connect to the VM and test your changes. SSH access is configured in core-terraform, and you must provide your own public key.

Testing

After generating the image in GitHub Actions, we run integration tests to ensure that the image is working as expected. We trigger a workflow run in the repository core-github-actions-templates, using the new image. Some important things to note:

1. The workflow used for testing (test-actions.yml) from core-github-actions-templates is also used internally in that repository, and changes there might break the tests here.
2. For dev, Actions Orchestrator has a lower limit of the maximum number of runners that can be created. This means that if the test-actions.yml workflow has added more jobs, the tests might fail in dev but not in prod. To remediate this, we must both increase the limit in Actions Orchestrator and increase the Azure quota for the number of VM cores available in the subscription and region.