Skip to content

Latest commit

 

History

History
263 lines (223 loc) · 44.8 KB

README.md

File metadata and controls

263 lines (223 loc) · 44.8 KB
Raw

Terraform module Self-Hosted Scalable GitHub Actions runners on AWS.

docs awesome-runners Terraform registry Terraform checks Lambdas

📄 Extensive documentation is available via our GitHub Pages Docs site.

📢 We maintain the project as a truly open-source project. We maintain the project on a best effort basis. We welcome contributions from the community. Feel free to help us answering issues, reviewing PRs, or maintaining and improving the project.

📢 v5 replaces Amazon Linux 2 with Amazon Linux 2023 as default OS. Check the PR for more details and other changes.

📢 For contributions to older versions you can make a PR to the related branch, e.g. v4. We have no release process in place for older versions.

This Terraform module creates the required infrastructure needed to host GitHub Actions self-hosted, auto-scaling runners on AWS spot instances. It provides the required logic to handle the life cycle for scaling up and down using a set of AWS Lambda functions. Runners are scaled down to zero to avoid costs when no workflows are active.

Runners overview

Features

  • Scaling: Scale up and down based on GitHub events
  • Sustainability: Scale down to zero when no jobs are running
  • Security: Runners are created on-demand and terminated after use (ephemeral runners)
  • Cost optimization: Runners are created on spot instances
  • Tailored software, hardware and network configuration: Bring your own AMI, define the instance types and subnets to use.
  • OS support: Linux (x64/arm64) and Windows
  • Multi-Runner: Create multiple runner configurations with a single deployment
  • GitHub cloud and GitHub Enterprise Server (GHES) support.
  • Org and repo level runners. enterprise level runners are not supported (yet).

Getting started

Check out the detailed instructions in the Getting Started section of the docs. On a high level, the following steps are required to get started:

  • Setup your AWS account
  • Create and configure a GitHub App
  • Download or build the required lambdas
  • Deploy the module using Terraform
  • Install the GitHub App to your organization or repositories and add your repositories to the runner group(s).

Check out the provided Terraform examples in the examples directory for different scenarios.

Configuration

Please check the configuration section of the docs for major configuration options. See the Terraform module documentation for all available options.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

We welcome contributions, please check out the contribution guide. Be aware we use pre commit hooks to update the docs.

Philips Forest

This module is part of the Philips Forest.

                                                     ___                   _
                                                    / __\__  _ __ ___  ___| |_
                                                   / _\/ _ \| '__/ _ \/ __| __|
                                                  / / | (_) | | |  __/\__ \ |_
                                                  \/   \___/|_|  \___||___/\__|

                                                                 Infrastructure

Talk to the forestkeepers in the runners-channel on Slack.

Slack

Terraform root module documention

Requirements

Name Version
terraform >= 1.3.0
aws ~> 5.27
random ~> 3.0

Providers

Name Version
aws 5.31.0
random 3.6.0

Modules

Name Source Version
ami_housekeeper ./modules/ami-housekeeper n/a
instance_termination_watcher ./modules/termination-watcher n/a
runner_binaries ./modules/runner-binaries-syncer n/a
runners ./modules/runners n/a
ssm ./modules/ssm n/a
webhook ./modules/webhook n/a

Resources

Name Type
aws_sqs_queue.queued_builds resource
aws_sqs_queue.queued_builds_dlq resource
aws_sqs_queue.webhook_events_workflow_job_queue resource
aws_sqs_queue_policy.build_queue_dlq_policy resource
aws_sqs_queue_policy.build_queue_policy resource
aws_sqs_queue_policy.webhook_events_workflow_job_queue_policy resource
random_string.random resource
aws_iam_policy_document.deny_unsecure_transport data source

Inputs

Name Description Type Default Required
ami_filter Map of lists used to create the AMI filter for the action runner AMI. map(list(string)) 
{
  "state": [
    "available"
  ]
}
 no
ami_housekeeper_cleanup_config Configuration for AMI cleanup.

amiFilters - Filters to use when searching for AMIs to cleanup. Default filter for images owned by the account and that are available.
dryRun - If true, no AMIs will be deregistered. Default false.
launchTemplateNames - Launch template names to use when searching for AMIs to cleanup. Default no launch templates.
maxItems - The maximum numer of AMI's tha will be queried for cleanup. Default no maximum.
minimumDaysOld - Minimum number of days old an AMI must be to be considered for cleanup. Default 30.
ssmParameterNames - SSM parameter names to use when searching for AMIs to cleanup. This parameter should be set when using SSM to configure the AMI to use. Default no SSM parameters.		 
object({
    amiFilters = optional(list(object({
      Name   = string
      Values = list(string)
      })),
      [{
        Name : "state",
        Values : ["available"],
        },
        {
          Name : "image-type",
          Values : ["machine"],
      }]
    )
    dryRun              = optional(bool, false)
    launchTemplateNames = optional(list(string))
    maxItems            = optional(number)
    minimumDaysOld      = optional(number, 30)
    ssmParameterNames   = optional(list(string))
  })
 {} no
ami_housekeeper_lambda_s3_key S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas. string null no
ami_housekeeper_lambda_s3_object_version S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket. string null no
ami_housekeeper_lambda_schedule_expression Scheduler expression for action runner binary syncer. string "rate(1 day)" no
ami_housekeeper_lambda_timeout Time out of the lambda in seconds. number 300 no
ami_housekeeper_lambda_zip File location of the lambda zip file. string null no
ami_id_ssm_parameter_name Externally managed SSM parameter (of data type aws:ec2:image) that contains the AMI ID to launch runner instances from. Overrides ami_filter string null no
ami_kms_key_arn Optional CMK Key ARN to be used to launch an instance from a shared encrypted AMI string null no
ami_owners The list of owners used to select the AMI of action runner instances. list(string) 
[
  "amazon"
]
 no
associate_public_ipv4_address Associate public IPv4 with the runner. Only tested with IPv4 bool false no
aws_partition (optiona) partition in the arn namespace to use if not 'aws' string "aws" no
aws_region AWS region. string n/a yes
block_device_mappings The EC2 instance block device configuration. Takes the following keys: device_name, delete_on_termination, volume_type, volume_size, encrypted, iops, throughput, kms_key_id, snapshot_id. 
list(object({
    delete_on_termination = optional(bool, true)
    device_name           = optional(string, "/dev/xvda")
    encrypted             = optional(bool, true)
    iops                  = optional(number)
    kms_key_id            = optional(string)
    snapshot_id           = optional(string)
    throughput            = optional(number)
    volume_size           = number
    volume_type           = optional(string, "gp3")
  }))
 
[
  {
    "volume_size": 30
  }
]
 no
cloudwatch_config (optional) Replaces the module's default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. string null no
create_service_linked_role_spot (optional) create the service linked role for spot instances that is required by the scale-up lambda. bool false no
delay_webhook_event The number of seconds the event accepted by the webhook is invisible on the queue before the scale up lambda will receive the event. number 30 no
disable_runner_autoupdate Disable the auto update of the github runner agent. Be aware there is a grace period of 30 days, see also the GitHub article bool false no
enable_ami_housekeeper Option to disable the lambda to clean up old AMIs. bool false no
enable_cloudwatch_agent Enables the cloudwatch agent on the ec2 runner instances. The runner uses a default config that can be overridden via cloudwatch_config. bool true no
enable_ephemeral_runners Enable ephemeral runners, runners will only be used once. bool false no
enable_event_rule_binaries_syncer DEPRECATED: Replaced by state_event_rule_binaries_syncer. bool null no
enable_fifo_build_queue Enable a FIFO queue to keep the order of events received by the webhook. Recommended for repo level runners. bool false no
enable_jit_config Overwrite the default behavior for JIT configuration. By default JIT configuration is enabled for ephemeral runners and disabled for non-ephemeral runners. In case of GHES check first if the JIT config API is avaialbe. In case you upgradeing from 3.x to 4.x you can set enable_jit_config to false to avoid a breaking change when having your own AMI. bool null no
enable_job_queued_check Only scale if the job event received by the scale up lambda is in the queued state. By default enabled for non ephemeral runners and disabled for ephemeral. Set this variable to overwrite the default behavior. bool null no
enable_managed_runner_security_group Enables creation of the default managed security group. Unmanaged security groups can be specified via runner_additional_security_group_ids. bool true no
enable_organization_runners Register runners to organization, instead of repo level bool false no
enable_runner_binaries_syncer Option to disable the lambda to sync GitHub runner distribution, useful when using a pre-build AMI. bool true no
enable_runner_detailed_monitoring Should detailed monitoring be enabled for the runner. Set this to true if you want to use detailed monitoring. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html for details. bool false no
enable_runner_on_demand_failover_for_errors Enable on-demand failover. For example to fall back to on demand when no spot capacity is available the variable can be set to InsufficientInstanceCapacity. When not defined the default behavior is to retry later. list(string) [] no
enable_runner_workflow_job_labels_check_all If set to true all labels in the workflow job must match the GitHub labels (os, architecture and self-hosted). When false if any label matches it will trigger the webhook. bool true no
enable_ssm_on_runners Enable to allow access to the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. bool false no
enable_user_data_debug_logging_runner Option to enable debug logging for user-data, this logs all secrets as well. bool false no
enable_userdata Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI. bool true no
enable_workflow_job_events_queue Enabling this experimental feature will create a secondory sqs queue to which a copy of the workflow_job event will be delivered. bool false no
ghes_ssl_verify GitHub Enterprise SSL verification. Set to 'false' when custom certificate (chains) is used for GitHub Enterprise Server (insecure). bool true no
ghes_url GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB string null no
github_app GitHub app parameters, see your github app. Ensure the key is the base64-encoded .pem file (the output of base64 app.private-key.pem, not the content of private-key.pem). 
object({
    key_base64     = string
    id             = string
    webhook_secret = string
  })
 n/a yes
idle_config List of time periods, defined as a cron expression, to keep a minimum amount of runners active instead of scaling down to 0. By defining this list you can ensure that in time periods that match the cron expression within 5 seconds a runner is kept idle. 
list(object({
    cron             = string
    timeZone         = string
    idleCount        = number
    evictionStrategy = optional(string, "oldest_first")
  }))
 [] no
instance_allocation_strategy The allocation strategy for spot instances. AWS recommends using price-capacity-optimized however the AWS default is lowest-price. string "lowest-price" no
instance_max_spot_price Max price price for spot instances per hour. This variable will be passed to the create fleet as max spot price for the fleet. string null no
instance_profile_path The path that will be added to the instance_profile, if not set the environment name will be used. string null no
instance_target_capacity_type Default lifecycle used for runner instances, can be either spot or on-demand. string "spot" no
instance_termination_watcher Configuration for the instance termination watcher. This feature is Beta, changes will not trigger a major release as long in beta.

enable: Enable or disable the spot termination watcher.
'enable_metrics': Enable or disable the metrics for the spot termination watcher.
memory_size: Memory size linit in MB of the lambda.
s3_key: S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas.
s3_object_version: S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket.
timeout: Time out of the lambda in seconds.
zip: File location of the lambda zip file.		 
object({
    enable = optional(bool, false)
    enable_metric = optional(object({
      spot_warning = optional(bool, false)
    }))
    memory_size       = optional(number, null)
    s3_key            = optional(string, null)
    s3_object_version = optional(string, null)
    timeout           = optional(number, null)
    zip               = optional(string, null)
  })
 {} no
instance_types List of instance types for the action runner. Defaults are based on runner_os (al2023 for linux and Windows Server Core for win). list(string) 
[
  "m5.large",
  "c5.large"
]
 no
job_queue_retention_in_seconds The number of seconds the job is held in the queue before it is purged. number 86400 no
key_name Key pair name string null no
kms_key_arn Optional CMK Key ARN to be used for Parameter Store. This key must be in the current account. string null no
lambda_architecture AWS Lambda architecture. Lambda functions using Graviton processors ('arm64') tend to have better price/performance than 'x86_64' functions. string "arm64" no
lambda_principals (Optional) add extra principals to the role created for execution of the lambda, e.g. for local testing. 
list(object({
    type        = string
    identifiers = list(string)
  }))
 [] no
lambda_runtime AWS Lambda runtime. string "nodejs20.x" no
lambda_s3_bucket S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly. string null no
lambda_security_group_ids List of security group IDs associated with the Lambda function. list(string) [] no
lambda_subnet_ids List of subnets in which the action runners will be launched, the subnets needs to be subnets in the vpc_id. list(string) [] no
lambda_tracing_mode DEPRECATED: Replaced by tracing_config. string null no
log_level Logging level for lambda logging. Valid values are 'silly', 'trace', 'debug', 'info', 'warn', 'error', 'fatal'. string "info" no
logging_kms_key_id Specifies the kms key id to encrypt the logs with. string null no
logging_retention_in_days Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653. number 180 no
metrics_namespace The namespace for the metrics created by the module. Merics will only be created if explicit enabled. string "GitHub Runners" no
minimum_running_time_in_minutes The time an ec2 action runner should be running at minimum before terminated, if not busy. number null no
pool_config The configuration for updating the pool. The pool_size to adjust to by the events triggered by the schedule_expression. For example you can configure a cron expression for weekdays to adjust the pool to 10 and another expression for the weekend to adjust the pool to 1. 
list(object({
    schedule_expression = string
    size                = number
  }))
 [] no
pool_lambda_memory_size Memory size limit for scale-up lambda. number 512 no
pool_lambda_reserved_concurrent_executions Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. number 1 no
pool_lambda_timeout Time out for the pool lambda in seconds. number 60 no
pool_runner_owner The pool will deploy runners to the GitHub org ID, set this value to the org to which you want the runners deployed. Repo level is not supported. string null no
prefix The prefix used for naming resources string "github-actions" no
queue_encryption Configure how data on queues managed by the modules in ecrypted at REST. Options are encryped via SSE, non encrypted and via KMSS. By default encryptes via SSE is enabled. See for more details the Terraform aws_sqs_queue resource https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue. 
object({
    kms_data_key_reuse_period_seconds = number
    kms_master_key_id                 = string
    sqs_managed_sse_enabled           = bool
  })
 
{
  "kms_data_key_reuse_period_seconds": null,
  "kms_master_key_id": null,
  "sqs_managed_sse_enabled": true
}
 no
redrive_build_queue Set options to attach (optional) a dead letter queue to the build queue, the queue between the webhook and the scale up lambda. You have the following options. 1. Disable by setting enabled to false. 2. Enable by setting enabled to true, maxReceiveCount to a number of max retries. 
object({
    enabled         = bool
    maxReceiveCount = number
  })
 
{
  "enabled": false,
  "maxReceiveCount": null
}
 no
repository_white_list List of github repository full names (owner/repo_name) that will be allowed to use the github app. Leave empty for no filtering. list(string) [] no
role_path The path that will be added to role path for created roles, if not set the environment name will be used. string null no
role_permissions_boundary Permissions boundary that will be added to the created roles. string null no
runner_additional_security_group_ids (optional) List of additional security groups IDs to apply to the runner. list(string) [] no
runner_architecture The platform architecture of the runner instance_type. string "x64" no
runner_as_root Run the action runner under the root user. Variable runner_run_as will be ignored. bool false no
runner_binaries_s3_logging_bucket Bucket for action runner distribution bucket access logging. string null no
runner_binaries_s3_logging_bucket_prefix Bucket prefix for action runner distribution bucket access logging. string null no
runner_binaries_s3_sse_configuration Map containing server-side encryption configuration for runner-binaries S3 bucket. any 
{
  "rule": {
    "apply_server_side_encryption_by_default": {
      "sse_algorithm": "AES256"
    }
  }
}
 no
runner_binaries_s3_versioning Status of S3 versioning for runner-binaries S3 bucket. Once set to Enabled the change cannot be reverted via Terraform! string "Disabled" no
runner_binaries_syncer_lambda_memory_size Memory size limit in MB for binary syncer lambda. number 256 no
runner_binaries_syncer_lambda_timeout Time out of the binaries sync lambda in seconds. number 300 no
runner_binaries_syncer_lambda_zip File location of the binaries sync lambda zip file. string null no
runner_boot_time_in_minutes The minimum time for an EC2 runner to boot and register as a runner. number 5 no
runner_credit_specification The credit option for CPU usage of a T instance. Can be unset, "standard" or "unlimited". string null no
runner_ec2_tags Map of tags that will be added to the launch template instance tag specifications. map(string) {} no
runner_egress_rules List of egress rules for the GitHub runner instances. 
list(object({
    cidr_blocks      = list(string)
    ipv6_cidr_blocks = list(string)
    prefix_list_ids  = list(string)
    from_port        = number
    protocol         = string
    security_groups  = list(string)
    self             = bool
    to_port          = number
    description      = string
  }))
 
[
  {
    "cidr_blocks": [
      "0.0.0.0/0"
    ],
    "description": null,
    "from_port": 0,
    "ipv6_cidr_blocks": [
      "::/0"
    ],
    "prefix_list_ids": null,
    "protocol": "-1",
    "security_groups": null,
    "self": null,
    "to_port": 0
  }
]
 no
runner_extra_labels Extra (custom) labels for the runners (GitHub). Labels checks on the webhook can be enforced by setting enable_runner_workflow_job_labels_check_all. GitHub read-only labels should not be provided. list(string) [] no
runner_group_name Name of the runner group. string "Default" no
runner_iam_role_managed_policy_arns Attach AWS or customer-managed IAM policies (by ARN) to the runner IAM role list(string) [] no
runner_log_files (optional) Replaces the module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. 
list(object({
    log_group_name   = string
    prefix_log_group = bool
    file_path        = string
    log_stream_name  = string
  }))
 null no
runner_metadata_options Metadata options for the ec2 runner instances. By default, the module uses metadata tags for bootstrapping the runner, only disable instance_metadata_tags when using custom scripts for starting the runner. map(any) 
{
  "http_endpoint": "enabled",
  "http_put_response_hop_limit": 1,
  "http_tokens": "required",
  "instance_metadata_tags": "enabled"
}
 no
runner_name_prefix The prefix used for the GitHub runner name. The prefix will be used in the default start script to prefix the instance name when register the runner in GitHub. The value is availabe via an EC2 tag 'ghr:runner_name_prefix'. string "" no
runner_os The EC2 Operating System type to use for action runner instances (linux,windows). string "linux" no
runner_run_as Run the GitHub actions agent as user. string "ec2-user" no
runners_ebs_optimized Enable EBS optimization for the runner instances. bool false no
runners_lambda_s3_key S3 key for runners lambda function. Required if using S3 bucket to specify lambdas. string null no
runners_lambda_s3_object_version S3 object version for runners lambda function. Useful if S3 versioning is enabled on source bucket. string null no
runners_lambda_zip File location of the lambda zip file for scaling runners. string null no
runners_maximum_count The maximum number of runners that will be created. number 3 no
runners_scale_down_lambda_memory_size Memory size limit in MB for scale-down lambda. number 512 no
runners_scale_down_lambda_timeout Time out for the scale down lambda in seconds. number 60 no
runners_scale_up_Lambda_memory_size Memory size limit in MB for scale-up lambda. number null no
runners_scale_up_lambda_memory_size Memory size limit in MB for scale-up lambda. number 512 no
runners_scale_up_lambda_timeout Time out for the scale up lambda in seconds. number 30 no
runners_ssm_housekeeper Configuration for the SSM housekeeper lambda. This lambda deletes token / JIT config from SSM.

schedule_expression: is used to configure the schedule for the lambda.
enabled: enable or disable the lambda trigger via the EventBridge.
lambda_memory_size: lambda memery size limit.
lambda_timeout: timeout for the lambda in seconds.
config: configuration for the lambda function. Token path will be read by default from the module.		 
object({
    schedule_expression = optional(string, "rate(1 day)")
    enabled             = optional(bool, true)
    lambda_memory_size  = optional(number, 512)
    lambda_timeout      = optional(number, 60)
    config = object({
      tokenPath      = optional(string)
      minimumDaysOld = optional(number, 1)
      dryRun         = optional(bool, false)
    })
  })
 
{
  "config": {}
}
 no
scale_down_schedule_expression Scheduler expression to check every x for scale down. string "cron(*/5 * * * ? *)" no
scale_up_reserved_concurrent_executions Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. number 1 no
ssm_paths The root path used in SSM to store configuration and secrets. 
object({
    root       = optional(string, "github-action-runners")
    app        = optional(string, "app")
    runners    = optional(string, "runners")
    webhook    = optional(string, "webhook")
    use_prefix = optional(bool, true)
  })
 {} no
state_event_rule_binaries_syncer Option to disable EventBridge Lambda trigger for the binary syncer, useful to stop automatic updates of binary distribution string "ENABLED" no
subnet_ids List of subnets in which the action runner instances will be launched. The subnets need to exist in the configured VPC (vpc_id), and must reside in different availability zones (see philips-labs#2904) list(string) n/a yes
syncer_lambda_s3_key S3 key for syncer lambda function. Required if using an S3 bucket to specify lambdas. string null no
syncer_lambda_s3_object_version S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket. string null no
tags Map of tags that will be added to created resources. By default resources will be tagged with name and environment. map(string) {} no
tracing_config Configuration for lambda tracing. 
object({
    mode                  = optional(string, null)
    capture_http_requests = optional(bool, false)
    capture_error         = optional(bool, false)
  })
 {} no
userdata_content Alternative user-data content, replacing the templated one. By providing your own user_data you have to take care of installing all required software, including the action runner and registering the runner. Be-aware configuration paramaters in SSM as well as tags are treated as internals. Changes will not trigger a breaking release. string null no
userdata_post_install Script to be ran after the GitHub Actions runner is installed on the EC2 instances string "" no
userdata_pre_install Script to be ran before the GitHub Actions runner is installed on the EC2 instances string "" no
userdata_template Alternative user-data template file path, replacing the default template. By providing your own user_data you have to take care of installing all required software, including the action runner. Variables userdata_pre/post_install are ignored. string null no
vpc_id The VPC for security groups of the action runners. string n/a yes
webhook_lambda_apigateway_access_log_settings Access log settings for webhook API gateway. 
object({
    destination_arn = string
    format          = string
  })
 null no
webhook_lambda_memory_size Memory size limit in MB for webhook lambda in. number 256 no
webhook_lambda_s3_key S3 key for webhook lambda function. Required if using S3 bucket to specify lambdas. string null no
webhook_lambda_s3_object_version S3 object version for webhook lambda function. Useful if S3 versioning is enabled on source bucket. string null no
webhook_lambda_timeout Time out of the webhook lambda in seconds. number 10 no
webhook_lambda_zip File location of the webhook lambda zip file. string null no
workflow_job_queue_configuration Configuration options for workflow job queue which is only applicable if the flag enable_workflow_job_events_queue is set to true. 
object({
    delay_seconds              = number
    visibility_timeout_seconds = number
    message_retention_seconds  = number
  })
 
{
  "delay_seconds": null,
  "message_retention_seconds": null,
  "visibility_timeout_seconds": null
}
 no

Outputs

Name Description
binaries_syncer n/a
instance_termination_watcher n/a
queues SQS queues.
runners n/a
ssm_parameters n/a
webhook n/a