terraform-aws-gitlab-runner-spot

Table of contents

1. Overview
2. AWS Gitlab Runner - Overview Diagram
3. Prerequisites
   • AWS
   • Service linked roles
   • GitLab runner token configuration
   • GitLab runner cache
   • Inputs
   • Outputs
   • Examples
4. License

Overview

This module creates an auto-scaling GitLab CI runner on AWS spot instances. The original setup of the module is based on the blog post: Auto scale GitLab CI runners and save 90% on EC2 costs.

The runners created by the module using spot instances for running the builds using the docker+machine executor.

• Shared cache in S3 with life cycle management to clear objects after x days.
• Logs streamed to CloudWatch.
• Runner agents registered automatically.

The runner agent is running on a single EC2 node and runners are created by docker machine using spot instances. Runners will scale automatically based on configuration. The module creates by default a S3 cache that is shared cross runners (spot instances).

AWS Gitlab Runner - Overview Diagram

Prerequisites

AWS

Ensure you have setup you AWS credentials. The module requires access to IAM, EC2, CloudWatch, S3 and SSM.

Service linked roles

The GitLab runner EC2 instance requires the following service linked roles:

• AWSServiceRoleForAutoScaling
• AWSServiceRoleForEC2Spot

GitLab runner token configuration

The runner is registered on initial deployment.

To register the runner automatically set the variable gitlab_runner_registration_config["token"]. This token value can be found in your GitLab project, group, or global settings. For a generic runner you can find the token in the admin section. Here is an example:

gitlab_runner_registration_config = {
  registration_token = "<registration token>"
  description        = "<some description>"
  locked_to_project  = "true"
  maximum_timeout    = "3600"
  access_level       = "<not_protected OR ref_protected, ref_protected runner will only run on pipelines triggered on protected branches>"
}

GitLab runner cache

The module creates a cache for the runner in S3. Old objects are automatically remove via a configurable life cycle policy on the bucket.

Inputs

The below outlines the current parameters and defaults.

Name	Description	Type	Default	Required
aws_region	Name of S3 region for the runner cache and SSM	string	""	Yes
aws_availability_zone	AWS availability zone ('a', 'b', 'c' etc)	string	a	No
vpc_id	The target VPC for the docker-machine and runner instances	string	""	Yes
subnet_id	Subnet used for hosting the GitLab runner	string	""	Yes
subnet_ids	List of subnets used for hosting the GitLab runners	list(string)	""	Yes
key_name	The name of the EC2 key pair to use	string	default	No
spot_price	Spot bid price for the GitLab Runner instance	string	0.06	No
enable_ssh_access	Enables SSH access to the GitLab Runner instance	bool	false	No
ssh_cidr_blocks	List of CIDR blocks to allow SSH Access to docker machine and the GitLab Runner	list(string)	[0.0.0.0/0]	No
gitlab_runner_registration_config		map(string)	(map)	No
schedule_config	Map containing the configuration of the ASG scale-in and scale-up for the runner instance	map	(map)	No
globals_concurrent	Concurrent value for the runners	number	10	No
runners_name	The Runner's description, just informatory	string	""	Yes
runners_url	The GitLab URL for the instance to connect to	string	""	Yes
runners_environment	Append or overwrite environment variables	list(string)	[]	No
runners_request_concurrency	Limit number of concurrent requests for new jobs from GitLab	number	1	No
runners_output_limit	Set maximum build log size in KB	number	4096	No
runners_limit	Limit how many jobs can be handled concurrently by this token	number	0	No
runners_docker_shm_size	Shared memory size for images (in bytes)	number	0	No
runners_cache_bucket_name	Name of the storage bucket where runner cache will be stored	string	""	Yes
runners_machine_idle_count	Number of machines that need to be created and waiting in Idle state	number	0	No
runners_machine_idle_time	Time (in seconds) for machine to be in Idle state before it is removed	number	600	No
runners_machine_max_builds	Builds count after which machine will be removed	number	0	No
runners_machine_off_peak_timezone	Off peak idle time zone of the runners	string	Australia/Sydney	No
runners_machine_off_peak_idle_count	Off peak idle count of the runners	number	0	No
runners_machine_off_peak_idle_time	Off peak idle time of the runners	number	0	No
runners_machine_off_peak_periods	Time periods when the scheduler is in the OffPeak mode. A list of cron-style patterns	list(string)	[]	No

Outputs

Name	Description
runner_as_group_name	Name of the autoscaling group for the gitlab-runner instance
runner_cache_bucket_arn	ARN of the S3 for the build cache.
runner_cache_bucket_name	Name of the S3 for the build cache.
runner_agent_role_arn	ARN of the role used for the ec2 instance for the GitLab runner agent.
runner_agent_role_name	Name of the role used for the ec2 instance for the GitLab runner agent.
runner_role_arn	ARN of the role used for the docker machine runners.
runner_role_name	Name of the role used for the docker machine runners.
runner_agent_sg_id	ID of the security group attached to the GitLab runner agent.
runner_sg_id	ID of the security group attached to the docker machine runners.

Examples

To create a Gitlab Runner:

variable "registration_token" {}

variable "enable_ssh_access" {
  default = false
}

variable "bucket_name" {
  default = "config-bucket-1c5a1978-d138-4084-a3b4-fd4c403a89a0"
}

data "aws_availability_zones" "available" {
  state = "available"
}

variable "vpc_id" {}
variable "subnet_ids" {
  type = list(string)
}

module "runner" {
  source = "[email protected]:cmdlabs/terraform-aws-gitlab-runner-spot.git"

  key_name = "default"

  aws_region = "ap-southeast-2"

  runners_cache_bucket_name = var.bucket_name

  vpc_id       = var.vpc_id
  subnet_ids   = var.subnet_ids
  subnet_id    = var.subnet_ids[0]

  runners_name = "test-runner"
  runners_url  = "https://gitlab.com"

  gitlab_runner_registration_config = {
    registration_token = var.registration_token
    description        = "runner default - auto"
    locked_to_project  = "true"
    run_untagged       = "false"
    maximum_timeout    = "3600"
    access_level       = "not_protected"
  }

  runners_machine_off_peak_timezone   = "Australia/Sydney"
  runners_machine_off_peak_idle_count = 0
  runners_machine_off_peak_idle_time  = 60
  runners_machine_off_peak_periods    = [
    "* * 0-9,17-23 * * mon-fri *",
    "* * * * * sat,sun *"
  ]

  enable_ssh_access = var.enable_ssh_access
}

To apply that:

▶ TF_VAR_registration_token=xxxxxxxx terraform apply

License

Apache 2.0.