Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Lambda Vars to Parameter Store #941

Merged
merged 36 commits into from
Aug 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
736a854
fix(scale): Refactor Runner Type and Owner (#871)
mcaulifn Jun 15, 2021
6e35845
feat: support multiple instance types (#898)
mcaulifn Jun 16, 2021
a768687
docs: fix lambda_security_group_ids incorrect description #738 (#902)
npalm Jun 17, 2021
5b5ee3b
fix: scale down runners (#905)
npalm Jun 17, 2021
8746249
chore: group upgrade lambda dependencies (#906)
npalm Jun 17, 2021
7bda880
feat(runner): Move Lambda Vars to Parameter Store
mcaulifn Jun 23, 2021
333eae7
Add test for ssm module (#1)
mcaulifn Jun 25, 2021
65535fb
Removing KMS/GH Auth from scale-down
mcaulifn Jun 25, 2021
b460a1a
Merge branch 'mcaulifn/ssm' of https://github.com/mcaulifn/terraform-…
mcaulifn Jun 25, 2021
aecc383
Add SSM permissions to runner policy
mcaulifn Jun 30, 2021
c84ae6c
Allow custom key_id
mcaulifn Jun 30, 2021
d2b61f1
Fixing for loop
mcaulifn Jun 30, 2021
8f34d71
Move SSM policy to Lambdas
mcaulifn Jul 1, 2021
7d44ac8
Fixing function call
mcaulifn Jul 1, 2021
66bd078
chore: Bump aws-sdk (#752) (#909)
dependabot[bot] Jun 18, 2021
77c8e13
chore: Bump aws-sdk (#752) (#908)
dependabot[bot] Jun 18, 2021
fe0a126
chore: Bump aws-sdk (#752) (#887)
dependabot[bot] Jun 18, 2021
174b4ca
chore: Bump aws-sdk (#752) (#885)
dependabot[bot] Jun 18, 2021
0f1493f
chore: Bump aws-sdk (#752) (#889)
dependabot[bot] Jun 18, 2021
5314469
chore: Bump aws-sdk (#752) (#892)
dependabot[bot] Jun 18, 2021
640ef5f
chore: Bump aws-sdk (#752) (#907)
dependabot[bot] Jun 18, 2021
1658635
chore: Bump aws-sdk (#752) (#864)
dependabot[bot] Jun 18, 2021
a401c0d
chore: Bump eslint in /modules/webhook/lambdas/webhook (#918)
dependabot[bot] Jun 24, 2021
870280b
chore: Bump typescript (#929)
dependabot[bot] Jun 24, 2021
0fdf8cf
chore: Bump @typescript-eslint/eslint-plugin (#928)
dependabot[bot] Jun 24, 2021
a6b9a29
chore: Bump typescript in /modules/webhook/lambdas/webhook (#926)
dependabot[bot] Jun 24, 2021
b5096bb
feat: Added support for white listing of repositories (#915)
ravenolf Jul 7, 2021
7ac1a25
Need `,` after list item
mcaulifn Jul 7, 2021
cbf7a70
Move Lambda Policy to data resource
mcaulifn Jul 7, 2021
d8e6fbd
Merge branch 'develop' into mcaulifn/ssm
mcaulifn Jul 7, 2021
d46f9e2
Merge branch 'develop' into mcaulifn/ssm
mcaulifn Jul 14, 2021
0c9aff5
Addressing PR comments, fixing lint
mcaulifn Jul 16, 2021
3104d34
Refactoring Parameters to SSM Module
mcaulifn Jul 27, 2021
7c09270
Merge branch 'develop' into mcaulifn/ssm
mcaulifn Jul 27, 2021
4ea2a17
Fixing rebase
mcaulifn Jul 27, 2021
19199e3
Using only key ARN as input value
mcaulifn Jul 29, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Scaling down the runners is at the moment brute-forced, every configurable amoun

Downloading the GitHub Action Runner distribution can be occasionally slow (more than 10 minutes). Therefore a lambda is introduced that synchronizes the action runner binary from GitHub to an S3 bucket. The EC2 instance will fetch the distribution from the S3 bucket instead of the internet.

Secrets and private keys which are passed to the lambdas as environment variables are encrypted by default by a KMS key managed by the module. Alternatively you can pass your own KMS key. Encryption via KMS can be complete disabled by setting `encrypt_secrets` to `false`.
Secrets and private keys are stored in SSM Parameter Store. These values are encrypted using the default KMS key for SSM or passing in a custom KMS key.

![Architecture](docs/component-overview.svg)

Expand Down Expand Up @@ -325,14 +325,13 @@ No requirements.
|------|--------|---------|
| runner_binaries | ./modules/runner-binaries-syncer | |
| runners | ./modules/runners | |
| ssm | ./modules/ssm | |
| webhook | ./modules/webhook | |

## Resources

| Name |
|------|
| [aws_kms_alias](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_alias) |
| [aws_kms_key](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) |
| [aws_resourcegroups_group](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/resourcegroups_group) |
| [aws_sqs_queue](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) |
| [random_string](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) |
Expand All @@ -351,7 +350,6 @@ No requirements.
| enable\_cloudwatch\_agent | Enabling the cloudwatch agent on the ec2 runner instances, the runner contains default config. Configuration can be overridden via `cloudwatch_config`. | `bool` | `true` | no |
| enable\_organization\_runners | Register runners to organization, instead of repo level | `bool` | `false` | no |
| enable\_ssm\_on\_runners | Enable to allow access the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. | `bool` | `false` | no |
| encrypt\_secrets | Encrypt secret variables for lambda's such as secrets and private keys. | `bool` | `true` | no |
| environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
| ghes\_url | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB | `string` | `null` | no |
| github\_app | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`). | <pre>object({<br> key_base64 = string<br> id = string<br> client_id = string<br> client_secret = string<br> webhook_secret = string<br> })</pre> | n/a | yes |
Expand All @@ -360,12 +358,11 @@ No requirements.
| instance\_type | [DEPRECATED] See instance\_types. | `string` | `"m5.large"` | no |
| instance\_types | List of instance types for the action runner. | `set(string)` | `null` | no |
| key\_name | Key pair name | `string` | `null` | no |
| kms\_key\_id | Custom KMS key to encrypted lambda secrets, if not provided and `encrypt_secrets` = `true` a KMS key will be created by the module. Secrets will be encrypted with a context `Environment = var.environment`. | `string` | `null` | no |
| kms\_key\_arn | Optional CMK Key ARN to be used for Parameter Store. This key must be in the current account. | `string` | `null` | no |
| lambda\_s3\_bucket | S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly. | `any` | `null` | no |
| lambda\_security\_group\_ids | List of security group IDs associated with the Lambda function. | `list(string)` | `[]` | no |
| lambda\_subnet\_ids | List of subnets in which the action runners will be launched, the subnets needs to be subnets in the `vpc_id`. | `list(string)` | `[]` | no |
| logging\_retention\_in\_days | Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653. | `number` | `180` | no |
| manage\_kms\_key | Let the module manage the KMS key. | `bool` | `true` | no |
| market\_options | Market options for the action runner instances. Setting the value to `null` let the scaler create on-demand instances instead of spot instances. | `string` | `"spot"` | no |
| minimum\_running\_time\_in\_minutes | The time an ec2 action runner should be running at minimum before terminated if non busy. | `number` | `5` | no |
| repository\_white\_list | List of repositories allowed to use the github app | `list(string)` | `[]` | no |
Expand Down Expand Up @@ -407,6 +404,7 @@ No requirements.
|------|-------------|
| binaries\_syncer | n/a |
| runners | n/a |
| ssm\_parameters | n/a |
| webhook | n/a |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

Expand Down
15 changes: 0 additions & 15 deletions kms.tf

This file was deleted.

33 changes: 22 additions & 11 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ locals {
runner_architecture = substr(var.instance_type, 0, 2) == "a1" || substr(var.instance_type, 1, 2) == "6g" ? "arm64" : "x64"

ami_filter = length(var.ami_filter) > 0 ? var.ami_filter : local.runner_architecture == "arm64" ? { name = ["amzn2-ami-hvm-2*-arm64-gp2"] } : { name = ["amzn2-ami-hvm-2.*-x86_64-ebs"] }

github_app_parameters = {
client_id = module.ssm.parameters.github_app_client_id
client_secret = module.ssm.parameters.github_app_client_secret
id = module.ssm.parameters.github_app_id
key_base64 = module.ssm.parameters.github_app_key_base64
}
}

resource "random_string" "random" {
Expand All @@ -26,19 +33,25 @@ resource "aws_sqs_queue" "queued_builds" {
tags = var.tags
}

module "ssm" {
source = "./modules/ssm"

kms_key_arn = var.kms_key_arn
environment = var.environment
github_app = var.github_app
tags = local.tags
}

module "webhook" {
source = "./modules/webhook"

aws_region = var.aws_region
environment = var.environment
tags = local.tags
encryption = {
kms_key_id = local.kms_key_id
encrypt = var.encrypt_secrets
}
kms_key_arn = var.kms_key_arn

sqs_build_queue = aws_sqs_queue.queued_builds
github_app_webhook_secret = var.github_app.webhook_secret
sqs_build_queue = aws_sqs_queue.queued_builds
github_app_webhook_secret_arn = module.ssm.parameters.github_app_webhook_secret.arn

lambda_s3_bucket = var.lambda_s3_bucket
webhook_lambda_s3_key = var.webhook_lambda_s3_key
Expand All @@ -60,10 +73,6 @@ module "runners" {
subnet_ids = var.subnet_ids
environment = var.environment
tags = local.tags
encryption = {
kms_key_id = local.kms_key_id
encrypt = var.encrypt_secrets
}

s3_bucket_runner_binaries = module.runner_binaries.bucket
s3_location_runner_binaries = local.s3_action_runner_url
Expand All @@ -78,7 +87,7 @@ module "runners" {
ami_owners = var.ami_owners

sqs_build_queue = aws_sqs_queue.queued_builds
github_app = var.github_app
github_app_parameters = local.github_app_parameters
enable_organization_runners = var.enable_organization_runners
scale_down_schedule_expression = var.scale_down_schedule_expression
minimum_running_time_in_minutes = var.minimum_running_time_in_minutes
Expand Down Expand Up @@ -118,6 +127,8 @@ module "runners" {
runner_iam_role_managed_policy_arns = var.runner_iam_role_managed_policy_arns

ghes_url = var.ghes_url

kms_key_arn = var.kms_key_arn
}

module "runner_binaries" {
Expand Down
6 changes: 2 additions & 4 deletions modules/runners/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,6 @@ No Modules.
| [aws_iam_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) |
| [aws_iam_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) |
| [aws_iam_role_policy_attachment](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) |
| [aws_kms_ciphertext](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_ciphertext) |
| [aws_kms_grant](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_grant) |
| [aws_lambda_event_source_mapping](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_event_source_mapping) |
| [aws_lambda_function](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function) |
| [aws_lambda_permission](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) |
Expand All @@ -98,15 +96,15 @@ No Modules.
| enable\_cloudwatch\_agent | Enabling the cloudwatch agent on the ec2 runner instances, the runner contains default config. Configuration can be overridden via `cloudwatch_config`. | `bool` | `true` | no |
| enable\_organization\_runners | n/a | `bool` | n/a | yes |
| enable\_ssm\_on\_runners | Enable to allow access the runner instances for debugging purposes via SSM. Note that this adds additional permissions to the runner instances. | `bool` | n/a | yes |
| encryption | KMS key to encrypted lambda environment secrets. Either provide a key and `encrypt` set to `true`. Or set the key to `null` and encrypt to `false`. | <pre>object({<br> kms_key_id = string<br> encrypt = bool<br> })</pre> | n/a | yes |
| environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
| ghes\_url | GitHub Enterprise Server URL. DO NOT SET IF USING PUBLIC GITHUB | `string` | `null` | no |
| github\_app | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`). | <pre>object({<br> key_base64 = string<br> id = string<br> client_id = string<br> client_secret = string<br> })</pre> | n/a | yes |
| github\_app\_parameters | Parameter Store for GitHub App Parameters. | <pre>object({<br> key_base64 = map(string)<br> id = map(string)<br> client_id = map(string)<br> client_secret = map(string)<br> })</pre> | n/a | yes |
| idle\_config | List of time period that can be defined as cron expression to keep a minimum amount of runners active instead of scaling down to 0. By defining this list you can ensure that in time periods that match the cron expression within 5 seconds a runner is kept idle. | <pre>list(object({<br> cron = string<br> timeZone = string<br> idleCount = number<br> }))</pre> | `[]` | no |
| instance\_profile\_path | The path that will be added to the instance\_profile, if not set the environment name will be used. | `string` | `null` | no |
| instance\_type | [DEPRECATED] See instance\_types. | `string` | `"m5.large"` | no |
| instance\_types | List of instance types for the action runner. | `set(string)` | `null` | no |
| key\_name | Key pair name | `string` | `null` | no |
| kms\_key\_arn | Optional CMK Key ARN to be used for Parameter Store. | `string` | `null` | no |
| lambda\_s3\_bucket | S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly. | `any` | `null` | no |
| lambda\_security\_group\_ids | List of security group IDs associated with the Lambda function. | `list(string)` | `[]` | no |
| lambda\_subnet\_ids | List of subnets in which the lambda will be launched, the subnets needs to be subnets in the `vpc_id`. | `list(string)` | `[]` | no |
Expand Down
24 changes: 0 additions & 24 deletions modules/runners/encrypt.tf

This file was deleted.

15 changes: 10 additions & 5 deletions modules/runners/lambdas/runners/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,35 @@
"build": "ncc build src/lambda.ts -o dist",
"dist": "yarn build && cd dist && zip ../runners.zip index.js",
"format": "prettier --write \"**/*.ts\"",
"format-check": "prettier --check \"**/*.ts\""
"format-check": "prettier --check \"**/*.ts\"",
"all": "yarn build && yarn format && yarn lint && yarn test"
},
"devDependencies": {
"@types/aws-lambda": "^8.10.75",
"@types/express": "^4.17.11",
"@types/jest": "^26.0.20",
"@types/jest": "^26.0.24",
"@typescript-eslint/eslint-plugin": "^4.17.0",
"@typescript-eslint/parser": "^4.22.0",
"@vercel/ncc": "^0.29.0",
"eslint": "^7.22.0",
"jest": "^26.6.3",
"eslint-plugin-prettier": "3.4.0",
"jest": "27.0.6",
"jest-mock-extended": "^1.0.13",
"moment-timezone": "^0.5.33",
"nock": "^13.0.11",
"ts-jest": "^26.5.5",
"prettier": "2.3.2",
"ts-jest": "^27.0.4",
"ts-node": "^10.1.0",
"ts-node-dev": "^1.1.6"
},
"dependencies": {
"@aws-sdk/client-ssm": "^3.18.0",
"@octokit/auth-app": "3.4.0",
"@octokit/rest": "^18.3.5",
"@octokit/types": "^6.13.0",
"@types/aws-lambda": "^8.10.75",
"@types/express": "^4.17.11",
"@types/node": "^15.12.2",
"@types/node": "^16.4.3",
"aws-sdk": "^2.888.0",
"cron-parser": "^3.3.0",
"typescript": "^4.2.3",
Expand Down
8 changes: 8 additions & 0 deletions modules/runners/lambdas/runners/src/local-down.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { scaleDown } from './scale-runners/scale-down';


export function run(): void {
scaleDown();
}

run();
32 changes: 32 additions & 0 deletions modules/runners/lambdas/runners/src/local.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import { scaleUp } from './scale-runners/scale-up';

const sqsEvent = {
Records: [
{
messageId: 'e8d74d08-644e-42ca-bf82-a67daa6c4dad',
// eslint-disable-next-line max-len
receiptHandle: 'AQEBCpLYzDEKq4aKSJyFQCkJduSKZef8SJVOperbYyNhXqqnpFG5k74WygVAJ4O0+9nybRyeOFThvITOaS21/jeHiI5fgaM9YKuI0oGYeWCIzPQsluW5CMDmtvqv1aA8sXQ5n2x0L9MJkzgdIHTC3YWBFLQ2AxSveOyIHwW+cHLIFCAcZlOaaf0YtaLfGHGkAC4IfycmaijV8NSlzYgDuxrC9sIsWJ0bSvk5iT4ru/R4+0cjm7qZtGlc04k9xk5Fu6A+wRxMaIyiFRY+Ya19ykcevQldidmEjEWvN6CRToLgclk=',
// eslint-disable-next-line max-len
body: { "id": 19072, "repositoryName": "ErrBud", "repositoryOwner": "ActionsTest", "eventType": "check_run", "installationId": 5 },
attributes: {
ApproximateReceiveCount: '1',
SentTimestamp: '1626450047230',
SequenceNumber: '18863115285800432640',
MessageGroupId: '19072',
SenderId: 'AROA5KW7SQ6TTB3PW6WPH:cicddev-webhook',
MessageDeduplicationId: '0c458eeb87b7f6d2607301268fd3bf33dd898a49ebd888754ff7db510c4bff1e',
ApproximateFirstReceiveTimestamp: '1626450077251'
},
messageAttributes: {},
md5OfBody: '4aef3bd70526e152e86426a0938cbec6',
eventSource: 'aws:sqs',
eventSourceARN: 'arn:aws:sqs:us-west-2:916370655143:cicddev-queued-builds.fifo',
awsRegion: 'us-west-2'
}
]
};
export function run(): void {
scaleUp(sqsEvent.Records[0].eventSource, sqsEvent.Records[0].body);
}

run();
Loading