Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new separate terraform plans for staging and production #309

Merged
merged 18 commits into from
Nov 1, 2021
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 165 additions & 0 deletions .github/workflows/terragrunt_plan_production.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
name: "Terragrunt plan PRODUCTION"

on:
pull_request:
paths:
- "aws/**"
- "env/production/**"
- "env/terragrunt.hcl"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To cut down on PR noise, it might make sense to set this to the same triggers as the merge_to_main_production.yml paths:
https://github.com/cds-snc/notification-terraform/blob/main/.github/workflows/merge_to_main_production.yml#L8-L10

That way you only see Prod plan comments if you're expecting to have terraform apply run against Prod when you merge.

- ".github/workflows/terragrunt_plan_production.yml"

env:
TARGET_ENV_PATH: production
AWS_ACCESS_KEY_ID: ${{ secrets.PRODUCTION_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.PRODUCTION_AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ca-central-1
TERRAFORM_VERSION: 0.14.4
TERRAGRUNT_VERSION: 0.31.0
TF_VAR_rds_cluster_password: ${{ secrets.PRODUCTION_RDS_CLUSTER_PASSWORD }}
TF_VAR_cloudwatch_opsgenie_alarm_webhook: ${{ secrets.PRODUCTION_CLOUDWATCH_OPSGENIE_ALARM_WEBHOOK }}
TF_VAR_cloudwatch_slack_webhook_warning_topic: ${{ secrets.PRODUCTION_CLOUDWATCH_SLACK_WEBHOOK }}
TF_VAR_cloudwatch_slack_webhook_critical_topic: ${{ secrets.PRODUCTION_CLOUDWATCH_SLACK_WEBHOOK }}
TF_VAR_cloudwatch_slack_webhook_general_topic: ${{ secrets.PRODUCTION_CLOUDWATCH_SLACK_WEBHOOK }}
TF_VAR_slack_channel_warning_topic: notification-ops
TF_VAR_slack_channel_critical_topic: notification-ops
TF_VAR_slack_channel_general_topic: notification-ops
TF_VAR_admin_client_secret: ${{ secrets.PRODUCTION_ADMIN_CLIENT_SECRET }}
TF_VAR_admin_client_user_name: ${{ secrets.PRODUCTION_ADMIN_CLIENT_USER_NAME }}
TF_VAR_api_host_name: ${{ secrets.PRODUCTION_API_HOST_NAME }}
TF_VAR_asset_domain: ${{ secrets.PRODUCTION_ASSET_DOMAIN }}
TF_VAR_asset_upload_bucket_name: ${{ secrets.PRODUCTION_ASSET_UPLOAD_BUCKET_NAME }}
TF_VAR_auth_tokens: ${{ secrets.PRODUCTION_AUTH_TOKENS }}
TF_VAR_base_domain: ${{ secrets.PRODUCTION_BASE_DOMAIN }}
TF_VAR_csv_upload_bucket_name: ${{ secrets.PRODUCTION_CSV_UPLOAD_BUCKET_NAME }}
TF_VAR_dangerous_salt: ${{ secrets.PRODUCTION_DANGEROUS_SALT }}
TF_VAR_documents_bucket: ${{ secrets.PRODUCTION_DOCUMENTS_BUCKET }}
TF_VAR_document_download_api_host: ${{ secrets.PRODUCTION_DOCUMENT_DOWNLOAD_API_HOST }}
TF_VAR_mlwr_host: "false"
TF_VAR_notification_queue_prefix: eks-notification-canada-ca
TF_VAR_redis_url: ${{ secrets.PRODUCTION_REDIS_URL }}
TF_VAR_secret_key: ${{ secrets.PRODUCTION_SECRET_KEY }}
TF_VAR_sqlalchemy_database_reader_uri: ${{ secrets.PRODUCTION_SQLALCHEMY_DATABASE_READER_URI }}
TF_VAR_sqlalchemy_database_uri: ${{ secrets.PRODUCTION_SQLALCHEMY_DATABASE_URI }}

jobs:
terragrunt-plan-production:
runs-on: ubuntu-latest
steps:

- name: Checkout
uses: actions/checkout@v2

- name: Setup Terraform
uses: hashicorp/setup-terraform@3d8debd658c92063839bc97da5c2427100420dec # v1.3.2
with:
terraform_version: ${{ env.TERRAFORM_VERSION }}
terraform_wrapper: false

- name: Setup Terragrunt
run: |
mkdir bin
wget -O bin/terragrunt https://github.com/gruntwork-io/terragrunt/releases/download/v$TERRAGRUNT_VERSION/terragrunt_linux_amd64
chmod +x bin/*
echo "$GITHUB_WORKSPACE/bin" >> $GITHUB_PATH

- name: Set INFRASTRUCTURE_VERSION
run: |
INFRASTRUCTURE_VERSION=`cat ./.github/workflows/infrastructure_version.txt`
echo "INFRASTRUCTURE_VERSION=$INFRASTRUCTURE_VERSION" >> $GITHUB_ENV

- uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721 # v2.10.2
id: filter
with:
filters: |
common:
- '.github/workflows/terragrunt-plan-production.yml'
- 'env/common/**'
- 'env/terragrunt.hcl'
- 'env/production/env_vars.hcl'
dns:
- 'aws/dns/**'
- 'env/production/dns/**'
eks:
- 'aws/eks/**'
- 'env/production/eks/**'
elasticache:
- 'aws/elasticache/**'
- 'env/production/elasticache/**'
rds:
- 'aws/rds/**'
- 'env/production/rds/**'
cloudfront:
- 'aws/cloudfront/**'
- 'env/production/cloudfront/**'
lambda-api:
- 'aws/lambda-api/**'
- 'env/production/lambda-api/**'

- name: Terragrunt plan common
if: ${{ steps.filter.outputs.common == 'true }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/common"
comment-delete: "true"
comment-title: "Production: common"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan dns
if: ${{ steps.filter.outputs.dns == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/dns"
comment-delete: "true"
comment-title: "Production: dns"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"
- name: Terragrunt plan eks
if: ${{ steps.filter.outputs.eks == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/eks"
comment-delete: "true"
comment-title: "Production: eks"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan elasticache
if: ${{ steps.filter.outputs.elasticache == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/elasticache"
comment-delete: "true"
comment-title: "Production: elasticache"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan rds
if: ${{ steps.filter.outputs.rds == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/rds"
comment-delete: "true"
comment-title: "Production: rds"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan cloudfront
if: ${{ steps.filter.outputs.cloudfront == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/cloudfront"
comment-delete: "true"
comment-title: "Production: cloudfront"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan lambda-api
if: ${{ steps.filter.outputs.lambda-api == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/production/lambda-api"
comment-delete: "true"
comment-title: "Production: lambda-api"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"
160 changes: 160 additions & 0 deletions .github/workflows/terragrunt_plan_staging.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
name: "Terragrunt plan STAGING"

on:
pull_request:
paths:
- "aws/**"
- "env/staging/**"
- "env/terragrunt.hcl"
- ".github/workflows/terragrunt_plan_staging.yml"

env:
TARGET_ENV_PATH: staging
AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ca-central-1
TERRAFORM_VERSION: 0.14.4
TERRAGRUNT_VERSION: 0.31.0
TF_VAR_rds_cluster_password: ${{ secrets.STAGING_RDS_CLUSTER_PASSWORD }}
TF_VAR_cloudwatch_opsgenie_alarm_webhook: ""
TF_VAR_cloudwatch_slack_webhook_warning_topic: ${{ secrets.STAGING_CLOUDWATCH_SLACK_WEBHOOK }}
TF_VAR_cloudwatch_slack_webhook_critical_topic: ${{ secrets.STAGING_CLOUDWATCH_SLACK_WEBHOOK }}
TF_VAR_cloudwatch_slack_webhook_general_topic: ${{ secrets.STAGING_CLOUDWATCH_SLACK_WEBHOOK }}
TF_VAR_slack_channel_warning_topic: "notification-staging-ops"
TF_VAR_slack_channel_critical_topic: "notification-staging-ops"
TF_VAR_slack_channel_general_topic: "notification-staging-ops"
TF_VAR_admin_client_secret: ${{ secrets.STAGING_ADMIN_CLIENT_SECRET }}
TF_VAR_admin_client_user_name: ${{ secrets.STAGING_ADMIN_CLIENT_USER_NAME }}
TF_VAR_api_host_name: ${{ secrets.STAGING_API_HOST_NAME }}
TF_VAR_asset_domain: ${{ secrets.STAGING_ASSET_DOMAIN }}
TF_VAR_asset_upload_bucket_name: ${{ secrets.STAGING_ASSET_UPLOAD_BUCKET_NAME }}
TF_VAR_auth_tokens: ${{ secrets.STAGING_AUTH_TOKENS }}
TF_VAR_base_domain: ${{ secrets.STAGING_BASE_DOMAIN }}
TF_VAR_csv_upload_bucket_name: ${{ secrets.STAGING_CSV_UPLOAD_BUCKET_NAME }}
TF_VAR_dangerous_salt: ${{ secrets.STAGING_DANGEROUS_SALT }}
TF_VAR_documents_bucket: ${{ secrets.STAGING_DOCUMENTS_BUCKET }}
TF_VAR_document_download_api_host: ${{ secrets.STAGING_DOCUMENT_DOWNLOAD_API_HOST }}
TF_VAR_mlwr_host: "false"
TF_VAR_notification_queue_prefix: eks-notification-canada-ca
TF_VAR_redis_url: ${{ secrets.STAGING_REDIS_URL }}
TF_VAR_secret_key: ${{ secrets.STAGING_SECRET_KEY }}
TF_VAR_sqlalchemy_database_reader_uri: ${{ secrets.STAGING_SQLALCHEMY_DATABASE_READER_URI }}
TF_VAR_sqlalchemy_database_uri: ${{ secrets.STAGING_SQLALCHEMY_DATABASE_URI }}

jobs:
terragrunt-plan-staging:
runs-on: ubuntu-latest
steps:

- name: Checkout
uses: actions/checkout@v2

- name: Setup Terraform
uses: hashicorp/setup-terraform@3d8debd658c92063839bc97da5c2427100420dec # v1.3.2
with:
terraform_version: ${{ env.TERRAFORM_VERSION }}
terraform_wrapper: false

- name: Setup Terragrunt
run: |
mkdir bin
wget -O bin/terragrunt https://github.com/gruntwork-io/terragrunt/releases/download/v$TERRAGRUNT_VERSION/terragrunt_linux_amd64
chmod +x bin/*
echo "$GITHUB_WORKSPACE/bin" >> $GITHUB_PATH
- uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721 # v2.10.2
id: filter
with:
filters: |
common:
- '.github/workflows/terragrunt-plan-staging.yml'
- 'env/common/**'
- 'env/terragrunt.hcl'
- 'env/staging/env_vars.hcl'
dns:
- 'aws/dns/**'
- 'env/staging/dns/**'
eks:
- 'aws/eks/**'
- 'env/staging/eks/**'
elasticache:
- 'aws/elasticache/**'
- 'env/staging/elasticache/**'
rds:
- 'aws/rds/**'
- 'env/staging/rds/**'
cloudfront:
- 'aws/cloudfront/**'
- 'env/staging/cloudfront/**'
lambda-api:
- 'aws/lambda-api/**'
- 'env/staging/lambda-api/**'

- name: Terragrunt plan common
if: ${{ steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/common"
comment-delete: "true"
comment-title: "Staging: common"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan dns
if: ${{ steps.filter.outputs.dns == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/dns"
comment-delete: "true"
comment-title: "Staging: dns"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan eks
if: ${{ steps.filter.outputs.eks == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/eks"
comment-delete: "true"
comment-title: "Staging: eks"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan elasticache
if: ${{ steps.filter.outputs.elasticache == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/elasticache"
comment-delete: "true"
comment-title: "Staging: elasticache"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan rds
if: ${{ steps.filter.outputs.rds == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/rds"
comment-delete: "true"
comment-title: "Staging: rds"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan cloudfront
if: ${{ steps.filter.outputs.cloudfront == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/cloudfront"
comment-delete: "true"
comment-title: "Staging: cloudfront"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"

- name: Terragrunt plan lambda-api
if: ${{ steps.filter.outputs.lambda-api == 'true' || steps.filter.outputs.common == 'true' }}
uses: cds-snc/terraform-plan@v1
with:
directory: "env/staging/lambda-api"
comment-delete: "true"
comment-title: "Staging: lambda-api"
github-token: "${{ secrets.GITHUB_TOKEN }}"
terragrunt: "true"
6 changes: 3 additions & 3 deletions aws/rds/rds.tf
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ resource "aws_rds_cluster" "notification-canada-ca" {
preferred_maintenance_window = "wed:04:00-wed:04:30"
db_subnet_group_name = aws_db_subnet_group.notification-canada-ca.name
#tfsec:ignore:AWS051 - database is encrypted without a custom key and that's fine
storage_encrypted = true
deletion_protection = true
storage_encrypted = true
deletion_protection = true
db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.default.name

vpc_security_group_ids = [
Expand Down Expand Up @@ -126,4 +126,4 @@ resource "aws_db_event_subscription" "notification-canada-ca-cluster" {

# See https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html
# We are interested in all events so leaving out the event_categories parameter
}
}
14 changes: 7 additions & 7 deletions aws/rds/rds_proxy.tf
Original file line number Diff line number Diff line change
Expand Up @@ -29,20 +29,20 @@ resource "aws_secretsmanager_secret_version" "database_user" {
################################################################################

module "rds_proxy" {
source = "clowdhaus/rds-proxy/aws"
source = "clowdhaus/rds-proxy/aws"
version = "~> 2.0"

name = "rds-proxy"
iam_auth = "DISABLED"
iam_role_name = "rds-proxy-to-secrets-role"
iam_policy_name = "rds-proxy-to-secrets-policy"
name = "rds-proxy"
iam_auth = "DISABLED"
iam_role_name = "rds-proxy-to-secrets-role"
iam_policy_name = "rds-proxy-to-secrets-policy"

idle_client_timeout = 1800
max_connections_percent = 90
require_tls = false

vpc_subnet_ids = var.vpc_private_subnets
vpc_security_group_ids = [var.eks_cluster_securitygroup]
vpc_subnet_ids = var.vpc_private_subnets
vpc_security_group_ids = [var.eks_cluster_securitygroup]

db_proxy_endpoints = {
read_write = {
Expand Down
8 changes: 2 additions & 6 deletions aws/rds/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,9 @@ variable "kms_arn" {
type = string
}

variable "rds_server_db_user" {
type = string
}

Comment on lines -9 to -12
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable is not used in the code. It was added in #282 where it was briefly used, but in the end the code that used it was deleted from the PR in 8b36b21 (it was in the secrets.tf file that was deleted in that commit)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for cleaning it up.

variable "rds_cluster_password" {
type = string
sensitive = true
type = string
sensitive = true
}

variable "rds_instance_count" {
Expand Down
Loading