Add support for ALB routing with specific application health checks

Update the original `lb_listener_rules` module, that hasn't been used so far,
to support a more specific case, where we want to add a target group and listener
rule for that target based on host headers. Each target group has its own health
check, so we can ensure traffic is only routing when the application is up and
healthy.

We are testing this module with the backend LB. We can choose when to use the
application healthchecks and target groups with the `enable_lb_app_healthchecks`
variable. By default the LBs continue using the default target group. When
we set `enable_lb_app_healthchecks` to true, we are creating the LB forward rules
from the service cnames variable.

This expression:
```
rules_host = ["${compact(split(",", var.enable_lb_app_healthchecks ? join(",", var.backend_public_service_cnames) : ""))}"]
```

is required because we can't use a conditional with list variables:
hashicorp/terraform#12453

This is fixed in Terraform 0.12, until then we need to implement a workaround.

terraform/modules/aws/lb_listener_rules/README.md

@@ -3,54 +3,32 @@
 This module creates Load Balancer listener rules and target groups for
 an existing listener resource.
 
-You can specify rules based on host header with the `rules_host` variable,
-path pattern with the `rules_path` variable, or both with the `rules_host_and_path`
-variable. Rules from the three variables are merged and prioritised in the order
-`rules_host_and_path`, `rules_host` and `rules_path`.
-
-The three variables are map types. The values of the maps define the target group
-port and protocol where requests are routed when they meet the rule condition, with
-the format TARGET_GROUP_PROTOCOL:TARGET_GROUP_PORT.
-
-The keys of `rules_host` are evaluated against the Host header of the request. The
-keys of `rules_path` are evaluated against the path of the request. If the
-`rules_host_and_path` variable is provided, the key has the format FIELD:VALUE.
-FIELD must be one of 'path-pattern' for path based routing or 'host-header' for host
-based routing.
-
-```
-rules_host {
-  "www.example1.com" = "HTTP:8080"
-  "www.example2.com" = "HTTPS:9091"
-  "www.example3.*"   = "HTTP:8080"
-}
-
-rules_host_and_path {
- "host-header:www.example1.com" = "HTTP:8080"
- "host-header:www.example2.com" = "HTTPS:9091"
- "path-pattern:/example3"       = "HTTPS:9091"
-}
-```
-
 Limitations:
  - The target group deregistration_delay, health_check_interval and health_check_timeout
 values can be configured with variables, but will be the same for all the target groups
  - With Terraform we can't provide a 'count' or list for listener_rule condition blocks,
 so at the moment only one condition can be specified per rule
+ - At the moment this module only implements Host Header based rules
 
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|:----:|:-----:|:-----:|
+| autoscaling_group_name | Name of ASG to associate with the target group. | string | - | yes |
+| default_tags | Additional resource tags | map | `<map>` | no |
 | listener_arn | ARN of the listener. | string | - | yes |
-| name | Prefix of the target group names. The final name is name-PROTOCOL-PORT. | string | - | yes |
-| rules_host | A map with the value of a host-header rule condition and the target group associated. | map | `<map>` | no |
-| rules_host_and_path | A map with the value of a rule with the format FIELD:VALUE and the target group associated. FIELD can be one of 'host-header' or 'path-pattern' | map | `<map>` | no |
-| rules_path | A map with the value of a path-pattern rule condition and the target group associated | map | `<map>` | no |
+| name | Prefix of the target group names. The final name is name-rulename. | string | - | yes |
+| priority_offset | first priority number assigned to the rules managed by the module. | string | `1` | no |
+| rules_host | A list with the values to create Host-header based listener rules and target groups. | list | `<list>` | no |
+| rules_host_domain | Host header domain to append to the hosts in rules_host. | string | `*` | no |
 | target_group_deregistration_delay | The amount time for Elastic Load Balancing to wait before changing the state of a deregistering target from draining to unused. | string | `300` | no |
 | target_group_health_check_interval | The approximate amount of time, in seconds, between health checks of an individual target. Minimum value 5 seconds, Maximum value 300 seconds. | string | `30` | no |
+| target_group_health_check_matcher | The HTTP codes to use when checking for a successful response from a target. | string | `200-399` | no |
+| target_group_health_check_path_prefix | The prefix destination for the health check request. | string | `/_healthcheck-` | no |
 | target_group_health_check_timeout | The amount of time, in seconds, during which no response means a failed health check. | string | `5` | no |
+| target_group_port | The port on which targets receive traffic. | string | `80` | no |
+| target_group_protocol | The protocol to use for routing traffic to the targets. | string | `HTTP` | no |
 | vpc_id | The ID of the VPC in which the default target groups are created. | string | - | yes |
 
 ## Outputs

terraform/modules/aws/lb_listener_rules/main.tf

@@ -4,75 +4,58 @@
 * This module creates Load Balancer listener rules and target groups for
 * an existing listener resource.
 *
-* You can specify rules based on host header with the `rules_host` variable,
-* path pattern with the `rules_path` variable, or both with the `rules_host_and_path`
-* variable. Rules from the three variables are merged and prioritised in the order
-* `rules_host_and_path`, `rules_host` and `rules_path`.
-*
-* The three variables are map types. The values of the maps define the target group
-* port and protocol where requests are routed when they meet the rule condition, with
-* the format TARGET_GROUP_PROTOCOL:TARGET_GROUP_PORT.
-*
-* The keys of `rules_host` are evaluated against the Host header of the request. The
-* keys of `rules_path` are evaluated against the path of the request. If the 
-* `rules_host_and_path` variable is provided, the key has the format FIELD:VALUE.
-* FIELD must be one of 'path-pattern' for path based routing or 'host-header' for host
-* based routing.
-*
-*```
-* rules_host {
-*   "www.example1.com" = "HTTP:8080"
-*   "www.example2.com" = "HTTPS:9091"
-*   "www.example3.*"   = "HTTP:8080"
-* }
-*
-* rules_host_and_path {
-*  "host-header:www.example1.com" = "HTTP:8080"
-*  "host-header:www.example2.com" = "HTTPS:9091"
-*  "path-pattern:/example3"       = "HTTPS:9091"
-* }
-*```
-*
 * Limitations:
 *  - The target group deregistration_delay, health_check_interval and health_check_timeout
 * values can be configured with variables, but will be the same for all the target groups
 *  - With Terraform we can't provide a 'count' or list for listener_rule condition blocks,
 * so at the moment only one condition can be specified per rule
+*  - At the moment this module only implements Host Header based rules
 */
 
+variable "default_tags" {
+  type        = "map"
+  description = "Additional resource tags"
+  default     = {}
+}
+
 variable "listener_arn" {
   type        = "string"
   description = "ARN of the listener."
 }
 
 variable "rules_host" {
-  type        = "map"
-  description = "A map with the value of a host-header rule condition and the target group associated."
-  default     = {}
+  type        = "list"
+  description = "A list with the values to create Host-header based listener rules and target groups."
+  default     = []
 }
 
-variable "rules_path" {
-  type        = "map"
-  description = "A map with the value of a path-pattern rule condition and the target group associated"
-  default     = {}
+variable "rules_host_domain" {
+  type        = "string"
+  description = "Host header domain to append to the hosts in rules_host."
+  default     = "*"
 }
 
-variable "rules_host_and_path" {
-  type        = "map"
-  description = "A map with the value of a rule with the format FIELD:VALUE and the target group associated. FIELD can be one of 'host-header' or 'path-pattern'"
-  default     = {}
+variable "name" {
+  type        = "string"
+  description = "Prefix of the target group names. The final name is name-rulename."
 }
 
-variable "name" {
+variable "priority_offset" {
   type        = "string"
-  description = "Prefix of the target group names. The final name is name-PROTOCOL-PORT."
+  description = "first priority number assigned to the rules managed by the module."
+  default     = 1
 }
 
 variable "vpc_id" {
   type        = "string"
   description = "The ID of the VPC in which the default target groups are created."
 }
 
+variable "autoscaling_group_name" {
+  type        = "string"
+  description = "Name of ASG to associate with the target group."
+}
+
 variable "target_group_deregistration_delay" {
   type        = "string"
   description = "The amount time for Elastic Load Balancing to wait before changing the state of a deregistering target from draining to unused."
@@ -91,56 +74,74 @@ variable "target_group_health_check_timeout" {
   default     = 5
 }
 
-# Resources
-#--------------------------------------------------------------
+variable "target_group_port" {
+  type        = "string"
+  description = "The port on which targets receive traffic."
+  default     = 80
+}
 
-locals {
-  hosts = "${zipmap(formatlist("%s:%s", "host-header", keys(var.rules_host)), values(var.rules_host))}"
-  paths = "${zipmap(formatlist("%s:%s", "path-pattern", keys(var.rules_path)), values(var.rules_path))}"
-  rules = "${merge(var.rules_host_and_path, local.hosts, local.paths)}"
+variable "target_group_protocol" {
+  type        = "string"
+  description = "The protocol to use for routing traffic to the targets."
+  default     = "HTTP"
 }
 
-locals {
-  target_groups = "${distinct(values(local.rules))}"
+variable "target_group_health_check_path_prefix" {
+  type        = "string"
+  description = "The prefix destination for the health check request."
+  default     = "/_healthcheck-"
+}
+
+variable "target_group_health_check_matcher" {
+  type        = "string"
+  description = "The HTTP codes to use when checking for a successful response from a target."
+  default     = "200-399"
 }
 
+# Resources
+#--------------------------------------------------------------
+
 resource "aws_lb_target_group" "tg" {
-  count                = "${length(local.target_groups)}"
-  name                 = "${var.name}-${replace(element(local.target_groups, count.index), ":", "-")}"
-  port                 = "${element(split(":", element(local.target_groups, count.index)), 1)}"
-  protocol             = "${element(split(":", element(local.target_groups, count.index)), 0)}"
+  count                = "${length(var.rules_host)}"
+  name                 = "${format("%.10s-%.21s", var.name, var.rules_host[count.index])}"
+  port                 = "${var.target_group_port}"
+  protocol             = "${var.target_group_protocol}"
   vpc_id               = "${var.vpc_id}"
   deregistration_delay = "${var.target_group_deregistration_delay}"
 
   health_check {
     interval            = "${var.target_group_health_check_interval}"
-    path                = "/"
-    matcher             = "200-499"
+    path                = "${var.target_group_health_check_path_prefix}${var.rules_host[count.index]}"
+    matcher             = "${var.target_group_health_check_matcher}"
     port                = "traffic-port"
-    protocol            = "${element(split(":", element(local.target_groups, count.index)), 0)}"
+    protocol            = "${var.target_group_protocol}"
     healthy_threshold   = 2
     unhealthy_threshold = 2
     timeout             = "${var.target_group_health_check_timeout}"
   }
+
+  tags = "${var.default_tags}"
 }
 
-locals {
-  target_groups_arns = "${zipmap(aws_lb_target_group.tg.*.name, aws_lb_target_group.tg.*.arn)}"
+resource "aws_autoscaling_attachment" "tg" {
+  count                  = "${length(var.rules_host)}"
+  autoscaling_group_name = "${var.autoscaling_group_name}"
+  alb_target_group_arn   = "${aws_lb_target_group.tg.*.arn[count.index]}"
 }
 
 resource "aws_lb_listener_rule" "routing" {
-  count        = "${length(keys(local.rules))}"
+  count        = "${length(var.rules_host)}"
   listener_arn = "${var.listener_arn}"
-  priority     = "${count.index + 1}"
+  priority     = "${count.index + var.priority_offset}"
 
   action {
     type             = "forward"
-    target_group_arn = "${lookup(local.target_groups_arns, "${var.name}-${replace(element(values(local.rules), count.index), ":", "-")}")}"
+    target_group_arn = "${aws_lb_target_group.tg.*.arn[count.index]}"
   }
 
   condition {
-    field  = "${element(split(":", element(keys(local.rules), count.index)), 0)}"
-    values = ["${element(split(":", element(keys(local.rules), count.index)), 1)}"]
+    field  = "host-header"
+    values = ["${var.rules_host[count.index]}.${var.rules_host_domain}"]
   }
 }
 

terraform/projects/infra-public-services/README.md

@@ -58,6 +58,7 @@ This project adds global resources for app components:
 | elb_public_secondary_certname | The ACM secondary cert domain name to find the ARN of | string | - | yes |
 | email_alert_api_internal_service_names |  | list | `<list>` | no |
 | email_alert_api_public_service_names |  | list | `<list>` | no |
+| enable_lb_app_healthchecks | Use application specific target groups and healthchecks based on the list of services in the cname variable. | string | `false` | no |
 | feedback_public_service_names |  | list | `<list>` | no |
 | frontend_internal_service_cnames |  | list | `<list>` | no |
 | frontend_internal_service_names |  | list | `<list>` | no |

terraform/projects/infra-public-services/main.tf

@@ -38,6 +38,12 @@ variable "app_stackname" {
   default     = "blue"
 }
 
+variable "enable_lb_app_healthchecks" {
+  type        = "string"
+  description = "Use application specific target groups and healthchecks based on the list of services in the cname variable."
+  default     = false
+}
+
 variable "apt_public_service_names" {
   type    = "list"
   default = []
@@ -518,6 +524,7 @@ module "backend_public_lb" {
 resource "aws_lb_listener_rule" "backend_alb_blocked_host_headers" {
   count        = "${length(var.backend_alb_blocked_host_headers)}"
   listener_arn = "${element(module.backend_public_lb.load_balancer_ssl_listeners, 0)}"
+  priority     = "${count.index + 1}"
 
   action {
     type             = "fixed-response"
@@ -535,6 +542,18 @@ resource "aws_lb_listener_rule" "backend_alb_blocked_host_headers" {
   }
 }
 
+module "backend_public_lb_rules" {
+  source                 = "../../modules/aws/lb_listener_rules"
+  name                   = "backend"
+  autoscaling_group_name = "${data.aws_autoscaling_groups.backend.names[0]}"
+  rules_host_domain      = "${var.aws_environment}.*"
+  vpc_id                 = "${data.terraform_remote_state.infra_vpc.vpc_id}"
+  listener_arn           = "${module.backend_public_lb.load_balancer_ssl_listeners[0]}"
+  rules_host             = ["${compact(split(",", var.enable_lb_app_healthchecks ? join(",", var.backend_public_service_cnames) : ""))}"]
+  priority_offset        = "${length(var.backend_alb_blocked_host_headers) + 1}"
+  default_tags           = "${map("Project", var.stackname, "aws_migration", "backend", "aws_environment", var.aws_environment)}"
+}
+
 resource "aws_route53_record" "backend_public_service_names" {
   count   = "${length(var.backend_public_service_names)}"
   zone_id = "${data.terraform_remote_state.infra_root_dns_zones.external_root_zone_id}"