Export private IPs/ENIs of internal NLBs as attributes on the resource

[IainSteers]

We've found ourselves using internal NLBs for several services which need IP addressability, and are finding it annoyingly tedious to automate provisioning and registration via the aws terraform module.

For internal NLBs were there isn't any EIPs the entry-point to the NLB is either the DNS A record and/or going via the AWS GUI/API to manually grab the private IPs of the NLB's network interfaces.

The aws_network_interface data-source doesn't seem to help here because you need to know the ENI id first.

We lock down our security groups and don't want to just randomly open entire vpc cidrs for the sake of health checks so need the private ips for that as well as services where DNS isn't available/usable.

Terraform Version

v0.10.8
Latest released aws provider

Affected Resource(s)

• aws_lb

Terraform Configuration Files

resource "aws_lb_target_group" "nlb_tcp" {
  name     = "test-nlb-tcp"
  port     = 6356
  protocol = "TCP"
  vpc_id   = "vpc-abc134"

  health_check {
    protocol = "TCP"
    port     = "5588"

    healthy_threshold   = 3
    unhealthy_threshold = 3
    interval            = 10
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb_listener" "test" {
  load_balancer_arn = "${aws_lb.test_nlb.arn}"

  port     = "516"
  protocol = "TCP"

  default_action {
    target_group_arn = "${aws_lb_target_group.nlb_tcp.arn}"
    type             = "forward"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_lb" "test_nlb" {
  name     = "${var.mod_name}-nlb"
  internal = true

  load_balancer_type = "network"

  subnets = [
    "subnet-abc1234",
    "subnet-xyz1234",
  ]
}

Is there any possibility of either getting the ENI ids or private IPs exported as attributes of the lb resource?

[bflad]

Probably worth noting there is already a PR (pending discussion) of exporting the private IPs of aws_lb here: #2901

[saxonww]

We ran into this earlier today, and there is an existing workaround using aws_network_interface data source:

variable "vpc_id" {
  description = "The AWS VPC ID"
}

variable "subnets" {
  type = "list"
  description = "A list of subnet IDs in the above VPC"
}

resource "aws_lb" "example_lb" {
  load_balancer_type = "network"
  name               = "example-lb"
  internal           = true
  subnets            = ["${var.subnets}"]
}

data "aws_network_interface" "example_lb" {
  count = "${length(var.subnets)}"

  filter = {
    name   = "description"
    values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
  }

  filter = {
    name   = "subnet-id"
    values = ["${element(var.subnets, count.index)}"]
  }
}

resource "aws_security_group" "lb_sg" {
  vpc_id = "${var.vpc_id}"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "tcp"
    cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.example_lb.*.private_ips))}"]
    description = "Permit connection from load balancer"
  }
}

[IainSteers]

@saxonww yes that's the obvious work-around.

However, it's hardly efficient nor first-class and gets crufty quickly.

[saxonww]

@IainSteers I agree, I mostly put it here so that other people encountering this issue could find it. Your original submission read (to me) like you were saying the data source would not work; sorry if I misunderstood.

[omerfsen]

variable "vpc_id" {
  description = "The AWS VPC ID"
}

variable "subnets" {
  type = "list"
  description = "A list of subnet IDs in the above VPC"
}

resource "aws_lb" "example_lb" {
  load_balancer_type = "network"
  name               = "example-lb"
  internal           = true
  subnets            = ["${var.subnets}"]
}

data "aws_network_interface" "example_lb" {
  count = "${length(var.subnets)}"

  filter = {
    name   = "description"
    values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
  }

  filter = {
    name   = "subnet-id"
    values = ["${element(var.subnets, count.index)}"]
  }
}

resource "aws_security_group" "lb_sg" {
  vpc_id = "${var.vpc_id}"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "tcp"
    cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.example_lb.*.private_ips))}"]
    description = "Permit connection from load balancer"
  }
}

This solution does not work as if you use NLB SGs you have created (Yes NLBs can't have SG as of yet) as source it only adds Assets that is using this SG is added namely

When you specify a security group as a source, it allows traffic from the instances associated with the security group. It does not add the rules from the source security group.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-rules

So instead of creating an SG for NLB you just add following to your target SG that you do health check against

  cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.privatelink.*.private_ips))}"]

So sample one would be:

# Add NLB Internal IPs to Instance's SGs (We allow ALL traffic from NLB not just the port/s that are healthchecked) 
resource "aws_security_group_rule" "target_service_sg" {
  type                      = "ingress"
  from_port                 = 0
  to_port                   = 0
  protocol                  = "-1"
  cidr_blocks = ["${formatlist("%s/32",flatten(data.aws_network_interface.privatelink.*.private_ips))}"]
  security_group_id         = "${var.target_service_sg_id}"
}

[nbouscal]

For anyone still dealing with this issue while the PR linked above is in purgatory, to maybe save you a bit of time rewriting the aws_network_interface solution for 0.12, here's an updated example corresponding to what I'm now using in 0.12.13:

data "aws_network_interface" "example_lb" {
  for_each = var.subnets

  filter {
    name   = "description"
    values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
  }

  filter {
    name   = "subnet-id"
    values = [each.value]
  }
}

resource "aws_security_group" "lb_sg" {
  vpc_id = var.vpc_id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "tcp"
    cidr_blocks = formatlist("%s/32", [for eni in data.aws_network_interface.example_lb : eni.private_ip])
    description = "Permit connection from load balancer"
  }
}

(I'm not an HCL expert, this is just what I got to work, feel free to comment if it could be done better.)

[80kk]

Any ideas how to use that for target group attachment which expects IPs rather than eni IDs?

[maryelizbeth]

Hi Y’all,

Per the conversation on #2901, we’ll work with the author of #11404 to address any changes that may be needed to merge the PR and will merge it once the work is complete.

As a result, we will close this issue in order to focus conversation on #11404.

If you feel that #11404 does not adequately address your workflow, please open a new GitHub Issue describing the gap. If you have concerns regarding the design of the upstream API, please reach out to AWS.

We apologize for letting this linger without response and will work to merge #11404 in an upcoming release.

Once that is merged, it will close #11403 & #11044.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!