Create a new MSK Nodes Data Source to allow MSK Broker Node Properties to be Retrieved

[james-bjss]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The current aws_msk_cluster resource has a bootstrap broker attribute that returns a comma delimited string of hostnames and ports; however this only returns one hostname per availability zone. Meaning in a 6 Node cluster only 3 hosts are returned.

Furthermore, the returned list is not stable and returns different hostnames on subsequent calls/refreshes.

Use Case
We would like to enumerate the all the hostnames of each broker node so we can create Consul/DNS aliases. This data source will return a list of brokers with their associated hostnames. There are other Broker properties exposed which may be useful to others such as the subnets the brokers reside in and the attached ENIs

More Info

• I considered adding this as an attribute on the existing broker, but thought that it may be useful to split this out into its own resource.
• The Go SDK docs also mention being able to return a list of ZK nodes however these do not seem to be returned when checking on the CLI. Would it be worth distinguishing the resource name?
• The datasource is plural, hopefully this is acceptable? Would we consider a singular form for single brokers?

New or Affected Resource(s)

• aws_msk_nodes

Potential Terraform Configuration

resource "aws_vpc" "vpc" {
  cidr_block = "192.168.0.0/22"
}

data "aws_availability_zones" "azs" {
  state = "available"
}

resource "aws_subnet" "subnet_az1" {
  availability_zone = data.aws_availability_zones.azs.names[0]
  cidr_block        = "192.168.0.0/24"
  vpc_id            = aws_vpc.vpc.id
}

resource "aws_subnet" "subnet_az2" {
  availability_zone = data.aws_availability_zones.azs.names[1]
  cidr_block        = "192.168.1.0/24"
  vpc_id            = aws_vpc.vpc.id
}

resource "aws_subnet" "subnet_az3" {
  availability_zone = data.aws_availability_zones.azs.names[2]
  cidr_block        = "192.168.2.0/24"
  vpc_id            = aws_vpc.vpc.id
}


resource "aws_security_group" "sg" {
  vpc_id = aws_vpc.vpc.id
}

resource "aws_msk_cluster" "test" {
  cluster_name           = "test-tf"
  kafka_version          = "2.2.1"
  number_of_broker_nodes = 3

  broker_node_group_info {
    client_subnets  = [aws_subnet.subnet_az1.id, aws_subnet.subnet_az2.id, aws_subnet.subnet_az3.id]
    ebs_volume_size = 10
    instance_type   = "kafka.t3.small"
    security_groups = [aws_security_group.sg.id]
  }

  tags = {
    foo = "bar"
  }
}

data "aws_msk_nodes" "test" {
  cluster_arn = aws_msk_cluster.test.arn
}

output "nodes" {
    value = data.aws_msk_nodes.test
}

Outputs:

cluster = "arn:aws:kafka:us-east-1:<account>:cluster/test-tf/ce9cc137-148e-483d-aca7-fa44bc0714da-16"
nodes = {
  "cluster_arn" = "arn:aws:kafka:us-east-1:<account>:cluster/test-tf/ce9cc137-148e-483d-aca7-fa44bc0714da-16"
  "id" = "arn:aws:kafka:us-east-1:<account>:cluster/test-tf/ce9cc137-148e-483d-aca7-fa44bc0714da-16"
  "nodes" = tolist([
    {
      "attached_eni_id" = "eni-08641675c74d1d191"
      "broker_id" = 1
      "client_subnet" = "subnet-0a522a3cbcdf4e576"
      "endpoints" = tolist([
        "b-1.test-tf.lzys77.c16.kafka.us-east-1.amazonaws.com",
      ])
    },
    {
      "attached_eni_id" = "eni-09417d8491dd32a29"
      "broker_id" = 2
      "client_subnet" = "subnet-0d27610f064a4324e"
      "endpoints" = tolist([
        "b-2.test-tf.lzys77.c16.kafka.us-east-1.amazonaws.com",
      ])
    },
    {
      "attached_eni_id" = "eni-0d2926b41d820b4ab"
      "broker_id" = 3
      "client_subnet" = "subnet-00d579c482c938ad0"
      "endpoints" = tolist([
        "b-3.test-tf.lzys77.c16.kafka.us-east-1.amazonaws.com",
      ])
    },
  ])
}

References

• MSK API Reference
• Go API Reference
• Bootstrap broker issue Don't list bootstrap servers as a diff when applying #19659
• PR with Proposed Changes Add New MSK Broker Node Data Resource #20615

[github-actions]

This functionality has been released in v3.60.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.