MSK list_nodes ClientVpcIpAddress support

[carl34]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Example:

output "broker_client_vpc_ip_address" {
  description = "Broker IP Addresses"
  value       "${aws_msk_cluster.example.broker_client_vpc_ip_addresses}"
}

broker_client_vpc_ip_addresses = 10.0.0.1,10.0.0.2,10.0.0.3

Closes #11085

Release note for CHANGELOG:

Add list_nodes support to the aws_msk_cluster resource with a new output containing a comma separated list of all ClientVpcIpAddress values for a cluster.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSMskCluster_NumberOfBrokerNodes'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -count 1 -parallel 20 -run=TestAccAWSMskCluster_NumberOfBrokerNodes -timeout 120m
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSMskCluster_NumberOfBrokerNodes
=== PAUSE TestAccAWSMskCluster_NumberOfBrokerNodes
=== CONT  TestAccAWSMskCluster_NumberOfBrokerNodes
--- PASS: TestAccAWSMskCluster_NumberOfBrokerNodes (1357.48s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws   1358.889s
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/flatmap  0.471s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/keyvaluetags 0.346s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/naming   0.345s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/service/batch/equivalency    0.502s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/service/eks/token    0.093s [no tests to run]
?       github.com/terraform-providers/terraform-provider-aws/awsproviderlint   [no test files]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/awsproviderlint/passes    0.374s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/awsproviderlint/passes/AWSAT001   0.229s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/awsproviderlint/passes/AWSR001    0.302s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/awsproviderlint/passes/fmtsprintfcallexpr 0.371s [no tests to run]

...

[carl34]

Pushed candidate unit test updates, but they have not yet been run. Working out details on running testacc the the available environment. Please let me know if there are any suggestions.

[carl34]

Unit tests updated, output from TestAccAWSMskCluster_NumberOfBrokerNodes included above.

[DrFaust92]

some comments, ill take a look regarding cluster expansion and readiness (maybe some wait logic is needed here)

[carl34]

@bflad @DrFaust92 thanks for the guidance on this effort, please let me know if there are any new comments.

[carl34]

Please let me know if any steps remain before this code can be merged.

[dblooman]

Can we get this merged in so we can have a bumper MSK release

[sgLancelot]

Supporting this pull request. My current work around is doing my head in.

[maheshmadpathi]

Supporting this PR.

[rattboi]

Could really use this right now. Supporting this pr.

[carl34]

Thanks for the support @bflad @DrFaust92 @dblooman @sgLancelot @maheshmadpathi @rattboi, I fixed the merge conflicts from when SASL/SCRAM was added.

[carl34]

Squashed commits

[carl34]

Requesting a maintainer review please.

[ewbankkit]

@carl34 Thanks for the contribution 🎉 👏.
I think the best solution for this will be a new aws_msk_nodes data source as implemented in #20615.

[carl34]

Thanks @ewbankkit for the reference to #20615, the additional resource is a great addition.

Is the list of clientVpcIpAddress values for a cluster or individual brokers present in a branch or planned to be made available to address #11085 (comment)? An example use-case where it would be convenient to have the full list of clientVpcIpAddress values at the cluster level is the creation of rr-dns entries which include all brokers in a cluster.

Using the BootstrapBrokers bootstrapBrokerString property and looking up the individual IP addresses is one option, but this property includes one broker per AZ rather than the full list.

If the endpoints property in #20615 has the hostname for each broker in a cluster, could a similar property with the list of all clientVpcIpAddress values be added?

If anyone has found a solution to the referenced issue without requiring external scripts, please let us know.

[carl34]

Thanks @ewbankkit for adding client_vpc_ip_address per #20615 (comment), 60e8f5d

With this addition I am hoping that there is a way to obtain a list of all cluster IP addresses which would satisfy #11085 (comment) for use as records in https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route53_record.

For example, given the variable that was planned in this PR,

output "broker_client_vpc_ip_address" {
  description = "Broker IP Addresses"
  value       "${aws_msk_cluster.example.broker_client_vpc_ip_addresses}"
}

broker_client_vpc_ip_addresses = 10.0.0.1,10.0.0.2,10.0.0.3

An A record like the following could be created,

resource "aws_route53_record" "kafka" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "kafka"
  type    = "A"
  ttl     = "300"
  records = [${aws_msk_cluster.example.broker_client_vpc_ip_addresses}]
}

I don't have a current TF dev environment available and I'm not sure how to perform the join using aws_msk_broker_nodes and client_vpc_ip_address with TF, if it is not too much trouble could we post an example of how to accomplish this with a wildcard, loop, or some other operation? Would something along these lines be feasible to obtain the desired list?

resource "aws_route53_record" "kafka" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "kafka"
  type    = "A"
  ttl     = "300"
  records = [${aws_msk_broker_nodes.example.*.client_vpc_ip_address}]
}

[ewbankkit]

@carl34 You can use a Terraform for expression to loop over the node_info_list attribute:

data "aws_msk_broker_nodes" "example" {
  cluster_arn = "..."
}

output "client_vpc_ip_addresses" {
  value = join(",", [for node in data.aws_msk_broker_nodes.example.node_info_list : node.client_vpc_ip_address])
}

% terraform apply
...
Outputs:

client_vpc_ip_addresses = "192.168.1.52,192.168.0.248,192.168.2.11"

[carl34]

@carl34 You can use a Terraform for expression to loop over the node_info_list attribute:

data "aws_msk_broker_nodes" "example" {
  cluster_arn = "..."
}

output "client_vpc_ip_addresses" {
  value = join(",", [for node in data.aws_msk_broker_nodes.example.node_info_list : node.client_vpc_ip_address])
}

% terraform apply
...
Outputs:

client_vpc_ip_addresses = "192.168.1.52,192.168.0.248,192.168.2.11"

Great, thank you very much @ewbankkit.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.