aws_msk_cluster: support Cluster expansion and Open Monitoring

[marcoreni]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes: #11215
Closes: #10553
Relates: #10673 (fix acceptance tests)

Release note for CHANGELOG:

resource/aws_msk_cluster: support cluster expansion without cluster recreation [GH-10553]
resource/aws_msk_cluster: support changes to `enhanced_monitoring` without cluster recreation
resource/aws_msk_cluster: support `open_monitoring` configuration [GH-11215]

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSMskCluster_'
...
--- PASS: TestAccAWSMskCluster_basic (911.50s)
--- PASS: TestAccAWSMskCluster_BrokerNodeGroupInfo_EbsVolumeSize (1229.32s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionInTransit_ClientBroker (875.43s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionInTransit_InCluster (942.38s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionAtRestKmsKeyArn (1005.02s)
--- PASS: TestAccAWSMskCluster_EnhancedMonitoring (1151.67s)
--- PASS: TestAccAWSMskCluster_OpenMonitoring (1300.95s)
--- PASS: TestAccAWSMskCluster_NumberOfBrokerNodes (1575.17s)
--- PASS: TestAccAWSMskCluster_Tags (876.30s)
...

Since enhanced monitoring is update via the same API call of open monitoring, this came free.

In order to make the acceptance tests pass I explicitly declared encryption_info configuration for all test templates. I don't know if it may be useful to write some notes inside the tests to specify this behavior.

I also fixed some tests & docs that were probably outdated (zookeeper_connect_string is now a list of hostnames and ports).

[r3kzi]

any news about this?

[kujon]

I believe that MSK allows you to increase, but not decrease this value. Probably due to how Kafka requires partition reassignment when the cluster make up changes.

[marcoreni]

You are right. The same applies to the broker storage size (you can expand it but you cannot decrease it). However, no special validation rules have been applied in that case, so I assume that the same can be done here.

[ewbankkit]

If a CustomizeDiff function is added then we can ForceNew if the number of broker nodes is decreased.
A similar example:
https://github.com/terraform-providers/terraform-provider-aws/blob/4acecb05959a9e9b07bea9afb87d760ea887da07/aws/resource_aws_elasticsearch_domain.go#L31-L54

[anoop2811]

Hi, is there any update on this PR getting merged?

[ghost]

Hello, we are also waiting on this PR to be merged.

[dennisotugo]

Hello, waiting for this. :)

[bflad]

This looks fantastic, thanks for this @marcoreni 🚀

Output from acceptance testing:

--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionInTransit_ClientBroker (888.98s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionInTransit_InCluster (986.84s)
--- PASS: TestAccAWSMskCluster_EncryptionInfo_EncryptionAtRestKmsKeyArn (990.02s)
--- PASS: TestAccAWSMskCluster_Tags (996.50s)
--- PASS: TestAccAWSMskCluster_basic (1047.46s)
--- PASS: TestAccAWSMskCluster_EnhancedMonitoring (1065.19s)
--- PASS: TestAccAWSMskCluster_OpenMonitoring (1134.00s)
--- PASS: TestAccAWSMskCluster_NumberOfBrokerNodes (1273.72s)
--- PASS: TestAccAWSMskCluster_BrokerNodeGroupInfo_EbsVolumeSize (1308.71s)

[ghost]

This has been released in version 2.51.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!