Add Kafka support to IOT topic rules

[jalavoy]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #17506.
Supersedes #18639.
Closes #11940.
Closes #19904.
Closes #12520.
Closes #19099.
Closes #15763.

I'm having collisions with my production AWS configuration on my dev machine, so I'm unable to run these successfully using the test suite. However, loading the provider into my production terraform repo does properly produce the resource and otherwise looks like it works. Someone with a more appropriate dev environment for this project I believe will have passing tests.

Special thanks to @toddraddigan for the initial build of this from #18639. I took his code and updated it to the standards the code has today, and fixed a bug with the ack option not applying correctly.

[ewbankkit]

LGTM 🚀.

Commercial

% make testacc TESTARGS='-short -run=TestAccIoTTopicRule_' PKG=iot ACCTEST_PARALLELISM=3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iot/... -v -count 1 -parallel 3  -short -run=TestAccIoTTopicRule_ -timeout 180m
=== RUN   TestAccIoTTopicRule_basic
=== PAUSE TestAccIoTTopicRule_basic
=== RUN   TestAccIoTTopicRule_disappears
=== PAUSE TestAccIoTTopicRule_disappears
=== RUN   TestAccIoTTopicRule_tags
=== PAUSE TestAccIoTTopicRule_tags
=== RUN   TestAccIoTTopicRule_cloudWatchAlarm
=== PAUSE TestAccIoTTopicRule_cloudWatchAlarm
=== RUN   TestAccIoTTopicRule_cloudWatchLogs
=== PAUSE TestAccIoTTopicRule_cloudWatchLogs
=== RUN   TestAccIoTTopicRule_cloudWatchMetric
=== PAUSE TestAccIoTTopicRule_cloudWatchMetric
=== RUN   TestAccIoTTopicRule_dynamoDB
=== PAUSE TestAccIoTTopicRule_dynamoDB
=== RUN   TestAccIoTTopicRule_dynamoDBv2
=== PAUSE TestAccIoTTopicRule_dynamoDBv2
=== RUN   TestAccIoTTopicRule_elasticSearch
=== PAUSE TestAccIoTTopicRule_elasticSearch
=== RUN   TestAccIoTTopicRule_firehose
=== PAUSE TestAccIoTTopicRule_firehose
=== RUN   TestAccIoTTopicRule_Firehose_separator
=== PAUSE TestAccIoTTopicRule_Firehose_separator
=== RUN   TestAccIoTTopicRule_http
=== PAUSE TestAccIoTTopicRule_http
=== RUN   TestAccIoTTopicRule_IoT_analytics
=== PAUSE TestAccIoTTopicRule_IoT_analytics
=== RUN   TestAccIoTTopicRule_IoT_events
=== PAUSE TestAccIoTTopicRule_IoT_events
=== RUN   TestAccIoTTopicRule_kafka
    topic_rule_test.go:861: skipping long-running test in short mode
--- SKIP: TestAccIoTTopicRule_kafka (0.00s)
=== RUN   TestAccIoTTopicRule_kinesis
=== PAUSE TestAccIoTTopicRule_kinesis
=== RUN   TestAccIoTTopicRule_lambda
=== PAUSE TestAccIoTTopicRule_lambda
=== RUN   TestAccIoTTopicRule_republish
=== PAUSE TestAccIoTTopicRule_republish
=== RUN   TestAccIoTTopicRule_republishWithQos
=== PAUSE TestAccIoTTopicRule_republishWithQos
=== RUN   TestAccIoTTopicRule_s3
=== PAUSE TestAccIoTTopicRule_s3
=== RUN   TestAccIoTTopicRule_sns
=== PAUSE TestAccIoTTopicRule_sns
=== RUN   TestAccIoTTopicRule_sqs
=== PAUSE TestAccIoTTopicRule_sqs
=== RUN   TestAccIoTTopicRule_Step_functions
=== PAUSE TestAccIoTTopicRule_Step_functions
=== RUN   TestAccIoTTopicRule_Timestream
=== PAUSE TestAccIoTTopicRule_Timestream
=== RUN   TestAccIoTTopicRule_errorAction
=== PAUSE TestAccIoTTopicRule_errorAction
=== RUN   TestAccIoTTopicRule_updateKinesisErrorAction
=== PAUSE TestAccIoTTopicRule_updateKinesisErrorAction
=== CONT  TestAccIoTTopicRule_basic
=== CONT  TestAccIoTTopicRule_IoT_events
=== CONT  TestAccIoTTopicRule_dynamoDBv2
--- PASS: TestAccIoTTopicRule_basic (18.51s)
=== CONT  TestAccIoTTopicRule_IoT_analytics
--- PASS: TestAccIoTTopicRule_IoT_events (24.76s)
=== CONT  TestAccIoTTopicRule_http
--- PASS: TestAccIoTTopicRule_dynamoDBv2 (32.81s)
=== CONT  TestAccIoTTopicRule_Firehose_separator
--- PASS: TestAccIoTTopicRule_IoT_analytics (31.11s)
=== CONT  TestAccIoTTopicRule_firehose
--- PASS: TestAccIoTTopicRule_firehose (27.26s)
=== CONT  TestAccIoTTopicRule_elasticSearch
--- PASS: TestAccIoTTopicRule_http (54.29s)
=== CONT  TestAccIoTTopicRule_sns
--- PASS: TestAccIoTTopicRule_Firehose_separator (49.06s)
=== CONT  TestAccIoTTopicRule_updateKinesisErrorAction
--- PASS: TestAccIoTTopicRule_elasticSearch (34.06s)
=== CONT  TestAccIoTTopicRule_errorAction
--- PASS: TestAccIoTTopicRule_sns (34.99s)
=== CONT  TestAccIoTTopicRule_Timestream
--- PASS: TestAccIoTTopicRule_updateKinesisErrorAction (48.09s)
=== CONT  TestAccIoTTopicRule_Step_functions
--- PASS: TestAccIoTTopicRule_errorAction (27.22s)
=== CONT  TestAccIoTTopicRule_sqs
--- PASS: TestAccIoTTopicRule_Timestream (27.11s)
=== CONT  TestAccIoTTopicRule_tags
--- PASS: TestAccIoTTopicRule_Step_functions (35.33s)
=== CONT  TestAccIoTTopicRule_cloudWatchAlarm
--- PASS: TestAccIoTTopicRule_sqs (33.43s)
=== CONT  TestAccIoTTopicRule_republish
--- PASS: TestAccIoTTopicRule_tags (39.12s)
=== CONT  TestAccIoTTopicRule_s3
--- PASS: TestAccIoTTopicRule_cloudWatchAlarm (34.45s)
=== CONT  TestAccIoTTopicRule_republishWithQos
--- PASS: TestAccIoTTopicRule_s3 (25.49s)
=== CONT  TestAccIoTTopicRule_dynamoDB
--- PASS: TestAccIoTTopicRule_republish (34.25s)
=== CONT  TestAccIoTTopicRule_cloudWatchMetric
--- PASS: TestAccIoTTopicRule_republishWithQos (34.26s)
=== CONT  TestAccIoTTopicRule_lambda
--- PASS: TestAccIoTTopicRule_cloudWatchMetric (33.97s)
=== CONT  TestAccIoTTopicRule_disappears
--- PASS: TestAccIoTTopicRule_dynamoDB (38.81s)
=== CONT  TestAccIoTTopicRule_kinesis
--- PASS: TestAccIoTTopicRule_lambda (16.31s)
=== CONT  TestAccIoTTopicRule_cloudWatchLogs
--- PASS: TestAccIoTTopicRule_disappears (12.48s)
--- PASS: TestAccIoTTopicRule_kinesis (25.89s)
--- PASS: TestAccIoTTopicRule_cloudWatchLogs (26.63s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/iot	280.594s

GovCloud

% make testacc TESTARGS='-short -run=TestAccIoTTopicRule_' PKG=iot ACCTEST_PARALLELISM=3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iot/... -v -count 1 -parallel 3  -short -run=TestAccIoTTopicRule_ -timeout 180m
=== RUN   TestAccIoTTopicRule_basic
=== PAUSE TestAccIoTTopicRule_basic
=== RUN   TestAccIoTTopicRule_disappears
=== PAUSE TestAccIoTTopicRule_disappears
=== RUN   TestAccIoTTopicRule_tags
=== PAUSE TestAccIoTTopicRule_tags
=== RUN   TestAccIoTTopicRule_cloudWatchAlarm
=== PAUSE TestAccIoTTopicRule_cloudWatchAlarm
=== RUN   TestAccIoTTopicRule_cloudWatchLogs
=== PAUSE TestAccIoTTopicRule_cloudWatchLogs
=== RUN   TestAccIoTTopicRule_cloudWatchMetric
=== PAUSE TestAccIoTTopicRule_cloudWatchMetric
=== RUN   TestAccIoTTopicRule_dynamoDB
=== PAUSE TestAccIoTTopicRule_dynamoDB
=== RUN   TestAccIoTTopicRule_dynamoDBv2
=== PAUSE TestAccIoTTopicRule_dynamoDBv2
=== RUN   TestAccIoTTopicRule_elasticSearch
=== PAUSE TestAccIoTTopicRule_elasticSearch
=== RUN   TestAccIoTTopicRule_firehose
=== PAUSE TestAccIoTTopicRule_firehose
=== RUN   TestAccIoTTopicRule_Firehose_separator
=== PAUSE TestAccIoTTopicRule_Firehose_separator
=== RUN   TestAccIoTTopicRule_http
=== PAUSE TestAccIoTTopicRule_http
=== RUN   TestAccIoTTopicRule_IoT_analytics
=== PAUSE TestAccIoTTopicRule_IoT_analytics
=== RUN   TestAccIoTTopicRule_IoT_events
=== PAUSE TestAccIoTTopicRule_IoT_events
=== RUN   TestAccIoTTopicRule_kafka
    topic_rule_test.go:871: skipping long-running test in short mode
--- SKIP: TestAccIoTTopicRule_kafka (0.00s)
=== RUN   TestAccIoTTopicRule_kinesis
=== PAUSE TestAccIoTTopicRule_kinesis
=== RUN   TestAccIoTTopicRule_lambda
=== PAUSE TestAccIoTTopicRule_lambda
=== RUN   TestAccIoTTopicRule_republish
=== PAUSE TestAccIoTTopicRule_republish
=== RUN   TestAccIoTTopicRule_republishWithQos
=== PAUSE TestAccIoTTopicRule_republishWithQos
=== RUN   TestAccIoTTopicRule_s3
=== PAUSE TestAccIoTTopicRule_s3
=== RUN   TestAccIoTTopicRule_sns
=== PAUSE TestAccIoTTopicRule_sns
=== RUN   TestAccIoTTopicRule_sqs
=== PAUSE TestAccIoTTopicRule_sqs
=== RUN   TestAccIoTTopicRule_Step_functions
=== PAUSE TestAccIoTTopicRule_Step_functions
=== RUN   TestAccIoTTopicRule_Timestream
=== PAUSE TestAccIoTTopicRule_Timestream
=== RUN   TestAccIoTTopicRule_errorAction
=== PAUSE TestAccIoTTopicRule_errorAction
=== RUN   TestAccIoTTopicRule_updateKinesisErrorAction
=== PAUSE TestAccIoTTopicRule_updateKinesisErrorAction
=== CONT  TestAccIoTTopicRule_basic
=== CONT  TestAccIoTTopicRule_updateKinesisErrorAction
=== CONT  TestAccIoTTopicRule_IoT_analytics
    acctest.go:1018: skipping test for aws-us-gov/us-gov-west-1: Error running apply: exit status 1
        
        Error: creating IoT Topic Rule (tf_acc_test_0xidvjnmpl8h2r16r8bj): InvalidRequestException: The iotanalytics service has(have) not been supported in region us-gov-west-1. 
        
          with aws_iot_topic_rule.test,
          on terraform_plugin_test.tf line 44, in resource "aws_iot_topic_rule" "test":
          44: resource "aws_iot_topic_rule" "test" {
        
--- SKIP: TestAccIoTTopicRule_IoT_analytics (10.44s)
=== CONT  TestAccIoTTopicRule_errorAction
--- PASS: TestAccIoTTopicRule_basic (17.31s)
=== CONT  TestAccIoTTopicRule_Timestream
    acctest.go:1018: skipping test for aws-us-gov/us-gov-west-1: Error running apply: exit status 1
        
        Error: creating IoT Topic Rule (tf_acc_test_zfzeopvec2u2hqk0ijlb): InvalidRequestException: The timestream service has(have) not been supported in region us-gov-west-1. 
        
          with aws_iot_topic_rule.test,
          on terraform_plugin_test.tf line 44, in resource "aws_iot_topic_rule" "test":
          44: resource "aws_iot_topic_rule" "test" {
        
--- SKIP: TestAccIoTTopicRule_Timestream (8.85s)
=== CONT  TestAccIoTTopicRule_Step_functions
--- PASS: TestAccIoTTopicRule_errorAction (27.27s)
=== CONT  TestAccIoTTopicRule_sqs
--- PASS: TestAccIoTTopicRule_updateKinesisErrorAction (43.58s)
=== CONT  TestAccIoTTopicRule_sns
--- PASS: TestAccIoTTopicRule_Step_functions (27.61s)
=== CONT  TestAccIoTTopicRule_s3
--- PASS: TestAccIoTTopicRule_sqs (29.55s)
=== CONT  TestAccIoTTopicRule_kinesis
--- PASS: TestAccIoTTopicRule_sns (28.92s)
=== CONT  TestAccIoTTopicRule_IoT_events
--- PASS: TestAccIoTTopicRule_s3 (28.05s)
=== CONT  TestAccIoTTopicRule_republishWithQos
--- PASS: TestAccIoTTopicRule_kinesis (29.88s)
=== CONT  TestAccIoTTopicRule_republish
--- PASS: TestAccIoTTopicRule_IoT_events (27.63s)
=== CONT  TestAccIoTTopicRule_dynamoDB
--- PASS: TestAccIoTTopicRule_republishWithQos (30.30s)
=== CONT  TestAccIoTTopicRule_http
--- PASS: TestAccIoTTopicRule_republish (30.72s)
=== CONT  TestAccIoTTopicRule_Firehose_separator
--- PASS: TestAccIoTTopicRule_dynamoDB (43.26s)
=== CONT  TestAccIoTTopicRule_firehose
--- PASS: TestAccIoTTopicRule_http (61.48s)
=== CONT  TestAccIoTTopicRule_elasticSearch
--- PASS: TestAccIoTTopicRule_Firehose_separator (50.47s)
=== CONT  TestAccIoTTopicRule_dynamoDBv2
--- PASS: TestAccIoTTopicRule_firehose (35.44s)
=== CONT  TestAccIoTTopicRule_cloudWatchAlarm
--- PASS: TestAccIoTTopicRule_elasticSearch (38.77s)
=== CONT  TestAccIoTTopicRule_cloudWatchMetric
--- PASS: TestAccIoTTopicRule_dynamoDBv2 (34.97s)
=== CONT  TestAccIoTTopicRule_cloudWatchLogs
--- PASS: TestAccIoTTopicRule_cloudWatchAlarm (39.15s)
=== CONT  TestAccIoTTopicRule_tags
--- PASS: TestAccIoTTopicRule_cloudWatchMetric (37.97s)
=== CONT  TestAccIoTTopicRule_disappears
--- PASS: TestAccIoTTopicRule_cloudWatchLogs (37.08s)
=== CONT  TestAccIoTTopicRule_lambda
--- PASS: TestAccIoTTopicRule_disappears (16.93s)
--- PASS: TestAccIoTTopicRule_lambda (23.68s)
--- PASS: TestAccIoTTopicRule_tags (58.05s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/iot	279.720s

[ewbankkit]

@jalavoy @dbasedow @toddradigan @nikhil-goenka Thanks for the contribution 🎉 👏.

[ewbankkit]

Running golangci-lint manually:

% make golangci-lint
==> Checking source code with golangci-lint...

[github-actions]

This functionality has been released in v4.13.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.