Improve reading kinesis stream state

[hrist0stoichev]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Change summary

• Stop using DescribeStreamPages API as it is deprecated and start using DescribeStreamSummary along with ListShards to get information about the shards.
• Stop returning shards information to methods that don't really use it. This should save a number of calls as only 100 shards per request are returned and there is a limit for 100 transactions per second (10 for DescribeStreamPages). In some cases in which there are a lot of shards, this could lead to a timeout.
  • Until now, requests to obtain open and closed shards were made for both kinesis stream resource and data source. However, the resource never used this information so, in this MR, the resource only makes a request to fetch the Summary; thus, avoiding all subsequent requests for shards info.
• Include consumers_count in AWS Kinesis Stream Data Source as it is returned by DescribeStreamSummary API. This change is a bit unrelated to the other changes so, if necessary, I can extract it to another PR after this one is merged.

Release note for CHANGELOG:

Add `consumers_count` field to the AWS Kinesis Stream Data Source

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSKinesisStream_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSKinesisStream_ -timeout 120m
=== RUN   TestAccAWSKinesisStream_basic
=== PAUSE TestAccAWSKinesisStream_basic
=== RUN   TestAccAWSKinesisStream_createMultipleConcurrentStreams
=== PAUSE TestAccAWSKinesisStream_createMultipleConcurrentStreams
=== RUN   TestAccAWSKinesisStream_encryptionWithoutKmsKeyThrowsError
=== PAUSE TestAccAWSKinesisStream_encryptionWithoutKmsKeyThrowsError
=== RUN   TestAccAWSKinesisStream_encryption
=== PAUSE TestAccAWSKinesisStream_encryption
=== RUN   TestAccAWSKinesisStream_shardCount
=== PAUSE TestAccAWSKinesisStream_shardCount
=== RUN   TestAccAWSKinesisStream_retentionPeriod
=== PAUSE TestAccAWSKinesisStream_retentionPeriod
=== RUN   TestAccAWSKinesisStream_shardLevelMetrics
=== PAUSE TestAccAWSKinesisStream_shardLevelMetrics
=== RUN   TestAccAWSKinesisStream_enforceConsumerDeletion
=== PAUSE TestAccAWSKinesisStream_enforceConsumerDeletion
=== RUN   TestAccAWSKinesisStream_Tags
=== PAUSE TestAccAWSKinesisStream_Tags
=== RUN   TestAccAWSKinesisStream_UpdateKmsKeyId
=== PAUSE TestAccAWSKinesisStream_UpdateKmsKeyId
=== CONT  TestAccAWSKinesisStream_basic
=== CONT  TestAccAWSKinesisStream_shardLevelMetrics
=== CONT  TestAccAWSKinesisStream_UpdateKmsKeyId
=== CONT  TestAccAWSKinesisStream_encryption
=== CONT  TestAccAWSKinesisStream_createMultipleConcurrentStreams
=== CONT  TestAccAWSKinesisStream_Tags
=== CONT  TestAccAWSKinesisStream_enforceConsumerDeletion
=== CONT  TestAccAWSKinesisStream_encryptionWithoutKmsKeyThrowsError
=== CONT  TestAccAWSKinesisStream_retentionPeriod
=== CONT  TestAccAWSKinesisStream_shardCount
--- PASS: TestAccAWSKinesisStream_encryptionWithoutKmsKeyThrowsError (60.02s)
2020/10/05 16:56:57 [DEBUG] Trying to get account information via sts:GetCallerIdentity
--- PASS: TestAccAWSKinesisStream_basic (117.22s)
--- PASS: TestAccAWSKinesisStream_enforceConsumerDeletion (124.25s)
--- PASS: TestAccAWSKinesisStream_shardCount (232.51s)
--- PASS: TestAccAWSKinesisStream_retentionPeriod (248.00s)
--- PASS: TestAccAWSKinesisStream_UpdateKmsKeyId (270.69s)
--- PASS: TestAccAWSKinesisStream_createMultipleConcurrentStreams (279.40s)
--- PASS: TestAccAWSKinesisStream_shardLevelMetrics (284.66s)
--- PASS: TestAccAWSKinesisStream_Tags (304.51s)
--- PASS: TestAccAWSKinesisStream_encryption (309.90s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	311.540s

[zhelding]

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

[tacho]

@zhelding we have updated this PR to account for the refactoring in #21306

[ewbankkit]

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccKinesisStream_\|TestAccKinesisStreamDataSource_\|TestAccKinesisStreamConsumer_\|TestAccKinesisStreamConsumerDataSource_' PKG_NAME=internal/service/kinesis
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kinesis/... -v -count 1 -parallel 20 -run=TestAccKinesisStream_\|TestAccKinesisStreamDataSource_\|TestAccKinesisStreamConsumer_\|TestAccKinesisStreamConsumerDataSource_ -timeout 180m
=== RUN   TestAccKinesisStreamConsumerDataSource_basic
=== PAUSE TestAccKinesisStreamConsumerDataSource_basic
=== RUN   TestAccKinesisStreamConsumerDataSource_name
=== PAUSE TestAccKinesisStreamConsumerDataSource_name
=== RUN   TestAccKinesisStreamConsumerDataSource_arn
=== PAUSE TestAccKinesisStreamConsumerDataSource_arn
=== RUN   TestAccKinesisStreamConsumer_basic
=== PAUSE TestAccKinesisStreamConsumer_basic
=== RUN   TestAccKinesisStreamConsumer_disappears
=== PAUSE TestAccKinesisStreamConsumer_disappears
=== RUN   TestAccKinesisStreamConsumer_maxConcurrentConsumers
=== PAUSE TestAccKinesisStreamConsumer_maxConcurrentConsumers
=== RUN   TestAccKinesisStreamConsumer_exceedMaxConcurrentConsumers
=== PAUSE TestAccKinesisStreamConsumer_exceedMaxConcurrentConsumers
=== RUN   TestAccKinesisStreamDataSource_basic
=== PAUSE TestAccKinesisStreamDataSource_basic
=== RUN   TestAccKinesisStream_basic
=== PAUSE TestAccKinesisStream_basic
=== RUN   TestAccKinesisStream_disappears
=== PAUSE TestAccKinesisStream_disappears
=== RUN   TestAccKinesisStream_createMultipleConcurrentStreams
=== PAUSE TestAccKinesisStream_createMultipleConcurrentStreams
=== RUN   TestAccKinesisStream_encryptionWithoutKMSKeyThrowsError
=== PAUSE TestAccKinesisStream_encryptionWithoutKMSKeyThrowsError
=== RUN   TestAccKinesisStream_encryption
=== PAUSE TestAccKinesisStream_encryption
=== RUN   TestAccKinesisStream_shardCount
=== PAUSE TestAccKinesisStream_shardCount
=== RUN   TestAccKinesisStream_retentionPeriod
=== PAUSE TestAccKinesisStream_retentionPeriod
=== RUN   TestAccKinesisStream_shardLevelMetrics
=== PAUSE TestAccKinesisStream_shardLevelMetrics
=== RUN   TestAccKinesisStream_enforceConsumerDeletion
=== PAUSE TestAccKinesisStream_enforceConsumerDeletion
=== RUN   TestAccKinesisStream_tags
=== PAUSE TestAccKinesisStream_tags
=== RUN   TestAccKinesisStream_updateKMSKeyID
=== PAUSE TestAccKinesisStream_updateKMSKeyID
=== RUN   TestAccKinesisStream_basicOnDemand
=== PAUSE TestAccKinesisStream_basicOnDemand
=== RUN   TestAccKinesisStream_switchBetweenProvisionedAndOnDemand
=== PAUSE TestAccKinesisStream_switchBetweenProvisionedAndOnDemand
=== RUN   TestAccKinesisStream_failOnBadStreamCountAndStreamModeCombination
=== PAUSE TestAccKinesisStream_failOnBadStreamCountAndStreamModeCombination
=== CONT  TestAccKinesisStreamConsumerDataSource_basic
=== CONT  TestAccKinesisStream_encryption
=== CONT  TestAccKinesisStream_basicOnDemand
=== CONT  TestAccKinesisStreamConsumer_maxConcurrentConsumers
=== CONT  TestAccKinesisStream_enforceConsumerDeletion
=== CONT  TestAccKinesisStreamConsumer_exceedMaxConcurrentConsumers
=== CONT  TestAccKinesisStream_shardLevelMetrics
=== CONT  TestAccKinesisStream_encryptionWithoutKMSKeyThrowsError
=== CONT  TestAccKinesisStream_createMultipleConcurrentStreams
=== CONT  TestAccKinesisStream_disappears
=== CONT  TestAccKinesisStream_basic
=== CONT  TestAccKinesisStreamDataSource_basic
=== CONT  TestAccKinesisStream_updateKMSKeyID
=== CONT  TestAccKinesisStream_failOnBadStreamCountAndStreamModeCombination
=== CONT  TestAccKinesisStream_switchBetweenProvisionedAndOnDemand
=== CONT  TestAccKinesisStream_retentionPeriod
=== CONT  TestAccKinesisStream_shardCount
=== CONT  TestAccKinesisStreamConsumer_basic
=== CONT  TestAccKinesisStream_tags
=== CONT  TestAccKinesisStreamConsumerDataSource_name
--- PASS: TestAccKinesisStreamConsumer_basic (69.81s)
=== CONT  TestAccKinesisStreamConsumer_disappears
--- PASS: TestAccKinesisStreamConsumer_maxConcurrentConsumers (75.53s)
=== CONT  TestAccKinesisStreamConsumerDataSource_arn
--- PASS: TestAccKinesisStream_tags (81.39s)
--- PASS: TestAccKinesisStream_enforceConsumerDeletion (84.23s)
--- PASS: TestAccKinesisStream_basicOnDemand (84.96s)
--- PASS: TestAccKinesisStreamConsumer_exceedMaxConcurrentConsumers (85.90s)
--- PASS: TestAccKinesisStream_disappears (99.11s)
--- PASS: TestAccKinesisStreamConsumerDataSource_name (101.04s)
--- PASS: TestAccKinesisStreamConsumerDataSource_basic (120.26s)
--- PASS: TestAccKinesisStreamDataSource_basic (121.05s)
--- PASS: TestAccKinesisStream_updateKMSKeyID (122.22s)
--- PASS: TestAccKinesisStreamConsumer_disappears (61.40s)
--- PASS: TestAccKinesisStreamConsumerDataSource_arn (58.23s)
--- PASS: TestAccKinesisStream_shardLevelMetrics (137.81s)
--- PASS: TestAccKinesisStream_failOnBadStreamCountAndStreamModeCombination (140.09s)
--- PASS: TestAccKinesisStream_encryption (145.47s)
--- PASS: TestAccKinesisStream_encryptionWithoutKMSKeyThrowsError (149.48s)
--- PASS: TestAccKinesisStream_retentionPeriod (150.62s)
--- PASS: TestAccKinesisStream_basic (152.01s)
--- PASS: TestAccKinesisStream_switchBetweenProvisionedAndOnDemand (217.54s)
--- PASS: TestAccKinesisStream_createMultipleConcurrentStreams (288.96s)
--- PASS: TestAccKinesisStream_shardCount (766.41s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kinesis	770.104s

[ewbankkit]

@hrist0stoichev Thanks for the contribution 🎉 👏.
I ended up removing the consumer_count attribute as this interferes with the aws_kinesis_stream_consumer resource acceptance tests.
Please submit a new PR with this functionality if you need it.

[hrist0stoichev]

@ewbankkit thanks for reviewing and merging this one. We were building a custom version of terraform-provider-aws that contained this PR for a very long time. Now, with v3.71.0, we will be able to use the official build! :)

[github-actions]

This functionality has been released in v3.71.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.