aws: disable rate limits by default with AWS #3496
Comments
stanhu
added a commit
to stanhu/go-cloud
that referenced
this issue
Oct 9, 2024
Issuing many S3 uploads in quick succession causes the following error: operation error S3: PutObject, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested Disable this to restore the behavior of SDK v1. Relates to google#3496
stanhu
added a commit
to stanhu/go-cloud
that referenced
this issue
Oct 10, 2024
Issuing many S3 uploads in quick succession causes the following error: operation error S3: PutObject, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested By default, this rate limit is disabled to restore the same behavior as SDK v1. If `rate_limiter_capacity` is set, then client-side rate limits will take effect. Closes google#3496
stanhu
added a commit
to stanhu/go-cloud
that referenced
this issue
Oct 10, 2024
Issuing many S3 uploads in quick succession causes the following error: operation error S3: PutObject, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested By default, this rate limit is disabled to restore the same behavior as SDK v1. If `rate_limiter_capacity` is set, then client-side rate limits will take effect. Closes google#3496
stanhu
added a commit
to stanhu/go-cloud
that referenced
this issue
Oct 10, 2024
Issuing many S3 uploads in quick succession causes the following error: operation error S3: PutObject, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested By default, this rate limit is disabled to restore the same behavior as SDK v1. If `rate_limiter_capacity` is set, then client-side rate limits will take effect. Closes google#3496
|
@vangent Thank you for the quick review and merge! Do you know when you might want to cut a new tag? Otherwise I'll just use the pseudo-version. |
|
done |
|
@vangent Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
After upgrading to go-cloud v0.39.0, one of our users reported:
Terraform also has more details in hashicorp/terraform-provider-aws#36024 (comment). It seems the AWS team discussed this specific issue in aws/aws-sdk-go-v2#1665 (comment).
The bottom line is that SDK v2 adds new client rate limits, and users may hit this after switching from SDK v1. I recommend disabling these rate limits by default (as done in https://github.com/hashicorp/aws-sdk-go-base/pull/977/files) and provide a way to configure them if desired.
To Reproduce
I haven't tried this yet, but I imagine if you make lots of PutObject requests in quick succession you will hit these client rate limits.
Expected behavior
No rate limit errors.
Version
Which version(s) of the Go Cloud modules are you seeing the bug with?
v0.39.0
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: