I am now getting throttled on AWS api calls during the state refresh

[komealy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

terraform -version
Terraform v0.11.14

• provider.aws v2.39.0
• provider.external v1.2.0
• provider.null v2.1.2
• provider.random v2.2.1
• provider.template v2.1.2
  Your version of Terraform is out of date! The latest version
  is 0.12.16. You can update by downloading from www.terraform.io/downloads.html

Affected Resource(s)

multiple ones that run DescribeInstances and DescribeTags during the state refresh.

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

2019-11-21T08:58:24.412-0800 [DEBUG] plugin.terraform-provider-aws_v2.39.0_x4: HTTP/1.1 503 Service Unavailable

I was getting this while I was getting throttled. I no longer have the 503 responses and am no longer getting throttled.

Panic Output

none

Expected Behavior

refresh in less than 5 minutes.

Actual Behavior

Refresh is taking almost an hour due to caught throttle responses and needing to retry.

Steps to Reproduce

1. terraform apply

Important Factoids

My AWS support reps reported a dramatic increase in requests during this timeframe.

Note that the previous day is multiple runs on multiple environments. The ones in the last two hours was one single environment.

References

• #0000

[tyrken]

Strongly suspect you are (as we are) running terraform inside a container, and so hitting a bad experience from AWS's new Metadata feature. You MUST lengthen the default allowed max hop count to let the Metadata HTTP responses though the docker-bridge to the container hosted processes.

See aws/aws-sdk-go#2972 and https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#configuring-instance-metadata-service though it looks like the latest awscli needs an extra parameter to work, e.g.:

aws ec2 modify-instance-metadata-options --instance-id i-34215432543254235 --http-endpoint enabled --http-put-response-hop-limit 2

You can check your instance's current hop-limit via aws ec2 describe-instances

[justinretzolk]

Hey @komealy 👋 Thank you for taking the time to file this issue. Given the comment above and that there's been a number of AWS provider releases since you initially filed it, can you confirm whether you're still experiencing this behavior?

[komealy]

Wow this is old 😅

This no longer happens for me. We didn't have to change any ec2 settings, but newer versions of provider/terraform itself do not behave this way anymore.

We didn't update this particular CodeBuild job that still uses the old versions either. I suspect it was a backend issue inside AWS itself at the time that was returning the 503 errors.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.