aws_acm_info gets ThrottlingException error - believe it needs "backoff logic" applied to other modules

[jrglynn2]

Summary

Sometimes when running this module I get the error code: ThrottlingException and message: Rate exceeded. I believe this is a known issue, and has been fixed in other modules.

Current solution is to wait, and hope it works when re-run.

Thanks!

Issue Type

Bug Report

Component Name

aws_acm_info

Ansible Version

$ ansible --version
Ansible Full Version:  5.6.0
Ansible-Core Version:  ansible [core 2.12.4]

Collection Versions

$ ansible-galaxy collection list
community.aws    3.2.1

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.18.57
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /home/ec2-user/.local/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.21.57
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /home/ec2-user/.local/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

- name: Get Certificate Information
  delegate_to: localhost
  environment:
    AWS_PROFILE: "{{ aws_profile_used }}"
  community.aws.aws_acm_info:
    #domain_name: "{{ url_name + '.' + route53_zone }}"
    certificate_arn: "{{ acm_arn }}"
    region: "{{ region }}"
  register: acm_info

Expected Results

Retrieve information without getting throttled

Actual Results

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.ThrottlingException: An error occurred (ThrottlingException) when calling the ListTagsForCertificate operation (reached max retries: 4): Rate exceeded
fatal: [SERVER1 -> localhost]: FAILED! => changed=false
  boto3_version: 1.18.57
  botocore_version: 1.21.57
  error:
    code: ThrottlingException
    message: Rate exceeded
  message: Rate exceeded
  msg: 'Couldn''t obtain tags for domain SERVER1.com: An error occurred (ThrottlingException) when calling the ListTagsForCertificate operation (reached max retries: 4): Rate exceeded'
  response_metadata:
    http_headers:
      connection: close
      content-length: '58'
      content-type: application/x-amz-json-1.1
      date: Tue, 02 Aug 2022 09:09:55 GMT
      x-amzn-requestid: 07abff12-2721-4b24-a834-b3668c503dde
    http_status_code: 400
    max_attempts_reached: true
    request_id: 07abff12-2721-4b24-a834-b3668c503dde
    retry_attempts: 4

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:
None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[tremble]

It looks like ACM already has the retry code applied to it (it retried 4 times). Unfortunately it looks like ACM has very aggressive limits applied to it, so this is going to remain painful.

As part of ansible-collections/amazon.aws#946 I've bumped the max retries up to 10 and applied "jittered" backoff (which adds a little bit of randomness and improves things a little when API queries are being run in parallel). That's about all we can do I'm afraid.