Route53 API throttling

[apollo-3]

SUMMARY

This recent change throttles Route53 API heavily. The problem is in a jitter backoff, which relies on retry_interval attribute of the module (by default 500 secs). Default max_delay in a jitter is 60 seconds, so basically the minimum value of 60 is always chosen and it hits API badly with retry delays from 0 to 60 secs. There is no way to change neither the number of retries nor max delay time. Also inability to select backoff type is also disturbing, because exponential backoff would work in our use-case, but jittered exponential backoffs - not sure

ISSUE TYPE

• Bug Report

COMPONENT NAME

route53

ANSIBLE VERSION

2.10.0

[ansibullbot]

Files identified in the description:

• [plugins/modules/route53.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/route53.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @jillr @jimbydamonk @s-hertel @tremble @wimnat
click here for bot help

[gravesm]

cc @tremble

[goneri]

Hi @tremble,

I took the freedom to assign this to you. It's just to be sure this get your attention :-).

[tremble]

Thank you for taking the time to raise this issue.

It would be helpful if you could explain what problem you're seeing with less qualitative language: "Hits the API badly" could either be interpreted as "Hits the API very frequently", or "not frequently enough". If you're seeing errors from the module including these would enable us to identify exactly what behaviour you're seeing, especially if that behaviour isn't a direct result of the retry frequency or the back-off mechanism you're referring to. Knowing what frequency of API requests you're seeing, how many requests you're seeing from each task, and how many requests of which types you're seeing hitting the API in parallel would all assist in understanding the issue.

There are 3 different mechanisms being used to contact the APIs (direct calls, using boto3's paginator and using boto3's waiters), it's quite possible that just one of these requests is problematic, especially since one of them doesn't even use the retry mechanism you're referring to.

The use of jittered back-off is based upon a recommendation from one of Amazon's engineers (https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) if you have some (repeatable) experimental evidence you could use to counter their recommendations I'd be interested in seeing it.

[tremble]

I believe the issue that was being seen was actually due to a bug and should be fixed by #564.