Feature: Self throttling for ARM PUT requests

[theunrepentantgeek]

In theory, ASO could keep track of the number of ARM PUT requests it has issued and proactively delay further requests that would push it over the threshold for ARM throttling.

For example, taking a limit of 1200 PUTs per hour, we could keep track of the number of PUT requests made over the last 5 minutes and self-throttle to keep under 20 per minute.

Given our multitenancy support, we'd need to keep track per-subscription.

Possible extensions

• If it's possible to query ARM for the current limit, we could do that and dynamically adjust our request rate to dynamically balance with other activity on the same subscription.
• Are there scenarios where users might want to explicitly configure the ASO request rate limit?

This isn't necessarily obsoleted by #1491, as even with that in place we would have a risk of throttling after someone applies a large YAML file containing many resources.

[matthchr]

Throttling is 1200 per subscription per connection. I've seen guidance that we increase the number of possible connections per host to get access to significantly more throughput.

The suggestions I've seen:

1. Use multiple connections/connection pool (I believe can be configured in Go HTTP, see the MaxIdleConnsPerHost property.
2. Set ForceAttemptHTTP2 to false

If ForceAttemptHTTP2 is true (the default), connections using HTTP2 will be multiplexed over a single TCP connection, which means still hitting a single ARM FE. See golang/go#39302 for details about disabling HTTP2 (though we may not need to disable it, just not force it?).

More reading:

• Tune http.Client's Transport's MaxIdleConns/MaxIdleConnsPerHost OJ/gobuster#127
• default http client has way too low MaxIdleConnsPerHost azure-sdk-for-go#19192

[theunrepentantgeek]

Less of an issue now that we've switched to using multiple connections. We'll keep this in the backlog for now, but we should reassess the need for this in a few months if we've had no reports of throttling.

[matthchr]

We've decided we're not going to do this, and will instead pursue #3487, which we believe should also alleviate this issue but in a "better" way.