Bug: ASO SubscriptionRequestsThrottled

[MZODEH-se]

Version of Azure Service Operator
v2.0.0-beta.0

Describe the bug
We deployed ASO with a managed identity associated to the AAD pod identity. Aso has been deployed with Helm Chart and only
one Pod. is created
Aso is used to provision azure resources needed by couple of microservices.
Sometimes when trying to deploy resources with ASO, we are facing this error:

Number of write requests for subscription '' exceeded the limit of '1200' for time interval '01:00:00'. Please try again after '303' seconds.: -------------------------------------------------------------------------------- RESPONSE 429: 429 Too Many Requests ERROR CODE: SubscriptionRequestsThrottled -------------------------------------------------------------------------------- { "error": { "code": "SubscriptionRequestsThrottled"

To Reproduce
When deploying azure resources from AKS cluster using Azure service operator
Expected behavior
Capability to create as much as needed resources without being block by this throtteling.

Additional context
The request limit is set on the managed identity user by ASO. As workaround for the moment, we are deleting the "blacklisted" managed identity and creating a new one to unblock the deployment.
Can you please tell me if there is a way to handle this request limit .
Thanks in advance.
Michael

[theunrepentantgeek]

The request limit is set on the managed identity user by ASO.

The throttling you're seeing originates with Azure Resource Manager (ARM), and it's something we've recently had reported by other users as well (e.g. #2590). ASO will eventually achieve your desired goal state as it does retry - the throttling will only delay this.

Currently ASO reconciles resources by using a PUT each time, relying on ARM to do the right thing for achieving the goal state. It's on our roadmap (see #1491) to switch to a different approach, where we'll do a GET on each reconciliation and only a PUT when required. (GET requests have a much higher threshold for ARM throttling.)

In the short term, our upcoming beta.4 release includes a change (#2578) to increase the default syncPeriod from 15m to 1h, which will help avoid throttling.

As an immediate workaround, try explicitly configuring the syncPeriod to a longer interval - see our FAQ for a little more information.

[MZODEH-se]

Thank you for your answer.
As per you suggested we will upgrade to vesion beta.4.

BR,
Michael

[matthchr]

beta.4 isn't released yet, the latest release is beta.3 which doesn't have the default syncPeriod change yet.

You can manually configure the syncPeriod yourself by following the instructions in our FAQ. In beta.4 (release ETA end of Nov) the default will be 1h.

[MZODEH-se]

Ok We will wait for the -beta.4 release then.
And regarding the usage of GET rather than PUT to perfom the reconcile you confirm that this feature will be available in the -beta.5 version?
Thank you for your help.

[theunrepentantgeek]

We haven't yet finished the design for using GET vs PUT, though this is underway.

We've not yet decided whether this will be in the beta.5 release or not; that decision will be made most release of beta.4.