Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented Retry Policy #521

Merged
merged 23 commits into from
Nov 26, 2024
Merged

Conversation

4gust
Copy link
Collaborator

@4gust 4gust commented Oct 21, 2024

Pull Request: Implement Retry Policy for Client

Overview

This pull request introduces a retry policy for network calls in the client. The retry policy aims to enhance the resilience of our application by allowing it to automatically retry failed requests up to a maximum of three times.

Changes Made

  • Retry Policy Implementation:

    • A retry mechanism has been added to handle transient errors in network calls.
    • The default maximum retry count is set to 3.
  • Configuration Option to Disable Retry Policy:

    • Introduced an optional method WithRetryPolicyDisabled() for the client.
    • When this method is used, the maximum retry count is reduced to 1 (effectively disabling the retry mechanism).

Default Behavior (Retries Enabled)

By default, the client will automatically retry failed requests up to 3 times with delay of 1 second. Here’s how to create the client with the retry policy enabled:

client, err := mi.New(SystemAssigned())
if err != nil {
    // Handle error
}

Default Behavior (Retries Disabled)

By default, the client will not retry the failed request. Here’s how to create the client with the retry policy disabled:

client, err := mi.New(SystemAssigned(), WithRetryPolicyDisabled())
if err != nil {
    // Handle error
}

Error List when retry is done.

	http.StatusNotFound,                      // 404
	http.StatusGone,                          // 410
	http.StatusTooManyRequests,               // 429 // retry after.
	http.StatusInternalServerError,           // 500
	http.StatusNotImplemented,                // 501
	http.StatusBadGateway,                    // 502
	http.StatusServiceUnavailable,            // 503
	http.StatusGatewayTimeout,                // 504
	http.StatusHTTPVersionNotSupported,       // 505
	http.StatusVariantAlsoNegotiates,         // 506
	http.StatusInsufficientStorage,           // 507
	http.StatusLoopDetected,                  // 508
	http.StatusNotExtended,                   // 510
	http.StatusNetworkAuthenticationRequired, // 511

@4gust 4gust requested review from AndyOHart and chlowell and removed request for rayluo October 21, 2024 20:23
Copy link
Collaborator

@chlowell chlowell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a spec for this feature? It seems very basic e.g. no backoff, hardcoded retry count and delay. Blockers are

  • implicit upgrade to Go 1.21
  • cancellation
  • rewinding request bodies

apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
4gust added 2 commits October 22, 2024 23:16
Added test to check request body is same for each request.
@4gust
Copy link
Collaborator Author

4gust commented Oct 23, 2024

It is very basic retry, and has a fix 1 second between retry and max count of 3 retries
This is what we set for Private preview.

@4gust 4gust requested a review from chlowell October 23, 2024 16:51
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
Updating the error code for retry
@4gust 4gust requested a review from chlowell October 29, 2024 11:53
Updated the comment
@bgavrilMS
Copy link
Member

bgavrilMS commented Nov 5, 2024

Is there a spec for this feature? It seems very basic e.g. no backoff, hardcoded retry count and delay. Blockers are

  • implicit upgrade to Go 1.21
  • cancellation
  • rewinding request bodies

This is the retry policy that was discussed with MSI team and implemented in all MSALs. If you feel that this is wrong, let's pls open bugs on all MSALs. But it is important for MSALs to be consistent here.

Note that MSALs do not offer a DefaultAzureCredential chaining, so the usage scenario is different. Azure SDK can and should disable the retry policy so as to introduce your own.

@chlowell
Copy link
Collaborator

chlowell commented Nov 6, 2024

I'm sorry, I still don't know which status codes this policy is supposed to retry on. Right now, its retriable codes are reasonable in general cases but incorrect for IMDS. Do you intend it to apply IMDS retry guidance to all services?

4gust added 2 commits November 6, 2024 19:02
Removed the condition and copied no matter the body size.
Added a context cancel
@4gust 4gust requested a review from chlowell November 13, 2024 11:12
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity_test.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
apps/managedidentity/managedidentity.go Outdated Show resolved Hide resolved
@4gust 4gust merged commit 949376f into andyohart/managed-identity Nov 26, 2024
2 checks passed
@4gust 4gust deleted the 4gust/mi-retry-policy branch November 26, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants