-
Notifications
You must be signed in to change notification settings - Fork 860
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IMDS timeout increased to 10 seconds #19389
Conversation
In case of stressed servers 1000ms is very aggressive time. This PR is to just trying to get some attention in to this intermittent context deadline issue
Thank you for your contribution shashanksbs! We will review the pull request and get back to you soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this is timely! We (Azure SDK authentication team) recently began discussing ways to improve IMDS timeout behavior for DefaultAzureCredential. Simply lengthening the timeout is one of the options but would regress performance in some scenarios, and I don't want to make any changes here until we've explored other options. Note that only DefaultAzureCredential imposes this timeout. If it's a problem in your application, you can avoid it by using ManagedIdentityCredential directly or with ChainedTokenCredential.
To give some context on the timeout, we added it to limit the worst case running time of DefaultAzureCredential's first auth attempt. This is important when IMDS isn't available, for example in local development. The only way to learn whether IMDS is available is to try connecting to it, which can take longer than 10 seconds to time out on some platforms.
@chlowell @jhendrixMSFT @kenegozi Pod Identity can take up to a couple minutes to get ready for token requests from new pods. In case of clients that have shorter timeouts, the retries can be terminated and the client will not receive a token in the first attempt. This feature flag can use to prevent timeouts due to that delay. we added https://azure.github.io/aad-pod-identity/docs/configure/feature_flags/#set-retry-after-header-in-nmi-response this check in azure pod identity helm chart. But IMDS timeout issue still persists and sometimes we are getting no default identity is assigned to this resource error. these errors occur mostly on stressed servers, on normal day it works fine Please suggest any resolution for this |
The short timeout that you're changing here is only part of |
Hi @realDeveloper45. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
Hi @realDeveloper45. Thank you for your contribution. Since there hasn't been recent engagement, we're going to close this out. Feel free to respond with a comment containing "/reopen" if you'd like to continue working on these changes. Please be sure to use the command to reopen or remove the "no-recent-activity" label; otherwise, this is likely to be closed again with the next cleanup pass. |
In case of stressed servers 1000ms is very aggressive time. This PR is to just trying to get some attention in to this intermittent context deadline issue. Error Im facing is ManagedIdentityCredential: IMDS token request timed out error for long running jobs that makes many requests to azure secrets fetch data.