add timeout to IMDS when used in DefaultAzureCredential

[bmc-msft]

The DefaultAzureCredential iterates through multiple methods for obtaining a credential until one succeeds (or all fail).

Currently, the default is to check the Environment, IMDS, then the CLI.

In many non-Azure workloads (such as MSFT corp computers), the IMDS token request takes an exceedingly long time to fail. This makes using DefaultAzureCredential without disabling IMDS on corp machines painfully slow.

This PR adds a timeout of 1 second to IMDS, per research of @johnbatty (see below).

[rylev]

I think this makes a lot of sense!

[cataggar]

I’d like to review this more. The order should match other implementations lie .NET.

[cataggar]

.NET Azure.Identity.DefaultAzureCredentials has this order:

• EnvironmentCredential
• ManagedIdentityCredential
• SharedTokenCacheCredential
• VisualStudioCredential
• VisualStudioCodeCredential
• AzureCliCredential
• AzurePowerShellCredential
• InteractiveBrowserCredential

Java com.azure.identity.DefaultAzureCredential has this order:

• EnvironmentCredential
• ManagedIdentityCredential
• SharedTokenCacheCredential
• IntelliJCredential
• VisualStudioCodeCredential
• AzureCliCredential
• AzurePowerShellCredential
• Fails if none of the credentials above could be created.

It would be nice if all credential options failed fast if unavailable. I think our ImdsManagedIdentityCredential is actually a combination of these two source types in .NET:

• AppServiceV2019ManagedIdentitySource
• ImdsManagedIdentitySource

There are multiple sources for ManagedIdentity. I think we need to split them up.

        private static ManagedIdentitySource SelectManagedIdentitySource(ManagedIdentityClientOptions options)
        {
            return
                ServiceFabricManagedIdentitySource.TryCreate(options) ??
                AppServiceV2019ManagedIdentitySource.TryCreate(options) ??
                AppServiceV2017ManagedIdentitySource.TryCreate(options) ??
                CloudShellManagedIdentitySource.TryCreate(options) ??
                AzureArcManagedIdentitySource.TryCreate(options) ??
                TokenExchangeManagedIdentitySource.TryCreate(options) ??
                new ImdsManagedIdentitySource(options);
        }

FYI, the difference between the two AppService:

The AppServiceManagedIdentitySource validates the environment variables to see if the identity is available & fail fast if not:

        protected static bool TryValidateEnvVars(string msiEndpoint, string secret, out Uri endpointUri)
        {
            endpointUri = null;
            // if BOTH the env vars endpoint and secret values are null, this MSI provider is unavailable.
            // Also validate that IdentityServerThumbprint is null or empty to differentiate from Service Fabric.
            if (string.IsNullOrEmpty(msiEndpoint) || string.IsNullOrEmpty(secret))
            {
                return false;
            }

Perhaps we can make ImdsManagedIdentityCredential opt in if one of the environment variables is set or a combination of.

[yvespp]

What's the timeout of the IMDS request? Maybe that could be shortened to a few seconds?

[johnbatty]

I also just hit this - using DefaultAzureCredential on my local dev machine hangs for over 2 mins (135 secs) attempting to access the IMDS endpoint, before then continuing and successfully getting a token via AzureCliCredential.

A bit of searching shows that the other SDKs seem to have a short default timeout when trying to access IMDS from within DefaultAzureCredential, but NOT if directly using ImdsManagedIdentityCredential:

• Go SDK (1 sec timeout):
  • Enable custom IMDS timeout value azure-sdk-for-go#19238
  • More discussion here, highlighting that the shorter timeout is only be used when IMDS is accessed via DefaultAzureCredential (note that this PR is not currently accepted/merged):
    • IMDS timeout increased to 10 seconds azure-sdk-for-go#19389
  • This PR mentions that the shorter timeout is only used on the first connection attempt, and removed for subsequent attempts:
    • Refactor IMDS discovery to remove probing, stop caching failures azure-sdk-for-go#16267
• .NET SDK (1 sec timeout)
  • Recent issue complaining about IMDS timeout being reduced from 3s to 1s:
    • [BUG] IMDS token endpoint timeout was changed from 3s to 1s without consulting the IMDS IdentityProvider plugin owners azure-sdk-for-net#31452
  • PR changing timeout from 3s to 1s:
    • Constrain NetworkTimeout on IMDS calls from DefaultAzureCredential azure-sdk-for-net#24328

Seems like we should align with the behaviour of the Go and .NET SDKs here, and add a 1 sec timeout for first connection when used via DefaultAzureCredential, rather than changing the ordering of the credential providers.

[bmc-msft]

I've updated the PR to use futures-time to add a 1 second timeout to using IMS managed identity token fetch. which aligns with @johnbatty's analysis.

[bmc-msft]

Originally, the update was implemented using @yoshuawuyts 's futures-time crate, but rather than expanding on our dependencies, this reimplements the Timeout future from said crate and uses azure_core::sleep::Sleep for the timer.

[cataggar]

This looks like a good approach. Thank you both.