Unauthorized error when authenticated with UserAssignedManagedIdentity on Us Gov Cloud

[locmai]

Report

After helm install from the official chart and from after this PR: promitor/charts#36

I encounter the Operation returned an invalid status code 'Unauthorized' error while trying to query resources from AzureGraphResource after Authenticated.

Note: I have checked the identity binding (with Reader and Contributor roles of the subscription) with azure-cli image and I could az login --identity and fetch the any resources in the subscription.

Expected Behavior

After authenticated with the UserAssignedManagedIdentity, the client could query for the resources defined. In this case, I use the sample:

resourceDiscoveryGroups:
- name: container-registry-landscape
  type: ContainerRegistry

Actual Behavior

Got the below error log and the service discovery returned 500 HTTP code.

Steps to Reproduce the Problem

1. Create an AKS cluster on Azure Gov cloud subscription and enable the pod identity add-on.
2. Create the Azure User Assigned Identity, add the role assignment of Contributor, Reader and Monitoring Reader for it to the subscription.
3. Configure the Service Discovery with the label aadpodidbinding=
4. After the pod is running healthy, call the API health check with includeDendencies=true

kubectl port-forward -n default svc/promitor-agent-resource-discovery 8889
curl "http://localhost:8889/api/v1/health?includeDependencies=true"

5. Got the following result:

{
    "entries": {
        "azure-resource-graph": {
            "data": {
                "Subscription 4d47ec28-XXXX-XXXX-XXXX-XXXXXXXXXXXX": {
                    "isSuccessful": false,
                    "message": "Unable to discover resources due to an unexpected exception: Operation returned an invalid status code 'Unauthorized'"
                }
            },
            "description": "One or more subscriptions are unhealthy.",
            "duration": "00:00:01.0022206",
            "status": "Unhealthy",
            "tags": []
        }
    },
    "status": "Unhealthy",
    "totalDuration": "00:00:01.0037660"
}

Component

Resource Discovery

Version

0.4.0

Configuration

Configuration for Helm chart:

azureLandscape:
  cloud: "UsGov"
  tenantId: 5766f879-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  subscriptions:
  - 4d47ec28-XXXX-XXXX-XXXX-XXXXXXXXXXXX
azureAuthentication: 
  identity:
    binding: aad-promitor
    id: c5d63849-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  mode: UserAssignedManagedIdentity
image:
  repository: "ghcr.io/tomkerkhove/promitor-agent-resource-discovery"
  tag: "0.4.0"

resourceDiscoveryGroups:
- name: container-registry-landscape
  type: ContainerRegistry

resource-discovery-declaration.yaml configmap:

version: v1
azureLandscape:
  tenantId: 5766f879-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  subscriptions:
  - 4d47ec28-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  cloud: UsGov
resourceDiscoveryGroups:
- name: container-registry-landscape
  type: ContainerRegistry

runtime.yaml

server:
  httpPort: "88"
authentication:
  mode: UserAssignedManagedIdentity
  identityId: c5d63849-XXXX-XXXX-XXXX-XXXXXXXXXXXX
cache:
  enabled: "true"
  durationInMinutes: "5"
telemetry:
  applicationInsights:
    isEnabled: "false"
  containerLogs:
    isEnabled: "true"
  defaultVerbosity: "Error"

Logs

Unable to discover resources due to an unexpected exception: Operation returned an invalid status code 'Unauthorized'
Microsoft.Azure.Management.ResourceGraph.Models.ErrorResponseException: Operation returned an invalid status code 'Unauthorized'
at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClient.ResourcesWithHttpMessagesAsync(QueryRequest query, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.ResourceGraph.ResourceGraphClientExtensions.ResourcesAsync(IResourceGraphClient operations, QueryRequest query, CancellationToken cancellationToken)
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.<>c__DisplayClass13_0.<<QueryAsync>b__0>d.MoveNext() in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 80
--- End of stack trace from previous location where exception was thrown ---
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.<>c__DisplayClass14_0`1.<<InteractWithAzureResourceGraphAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.InteractWithAzureResourceGraphAsync[TResponse](String queryName, String query, List`1 targetSubscriptions, Func`2 interactionFunc) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 93
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.QueryAsync(String queryName, String query, List`1 targetSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 77
at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.QueryForResourcesAsync(String queryName, String query, List`1 targetSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 67
at Promitor.Agents.ResourceDiscovery.Health.AzureResourceGraphHealthCheck.ProbeSubscriptionAsync(String query, String subscription) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraphHealthCheck.cs:line 84

Platform

Microsoft Azure

Contact Details

[email protected]

[locmai]

Hi @tomkerkhove , I went down to this line

promitor/src/Promitor.Integrations.Azure/Authentication/AzureAuthenticationFactory.cs

Line 72 in c32348a

public static async Task<TokenCredentials> GetTokenCredentialsAsync(string resource, string tenantId, AzureAuthenticationInfo authenticationInfo)

for initialize the token credential that would be passed to the client and I think we should've set the AuthorityHost property for it (https://docs.microsoft.com/en-us/dotnet/api/azure.identity.tokencredentialoptions.authorityhost?view=azure-dotnet#Azure_Identity_TokenCredentialOptions_AuthorityHost) or else it would just go to https://login.microsoftonline.com ?

[gburton1]

Yeah, it looks like it needs to be this constructor instead, which provides TokenCredentialOptions:

public ManagedIdentityCredential (string clientId = default, Azure.Identity.TokenCredentialOptions options = default);

Current code will be getting the default TokenCredentialOptions, which will indeed be AuthorityHost = https://login.microsoftonline.com/, which is Azure Commercial and won't work for gov.

case AuthenticationMode.UserAssignedManagedIdentity:                    
  tokenCredential = new ManagedIdentityCredential(authenticationInfo.IdentityId);                    
  break;

https://github.com/tomkerkhove/promitor/blob/c32348a77b1d0f8e07e5e76dd8a0042b2e2a24fe/src/Promitor.Integrations.Azure/Authentication/AzureAuthenticationFactory.cs#L85..L87

[locmai]

Hi @tomkerkhove , I have opened the PR for this, I was able to successfully scrape from my AzureUSGov subscription:
#1648

I'm double-checking and updating on the PR/developers.md if I missed anything in the PR process now.

[tomkerkhove]

Thank you @locmai and @gburton1 for digging into this! I'll check the PR but this seems to be the fix indeed, thanks for investigating!

[locmai]

Thanks for the quick response! Actually I'm looking at the scraper now, so after we discovered the resources:

[05:03:57 INF] Discovered 32 resources for resource collection filtered-logic-apps-landscape.
[05:03:57 FTL] SubscriptionNotFound: The subscription '4d47ec28-1799-redacted' could not be found.

I think it's similar so 👀 on that.

[tomkerkhove]

Thanks. It could be that or permissions, but I'd expect an unauthorized there then.

It might be this: https://github.com/tomkerkhove/promitor/pull/1648/files#diff-630f87ab9d90eb937b2970b338be8e942706f90d9c248a618b158a23618d84f1R113

[tomkerkhove]

Azure Monitor uses https://github.com/tomkerkhove/promitor/pull/1648/files#diff-630f87ab9d90eb937b2970b338be8e942706f90d9c248a618b158a23618d84f1R116 but thath one looks OK though.