azidentity: azure_cli_credential panics in kubernetes pods before aad-pod-identity assigns an identity

[ryepup]

Bug Report

• import path of package in question: azure-sdk-for-go/sdk/azidentity

• SDK version: 0.13.2

• output of go version: go version go1.17.6 linux/amd64

• What happened?

We use kubernetes with aad-pod-identity to assign managed identities (MI) to our pods, and then our code uses azidentity.NewDefaultAzureCredential(nil) to create credentials.

There's a known issue with aad-pod-identity where it takes awhile to assign identities, and applications like mine will initially fail until aad-pod-identity catches up. See Azure/aad-pod-identity#181 for more, but I don't think it is relevant here.

Before aad-pod-identity has assigned the MI, my applications get panics:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x765b40]

goroutine 1 [running]:
    /builds/$MY_APP/.go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/azure_cli_credential.go:130 +0x320
    /builds/$MY_APP/.go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/azure_cli_credential.go:85 +0x48
    /builds/$MY_APP/.go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/azure_cli_credential.go:74 +0xe5
    /builds/$MY_APP/.go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/chained_token_credential.go:91 +0x45d
    /builds/$MY_APP/.go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/default_azure_credential.go:94 +0x5c

I see this when using blob storage and service bus APIs, the above is the common stack trace.

• What did you expect or want to happen?

An error to be returned that my application could handle.

• How can we reproduce it?

Call azidentity.NewDefaultAzureCredential(nil) in a k8s with aad-pod-identity before aad-pod-identity has assigned a pod.

• Anything else about my environment

This is a very low impact bug for me, k8s keeps trying to restart my applications and it self-heals. We monitor our logs for panics and this came up, so figured I'd let y'all know.

Analysis

I think the bug is here:

azure-sdk-for-go/sdk/azidentity/azure_cli_credential.go

Line 130 in df31752

if errors.As(err, &exErr); exErr.ExitCode() == 127 || strings.HasPrefix(msg, "'az' is not recognized") {

We're not checking the return value from errors.As; I think it's returning false, leaving exErr pointing to nil, and then we deference it to pull the exit code.

[chlowell]

Thanks for opening this issue! I created a PR with a fix for the panic. That doesn't address the root cause of your app's authentication failure though, which is that by default Pod Identity doesn't respond to token requests before assigning an identity to the pod. It accepts requests and just leaves the connection open, often longer than our ManagedIdentityCredential is willing to wait for a response. You could work around this in your code or let the pod crash loop until Pod Identity is ready, but I recommend instead configuring Pod Identity to respond with a Retry-After header while it works to assign identities. The Pod Identity documentation has instructions for that.