Azure OIDC accounts fail authentication when using Az.Accounts module 2.12.5 or later

[IsaacCalligeros95]

Severity

Low, workarounds available

Version

2023.4.*

Latest Version

I could reproduce the problem in the latest build

What happened?

When using the Azure PowerShell Modules with Az.Accounts 2.12.2 or later authentication fails at deployment time. This only occurs with Disable-AzAutoContextSave set, we cannot enable this as it would share authentication contexts between subsequent steps and deployments.

This happens across all Azure Steps with variations of the same error message depending on the step.

This issue has been raised with the Azure PowerShell team.
There are other permutations of this issue like Azure/azure-powershell#22480, the move of the Service Principal secrets in 2.13.1 may be related to this issue - Azure/azure-powershell#21593.

Reproduction

• Install the latest Az Modules Install-Module -Force -Name Az -AllowClobber -Scope AllUsers
• Run Disable-AzContextAutosave
• Connect-AzAccount -Environment AzureCloud -ApplicationId -Tenant -Subscription -FederatedToken

Error and Stacktrace

Persistence check failed. Inspect inner exception for details Could not find tenant id for provided tenant domain '<tenantId>'

More Information

There are a number of open issues with the Azure PowerShell team around this problem

• Azure OIDC login fails in Github Actions Azure/azure-powershell#22628
• Connect-AzAccount: ClientCertificateCredential authentication failed Azure/azure-powershell#22480
• Connect-AzAccount fails when using federated token with AzureChinaCloud Azure/azure-powershell#22647

Workaround

Use the OctopusDeploy/WorkerTools execution container

Or

Use PowerShell Core

Or

Use earlier versions of the Az.Accounts module, running Install-Module on Az will include the latest version of all other modules Az.Accounts Included. Version [2.12.3](https://www.powershellgallery.com/packages/Az.Accounts/2.12.3) and other modules compatible with this are required.

[benPearce1]

Additional workaround:
Change the Azure Script Step to a regular script step
Create a variable for your Azure Account
Place the following code at the top of the script, replace azure variable name with the name of your variable:

For Az PowerShell

Connect-AzAccount -Environment AzureCloud -ApplicationId $OctopusParameters["azure variable name.Client"] -Tenant $OctopusParameters["azure variable name.TenantId"] -Subscription $OctopusParameters["azure variable name.SubscriptionNumber"] -FederatedToken $OctopusParameters["azure variable name.OpenIdConnect.Jwt"] -Scope Process

For Az CLI

az login --service-principal=$OctopusParameters["azure variable name.SubscriptionNumber"] --username=$OctopusParameters["azure variable name.Client"] --tenant=$OctopusParameters["azure variable name.TenantId"] --federated-token=$OctopusParameters["azure variable name.OpenIdConnect.Jwt"]

Also the end of the script should do az logout to avoid sharing the login context between sessions

[Clare-Octopus]

Another report of this (in internal) - https://octopus.zendesk.com/agent/tickets/172067
Customer reports the PowerShell workaround does not work for them.

Having looked at the slack discussion on this (internal - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1707240979399409) we are currently trying to workout if there is another workaround.

[OzoneNZ]

Hit this issue today unfortunately with our custom worker tools container image using [email protected].

After some digging around, this comment seems to have had its solution implemented upstream in Azure.Identity which underpins Az.Accounts:

• Azure OIDC login fails in Github Actions Azure/azure-powershell#22628 (comment)

I could be wrong, but this patch appears to be what was proposed above:

• ClientAssertionCredentialOptions supports TokenCachePersistenceOptions for configuring token cache persistence. Azure/azure-sdk-for-net#43633
• https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/identity/Azure.Identity/CHANGELOG.md#1120-2024-06-17

Unfortunately that upgrade of Azure.Identity to 1.12.0 doesn't appear to have happened in the Az.Accounts module yet :(

Az.Accounts still shows as using Azure.Identity version 1.10.3 in their mainline branch: https://github.com/Azure/azure-powershell/blob/main/src/Accounts/Accounts/ChangeLog.md?plain=1#L71

Also linking this reported issue as appears to be identical: OctopusDeploy/WorkerTools#95

[Clare-Octopus]

Hey @OzoneNZ thank you for reaching out, I have informed our engineers of your comment and will let you know what they say. They seem to think this PR from Azure may help fix the issue (along with some tweaks we may have to make once its merged) but since its still in draft form I commented on the Azure issue to see if we could move it along.

Will let you know if our engineers have any comments after reading yours, thank you for detailing what you have found!

I have seen you have put a support ticket in for this too thank you, I will link it here so we can track all tickets related to this internally - https://octopus.zendesk.com/agent/tickets/197599 (internal link).

[OzoneNZ]

Cheers @Clare-Octopus! At least in my interpretation (and I could be very wrong), that draft PR seems to have been an attempt to more quickly work around the shortcomings of Azure.Identity via changes in Az.Accounts, whereas at least on the surface, that latest PR I dug up does seem to directly implement the draft PR author's wishes for a permanent fix

"Identity to implement ISupportsTokenCachePersistenceOptions for ClientAssertionCredentialOptions so that we can remove all these codes."

Crossing my fingers that it is as simple as waiting on an Az.Accounts release to absorb that patch from Azure.Identity 🤞

[OzoneNZ]

Some movement!

Azure/azure-powershell#22628 (comment)
Azure/azure-powershell#25733

[OzoneNZ]

Az.Accounts 3.0.4 out now!

Will be testing it out soon in conjunction with our worker-tools container image under Octopus Deploy to see if the above issues are all solved

[purduebretty]

I used the latest version of the workertools from Octopus Deploy Labs and it resolved the issue for me: octopuslabs/azure-workertools:2.64.0

[OzoneNZ]

Can confirm all is working as expected now for us too 👍