Keyvault client fails to get secret after migrating keyvault to another tenant

[smitshah91]

We recently migrated our subscription and all resources including keyvault to another tenant. We did setup all necessary permissions but keyvault fails to get secret with below exception.

Operation execution on the AcisRuntime failed.

Error while retrieving key vault certificate. Identifier:MetadataServiceClientAdminCertificate. Path:https://dummyPath/secrets/Dummycert. Error: System.AggregateException: One or more errors occurred. ---> System.Exception: Failed to authenticate to AAD using certificate thumbprints: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)

at Microsoft.Azure.KeyVault.KeyVaultClient.d__cb.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Microsoft.WindowsAzure.Wapd.Acis.AcisCommon.Secrets.KeyVaultHelper.d__11.MoveNext() in d:\dbs\sh\easl\0616_191330\cmd\2\src\acis\aciscommon\Secrets\KeyVaultHelper.cs:line 120

--- End of inner exception stack trace ---

at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)

at Microsoft.WindowsAzure.Wapd.Acis.AcisCommon.Secrets.KeyVaultHelper.TryGetCertificate(String path, String& base64Cert, String& errorMessage) in d:\dbs\sh\easl\0616_191330\cmd\2\src\acis\aciscommon\Secrets\KeyVaultHelper.cs:line 48

---> (Inner Exception #0) System.Exception: Failed to authenticate to AAD using certificate thumbprints: 2FDAA5F17076A0A28B2A126391513B819A95483B

at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)

at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)

at Microsoft.Azure.KeyVault.KeyVaultClient.d__cb.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at Microsoft.WindowsAzure.Wapd.Acis.AcisCommon.Secrets.KeyVaultHelper.d__11.MoveNext() in d:\dbs\sh\easl\0616_191330\cmd\2\src\acis\aciscommon\Secrets\KeyVaultHelper.cs:line 120<---

.

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.WindowsAzure.Wapd.Acis.SME.Runtime.IAcisRuntimeController.PrepareExtension(String extensionName, String packageIdentifier, String packagePath, String tempCopiedPath, String outputPath, String packageTemp, String endpointName, Boolean isLpuDisabledForExtension, Boolean isSslValidationDisabled, Boolean isPackageUpload, Boolean shouldInstallCert, Boolean shouldValidateExtension, Boolean isSecretLoadScopedToEndpoint)

at Microsoft.WindowsAzure.Wapd.Acis.SME.AcisRuntimeManager.CreateFromStorage(String extensionName, String extensionIdentifier, String endpointName, Boolean createLocalRuntime, String jobId) in d:\dbs\sh\easl\0616_191330\cmd\3\src\acis\acissme\RuntimeManager\AcisRuntimeManager.cs:line 565

at Microsoft.WindowsAzure.Wapd.Acis.SME.AcisRuntimeManager.LoadRuntime(String extensionName, String packageIdentifier, String endpointName, String jobId) in d:\dbs\sh\easl\0616_191330\cmd\3\src\acis\acissme\RuntimeManager\AcisRuntimeManager.cs:line 250

at Microsoft.WindowsAzure.Wapd.Acis.AcisJobManager.GetRuntime(String extensionName, String packageIdentifier, String endpointName, String& runtimeId, String& uniqueJobId) in d:\dbs\sh\easl\0616_191330\cmd\3\src\acis\acissme\RuntimeManager\AcisJobManager.cs:line 242

at Microsoft.WindowsAzure.Wapd.Acis.AcisJobManager.ExecuteOperation(IAcisSMEUser user, String operationId, String extensionName, String packageIdentifier, MetaEndpoint endpoint, String operationFlags, MetaOperation operation, List`1 paramValues, Int32 batchOpIndex, AcisRuntimeStats& stats, Boolean isRhcOperation) in d:\dbs\sh\easl\0616_191330\cmd\3\src\acis\acissme\RuntimeManager\AcisJobManager.cs:line 110

Once we restart all nodes it starts fetching correct authority from path and able to get token.

We tried to force refresh AAD token like below.

private async Task<Secret> GetSecretWithRefreshTokenAsync(string path)
        {
            try
            {
                return await (_keyVaultClient.Value.GetSecretAsync(path));
            }
            catch (Exception ex)
            {
                TokenCache.DefaultShared.Clear();
                _keyVaultClient = new Lazy<KeyVaultClient>(
                () => new KeyVaultClient(_aadAccessTokenHandler.Value.GetAccessTokenAsync));
                return await (_keyVaultClient.Value.GetSecretAsync(path));
            }
        }

still does not work as it looks like KeyvaultClient itself caches the authority extracted from path and I do not see a way to force clear authority cache in KeyvaultClient.

[maggiepint]

Thanks for this feedback. I have routed your request to the appropriate team for follow-up.

/cc: @schaabs

[schaabs]

@smitshah91 Thank you for the detailed report. This is a known issue #5702, and unfortunately there is no way to force the KeyVaultClient to refresh the Authority. In fact the authority for each vault is cached staticly so restarting the process is the only way to clear the cache.

This issue should be fixed in the next release of the SDK, but until then restarting the process should fix the failure you're seeing. Please let me know if this doesn't fix your issue.

[schaabs]

This issue has been fixed in This issue has been addressed and this fix has been published in v3.0.4 (https://www.nuget.org/packages/Microsoft.Azure.KeyVault/3.0.4). This fix updates the challenge class whenever it receives a 401 response with the WWW-Authenticate header. The resulting behavior when a vault is migrated from one tenant to another is that the first call to the KeyVaultClient after the migration will still call the authentication callback with the original authority, and so that call will still fail with a 401 and the user would have to handle the exception. However subsequent calls to the KeyVaultClient will pass the correct authority to the authentication callback.