New-AzureRmDataLakeStoreAccount + KeyVault throws: The service principal of the Data Lake Store account [account name] does not have access to the Key Vault [key vault id]

[cmendible]

Cmdlet(s)

New-AzureRmDataLakeStoreAccount

PowerShell Version

5.1.14393.1198

Module Version

4.2.0

OS Version

10.0.14393.1198

Description

We have a script which creates a Key Vault a Key Vault Key and then a Data Lake Account encrypted with the Key Vault.

The script looks like this:

$keyVault = New-AzureRmKeyVault `
    -VaultName $vaultName `
    -ResourceGroupName  $resourceGroupName `
    -Location $resourceGroupLocation

$key = Add-AzureKeyVaultKey `
    -VaultName $vaultName `
    -Name "DataLakeKey" `
    -KeyFilePath $dataLakeCertFile `
    -KeyFilePassword $secDataLakeCertPassword

$dataLakeStore = New-AzureRmDataLakeStoreAccount `
            -ResourceGroupName $resourceGroupName `
            -Name $dataLakeStoreName `
            -Location $resourceGroupLocation `
            -Encryption "UserManaged" `
            -KeyVaultId $keyVault.ResourceId `
            -KeyName $key.Name `
            -KeyVersion $key.Version

Set-AzureRmKeyVaultAccessPolicy `
        -VaultName $vaultName `
        -ObjectId (Get-AzureRmADServicePrincipal -SearchString "RN_$dataLakeStoreName").Where( {$_.DisplayName -eq "RN_$dataLakeStoreName"}).Id `
        -PermissionsToKeys encrypt, decrypt, get

# Enable the keyvault for the data lake
Enable-AzureRmDataLakeStoreKeyVault -ResourceGroupName $resourceGroupName -Account $dataLakeStoreName

So first we create the Vault, then the Data Lake which in turns creates the RN_$dataLakeStoreName service principal and finally add this principal to the Vault Policy and enable the Vault.

This works without issues in version 4.1.0 of the AzureRM module, but once you run it with version 4.2.0 the line calling New-AzureRmDataLakeStoreAccount throws:

The service principal of the Data Lake Store account [data lake name] does not have access to the Key Vault [key vault id]

The new version is trying to validate the service principal which is created with the datalake so it seems to be a "race condition". May be it's related to this new feature:

Added a quality of life update to automatically trigger an enableKeyVault call when a user managed KeyVault is added or a key is rotated.

Let me know if you need more info.

Thanks!

[cormacpayne]

@begoldsm Hey Ben, would you mind taking a look at this issue?

[begoldsm]

That is indeed the case. @cmendible it looks like we missed a scenario where the service principal to use doesn't already exist during initial setup of the Data Lake store account (in that you define it's permissions after account creation). To work around this for now you can either:

• Revert back to the previous version of the ADLS cmdlets
• Configure permissions with a pre-existing key vault SPN that already has access to the key vault.

Given your script and scenario I would recommend the first option. We will have this change reverted ASAP and it will be in for the next release.

[begoldsm]

@cmendible also, the account is successfully created (even with the failure) I believe. So you could do the following:

• After account creation "failure":
  • Run the Set-AzureRMKeyVaultAccessPolicy cmdlet
  • Re-Run Enable-AzureRMDataLakeStoreKeyVault cmdlet.

[cmendible]

Hi @begoldsm,

I was actually working on a video showing the script running and didn't want any red messages on it jejejeje.

I ended up reverting back to the previous version of the ADLS cmdlets.

I'll keep watching the issue so as soon as you release the fix I'll give it try and come to you guys with feedback.