[BUG] Enabling secret store CSI driver from Azure Portal creates null config

[zioproto]

Enabling secret store CSI driver from Azure Portal creates null config

$ az aks show -n {} -g {}  | jq .addonProfiles.azureKeyvaultSecretsProvider.config
null

This is consistent both at cluster creation or updating the Cluster Configuration later with the Azure Portal.

When enabling the same addon from the az cli you get:

$ az aks show -n {} -g {} -o json | jq .addonProfiles.azureKeyvaultSecretsProvider.config
{
  "enableSecretRotation": "true",
  "rotationPollInterval": "2m"
}

This behavior is the root cause for bug Azure/azure-cli#23051 that fails with a python TypeError because null is not expected in .addonProfiles.azureKeyvaultSecretsProvider.config

To Reproduce
Steps to reproduce the behavior:

1. Create a AKS cluster with the Azure Portal and enable the secret store CSI drive addon.
2. Run az aks show -n {} -g {} | jq .addonProfiles.azureKeyvaultSecretsProvider.config and verify the output.

Expected behavior
This bug is to clarify the expected behaviour.

• If null is a valid value, then in az aks update --enable-secret-rotation fails when secret store CSI driver is enabled from Azure Portal azure-cli#23051 the CLI should be patched to handle this.
• If null is not a valid value, the AKS API should not accept the API call from Azure Portal that leads to the null config.

Environment:

• CLI Version azure-cli 2.37.0
• Kubernetes version 1.23

[raghulmsft]

Thanks for sharing this with us. I'll work with the team to look into.

[zioproto]

@raghulmsft Looking at the Create or Update API the ManagedClusterAddonProfile is very generic because it is the skeleton for any addon.

I cant find for the specific addon azureKeyvaultSecretsProvider the REST API documentation. I want to understand if an API call with an empty config is a valid API call.

The azure-cli folks fixed this problem on their side forcing a new API call that will populate the config with the default values: Azure/azure-cli#23088

While this workaround fixes the issue for azure-cli, there are other clients consuming the API, like Terraform for example.
We need to understand if config is allowed to be empty or not.

It the config cannot be empty, the AKS API endpoint should reject the API call when a request with an empty config comes in.

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

@jenetlan, @chandraneel, @raghulmsft would you be able to assist?

Issue Details

---

Enabling secret store CSI driver from Azure Portal creates null config

$ az aks show -n {} -g {}  | jq .addonProfiles.azureKeyvaultSecretsProvider.config
null

This is consistent both at cluster creation or updating the Cluster Configuration later with the Azure Portal.

When enabling the same addon from the az cli you get:

$ az aks show -n {} -g {} -o json | jq .addonProfiles.azureKeyvaultSecretsProvider.config
{
  "enableSecretRotation": "true",
  "rotationPollInterval": "2m"
}

This behavior is the root cause for bug Azure/azure-cli#23051 that fails with a python TypeError because null is not expected in .addonProfiles.azureKeyvaultSecretsProvider.config

To Reproduce
Steps to reproduce the behavior:

1. Create a AKS cluster with the Azure Portal and enable the secret store CSI drive addon.
2. Run az aks show -n {} -g {} | jq .addonProfiles.azureKeyvaultSecretsProvider.config and verify the output.

Expected behavior
This bug is to clarify the expected behaviour.

• If null is a valid value, then in az aks update --enable-secret-rotation fails when secret store CSI driver is enabled from Azure Portal azure-cli#23051 the CLI should be patched to handle this.
• If null is not a valid value, the AKS API should not accept the API call from Azure Portal that leads to the null config.

Environment:

• CLI Version azure-cli 2.37.0
• Kubernetes version 1.23

Author:	zioproto
Assignees:	-
Labels:	bug, azure/portal
Milestone:	-

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[chandraneel]

This issue is fixed and live in all regions.

[zioproto]

Hello,
I just created an AKS cluster with the portal (AKS version 1.23.12)

I confirm the config is now correct, and it is not null anymore when creating a new cluster.

az aks show -n test -g test -o json | jq .addonProfiles.azureKeyvaultSecretsProvider.config
{
  "enableSecretRotation": "false",
  "rotationPollInterval": "2m"
}

However, if the cluster is created with secret store CSI driver disabled, and then is enabled with the portal, I get again the null config:

az aks show -n test2 -g test2 -o json | jq .addonProfiles.azureKeyvaultSecretsProvider.config
null

more info:

 az aks show -n test2 -g test2 -o json | jq .addonProfiles.azureKeyvaultSecretsProvider
WARNING: The behavior of this command has been altered by the following extension: aks-preview
{
  "config": null,
  "enabled": true,
  "identity": {
    "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "objectId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "resourceId": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/MC_test2_test2_eastus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurekeyvaultsecretsprovider-test2"
  }
}

[chandraneel]

Thanks @zioproto for checking this. Taking another look.

[zioproto]

@raghulmsft why this was closed as completed ?
In comment #3045 (comment) I pointed out that the fix works only when creating a cluster but the bug is still there when updating a cluster.
I believe @chandraneel is still looking into this. Would it please be possible to open the issue again ? thanks

[raghulmsft]

Reopening to investigate the null config in the portal manage experience.

[rechevarria]

Fixed was applied to solve this, but we are currently in deployment lockdown. Next deployment will be on 11/29. Expect a little over 1week before changes reflect on portal.

[zioproto]

I tested again, and now from the portal I am not able to enable "Enable secret store CSI driver" anymore. The operation seems to fail silently after a few minutes of clicking the "apply" button.

[zioproto]

Steps to reproduce:

az aks create \
--location eastus \
--name test3 \
--resource-group myrg123 \
--network-plugin azure \
--kubernetes-version 1.23.12 \
--node-vm-size Standard_DC4s_v2 \
--node-count 1

Wait for the AKS cluster to be provisioned completely.

Go to the Portal, select the checkbox and apply ( see screenshot ).

---

It says Update In Progress but the feature is never activated.

---

[rechevarria]

I tested in ms.portal.azure.com and its currently working. azureKeyvaultSecretsProvider.config was null before the enabling the secret store CSI driver in portal, please see screenshot

Then, after enabling it and waiting for the save notification to complete azureKeyvaultSecretsProvider.config has correct value. Please see screenshot

[ghost]

Action required from @Azure/aks-pm

[zioproto]

Hello Folks,
after 1 month I am trying to reproduce again the steps at comment #3045 (comment)

I get again the same error, "update in progress" but nothing happens.

Using the browser developer tools I see TypeError: Cannot set properties of undefined (setting 'azureKeyvaultSecretsProvider')

[miwithro]

@raghulmsft any updates here?

[haugerbr]

@miwithro @zioproto Azure Portal team is looking into this and have a fix in review for this most recent issue.

[haugerbr]

A deployment with the fix is in progress and can be previewed at http://preview.portal.azure.com