Changing SKU Tier from paid to standard for azurerm_kubernetes_cluster fails

[Mahagon]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.4.0

AzureRM Provider Version

3.47.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

resource "azurerm_kubernetes_cluster" "k8s" {
  name                              = "skutest-saas-${var.environment}"
  location                          = data.azurerm_resource_group.saas.location
  resource_group_name               = data.azurerm_resource_group.saas.name
  dns_prefix                        = "k8s"
  kubernetes_version                = "1.24"
  role_based_access_control_enabled = true
  automatic_channel_upgrade         = "patch"
  sku_tier                          = "Standard"
  api_server_access_profile {
    authorized_ip_ranges = concat(var.api_server_authorized_ip_ranges, ["${data.azurerm_public_ip.loadbalancer-saas-buildagent.ip_address}/32"])
  }

  identity {
    type = "SystemAssigned"
  }
  default_node_pool {
    name                         = "system"
    enable_auto_scaling          = true
    min_count                    = var.min_system_agent_count
    max_count                    = var.max_system_agent_count
    vm_size                      = "Standard_E4ds_v5"
    os_disk_type                 = "Ephemeral"
    max_pods                     = 110
    only_critical_addons_enabled = true
    vnet_subnet_id               = azurerm_subnet.k8s-subnet.id
  }

  network_profile {
    load_balancer_sku  = "standard"
    network_plugin     = "azure"
    network_policy     = "calico"
    service_cidr       = "10.140.0.0/14"
    docker_bridge_cidr = "172.17.0.1/16"
    dns_service_ip     = "10.140.0.101"
  }

  maintenance_window {
    allowed {
      day   = "Monday"
      hours = [0, 23]
    }
    allowed {
      day   = "Tuesday"
      hours = [0, 23]
    }
    allowed {
      day   = "Wednesday"
      hours = [0, 23]
    }
    allowed {
      day   = "Thursday"
      hours = [0, 23]
    }
    allowed {
      day   = "Friday"
      hours = [0, 23]
    }
    allowed {
      day   = "Saturday"
      hours = [0, 23]
    }
    allowed {
      day   = "Sunday"
      hours = [0, 23]
    }
  }

  lifecycle {
    prevent_destroy = true
  }

  tags = {
  }
}

Debug Output/Panic Output

╷
│ Error: updating Managed Cluster (Subscription: "SUBSCRIPTIONID"
│ Resource Group Name: "RESOURCEGROUPNAME"
│ Managed Cluster Name: "CLUSTERNAME"): managedclusters.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="SkuNotAvailable" Message="Standard managed cluster SKU tier is invalid. 'Paid' has been replaced by 'Standard' since v2023-02-01. "
│ 
│   with azurerm_kubernetes_cluster.k8s,
│   on k8s.tf line 1, in resource "azurerm_kubernetes_cluster" "k8s":
│    1: resource "azurerm_kubernetes_cluster" "k8s" {
│ 
╵

Expected Behaviour

Changing the value in the terraformstate fromPaid to Standard

Actual Behaviour

Azure returns an error message, that Paid has been replaced by Standard

Steps to Reproduce

1. terraform apply with sku_tier = Paid
2. terraform apply with sku_tier = Standard

Important Factoids

No response

References

#20734 sku_tier standard was introduced in azurerm v3.46.0

[stevehipwell]

@Mahagon I saw a similar error when testing v3.46.0 straight after release and I suspect that it might due to the way the Azure API/machinery is deployed. If you wait for v2023-02-26 to have been released in your region and try again does it work?

[Mahagon]

Not sure if i read it wrong, but i guess "Currently in Operation" means it's released right?

Our AKS is deployed in "Germany West Central" and the "Currently in Operation" version is v20230226.

Still getting the same error

[stevehipwell]

Not sure if i read it wrong, but i guess "Currently in Operation" means it's released right?

@Mahagon it should do.

I've not tried this since the version rolled out in our regions but based on it still being broken for you on the latest version I think there might be a provider bug.

[mcscwizzy]

Ran into this issue today as well, so it is definitely a provider problem.

[stevehipwell]

I spoke to some of the AKS team today and they confirmed that the API version would be globally available before it was published and usable, I was confusing the AKS release versions which are also date based. So this is a provider issue.

[cippaciong]

Hello, I just received an email from Azure with respect to the changes to AKS SKU name and tiers (relevant snippet below).

I'm just doing a wild assumption here, but could it be that the error is caused by the fact that the provider is hardcoding the Basic SKUName instead of Base?

Provider: internal/services/containers/kubernetes_cluster_resource.go#L1433

terraform-provider-azurerm/internal/services/containers/kubernetes_cluster_resource.go

Line 1433 in 56bf26c

Name: utils.ToPtr(managedclusters.ManagedClusterSKUNameBasic), // the only possible value at this point

)
SDK: resource-manager/containerservice/2023-01-02-preview/managedclusters/constants.go#L536

Azure email

ManagedClusterSKU parameter changes in Azure API versions

	2023-01-01 API or earlier	Starting with 2023-02-01 API
ManagedClusterSKUName	"Basic"	"Base"
ManagedClusterSKUTier	"Free" "Paid"	"Free" "Standard"

Recommended action

To avoid disrupting your AKS workloads, update any automated scripts, CD pipelines, ARM templates, Terraform, or other third-party tooling that relies on the
ManagedClusterSKUName and ManagedClusterSKUTier parameters that use "Basic" or "Paid" to use "Base" or "Standard" when you adopt Azure API version 2023-02-01 or newer.

[george-zubrienko]

The easiest fix for this would be if the provider can update this field in a state on the next plan, instead of requesting a cluster SKU upgrade (imo).

[zioproto]

@lonegunmanb @jiaweitao001 please be aware of this issue. An email has been sent to AKS customers today about this issue.

[lra]

We got the email from Azure, and we can't update anything from "Paid" to "Standard" because of this issue. This is really confusing when you get this error message:

Original Error: Code="SkuNotAvailable" Message="Standard managed cluster SKU tier is invalid. 'Paid' has been replaced by 'Standard' since v2023-02-01. "

For context we are running the latest version of everything and are on lots of regions:

Terraform v1.4.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/azurerm v3.49.0

The portal already show our clusters on Standard, but not the API:

    },
    "sku": {
      "name": "Basic",
      "tier": "Paid"
    },

[lonegunmanb]

I'd like to submit a pr to solve this issue by doing following changes:

1. Upgrade aks client sdk's version from 2023-01-02-preview to 2023-02-01.
2. Create a new pluginsdk.StateUpgraderFunc struct which would change sku_tier from Paid to Standard in existing state files.
3. Remove Paid from valid values of sku_tier.
4. Update readme, warn our users that they should change Paid to Standard soon since Paid has been deprecated.

In general, we should submit a separate pr to upgrade sdk, but in this case we must update the existing state and all corresponding logic with new API.

Should we postpone such changes to v4.0? Or we can chose plan b:

1. Upgrade aks client sdk's version from 2023-01-02-preview to 2023-02-01.
2. We tolerate Paid for now sending Standard when reading Paid from the state, if we got Standard from the API, we check d.GetRawConfig() to how the user configured sku_tier. If it was Paid in the Terraform config file, then we set Paid to the state to avoid config drift.
3. Update readme, warn our users that they should change Paid to Standard soon since Paid has been deprecated.
4. Remove Paid from valid values of sku_tier in v4.0.

WDYT @stephybun ?

[ptisma]

We got the email from Azure, and we can't update anything from "Paid" to "Standard" because of this issue. This is really confusing when you get this error message:

Original Error: Code="SkuNotAvailable" Message="Standard managed cluster SKU tier is invalid. 'Paid' has been replaced by 'Standard' since v2023-02-01. "

For context we are running the latest version of everything and are on lots of regions:

Terraform v1.4.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/azurerm v3.49.0

The portal already show our clusters on Standard, but not the API:

    },
    "sku": {
      "name": "Basic",
      "tier": "Paid"
    },

Literally same thing except I am using the azurerm version 3.50.0, any update on this?

[boliu83]

Literally same thing except I am using the azurerm version 3.50.0, any update on this?

it seems even azurerm 3.50.0 is still using managedcluster API version 2023-01-02-preview so it doesn't support SKU name Base yet. just use "Paid" sku tier for now until newer azurerm using managedcluster API version v2023-02-01 or higher

[github-actions]

This functionality has been released in v3.51.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.