azurerm_kubernetes_cluster: "addon is not supported" error in Azure China with provider version >= 1.37.0

[embik]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

$ terraform version -v
Terraform v0.12.19
+ provider.azurerm v1.41.0

Affected Resource(s)

• azurerm_kubernetes_cluster

Terraform Configuration Files

provider "azurerm" {
  environment = "china"
}

resource "azurerm_kubernetes_cluster" "cluster" {
  name                = ""aks-cluster"
  location            = "chinaeast2"
  resource_group_name = "RESOURCE_GROUP_NAME"
  kubernetes_version  = "1.15.7"
  dns_prefix          = "DNS_PREFIX"
  node_resource_group = "NODE_RESOURCE_GROUP"

  agent_pool_profile {
    name = "agent-pool"

    type            = "VirtualMachineScaleSets"
    vm_size         = "Standard_E8s_v3"
    os_disk_size_gb = 64
    vnet_subnet_id  = "VNET_SUBNET_ID"
    max_pods        = 30
    node_taints     = []

    count = "NODE_COUNT"
  }

  service_principal {
    client_id     = "SERVICE_PRINCIPAL_ID"
    client_secret = "SERVICE_PRINCIPAL_SECRET"
  }

  linux_profile {
    admin_username = "ADMIN_USERNAME"
    ssh_key {
      key_data = "SSH_KEY_DATA"
    }
  }

  network_profile {
    network_plugin     = "azure"
    network_policy     = "azure"
    dns_service_ip     = "DNS_SERVICE_IP"
    service_cidr       = "SERVICE_CIDR"
    docker_bridge_cidr = "172.17.0.1/16"
    load_balancer_sku  = "basic"
  }

  role_based_access_control {
    enabled = true
  }
}

(note: this has been downgraded to agent_pool_profile because the last working version is 1.36.0, but default_node_pool has been tested as well)

Debug Output

This thing is embedded in a larger Terraform module, so I'm hesitant to share the whole debug log. Please let me know if the debug output here is really required since we're getting a relatively clear error from the Azure China API.

Panic Output

Expected Behavior

A AKS cluster should be created in Azure China.

Actual Behavior

Starting with the AzureRM provider version 1.37.0 (up to the current version 1.41.0), creating the cluster fails on Azure China with this error message:

Error: Error creating Managed Kubernetes Cluster "REDACTED" (Resource Group "REDACTED""): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="OperationNotAllowed" Message="Addon 'aciConnectorLinux' is not supported in this cloud environment."

  on .REDACTED/main.tf line 1, in resource "azurerm_kubernetes_cluster" "cluster":
   1: resource "azurerm_kubernetes_cluster" "cluster" {

With no changes to the resource, 1.36.0 successfully deploys the cluster.

Steps to Reproduce

1. terraform apply

Important Factoids

• This only affects Azure China
• probably caused by the changes to the resource in 1.37.0 (mentioned in the release notes)
• verified 1.36.0 as the last working version, both 1.37.0 and 1.41.0 are affected (did not check the versions in between)

References

• #0000

[dubuc]

Hi @andyzhangx, do you know of a workaround for this? Have you received customer complaints about the current Terraform azurerm provider integration with the Azure China Cloud?

[andyzhangx]

Hi @andyzhangx, do you know of a workaround for this? Have you received customer complaints about the current Terraform azurerm provider integration with the Azure China Cloud?

@dubuc I don't have the answer since I don't work on terraform, per the error msg, ACI is not supported on Azure China, so is there any way to remove that addon aciConnectorLinux explicitly, you may specify addon field in your config

[dubuc]

@andyzhangx Thanks, we did try specifying it and leaving it empty, but the ARM request was including it as a request. I guess a discrepancy in apiversions. have a good weekend!

[dubuc]

@jackofallops Hello, I was trying this with the new 2.0.0 provider, and this issue is still present. Could we prioritize this or assign the right persons to the ticket? This makes it impossible to deploy Chinese clusters with new providers.

Here is my terraform code with the new provider version.

# ./providers.tf
provider "azurerm" {
  version = "=2.0.0"
  features {}
}

# ./main.tf
resource "azurerm_resource_group" "example" {
  name     = "gilles-poc-tf"
  location = "chinaeast2"
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "gilles-poc-tf"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "gilles-poc"

  kubernetes_version = "1.15.7"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  service_principal {
    client_id     = "0000-0000-0000-0000"
    client_secret = "0000-0000-0000-0000"
  }

  tags = {
    Environment = "Development"
  }
}

output "client_certificate" {
  value = azurerm_kubernetes_cluster.example.kube_config.0.client_certificate
}

output "kube_config" {
  value = azurerm_kubernetes_cluster.example.kube_config_raw
}

Here is the error output.

Error: Error creating Managed Kubernetes Cluster "gilles-poc-tf" (Resource Group "gilles-poc-tf"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="OperationNotAllowed" Message="Addon 'aciConnectorLinux' is not supported in this cloud environment."

  on main.tf line 6, in resource "azurerm_kubernetes_cluster" "example":
   6: resource "azurerm_kubernetes_cluster" "example" {

[hasusuf]

We also got hit by this one, even explicitly disabling the aciConnectorLinux didn't work

addon_profile {
  aci_connector_linux {
    subnet_name = "subnetwork-name"
    enabled     = false
  }
}

[ghost]

This has been released in version 2.5.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.5.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!