pod_cidr seems to be ignored and set to the subnet of vnet_subnet_id

[yves-vogl]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7
+ provider.azurerm v1.12.0
+ provider.kubernetes v1.1.0
+ provider.local v1.1.0
+ provider.random v1.3.1
+ provider.tls v1.1.0

Affected Resource(s)

• azurerm v1.12.0

Terraform Configuration Files

aks_virtual_network_address_space = "10.240.0.0/16"
aks_virtual_network_node_cidr     = "10.240.0.0/20"
aks_virtual_network_pod_cidr      = "10.240.32.0/20"
aks_virtual_network_service_cidr  = "10.240.64.0/20"
aks_dns_service_ip                = "10.240.64.10"
aks_docker_bridge_cidr            = "10.240.96.1/20"


resource "azurerm_kubernetes_cluster" "aks" {

  resource_group_name = "${azurerm_resource_group.main.name}"
  location            = "${azurerm_resource_group.main.location}"

  name                = "${var.aks_cluster_name}"
  kubernetes_version  = "${var.aks_kubernetes_version}"

  dns_prefix          = "${var.aks_cluster_dns_prefix}"

  linux_profile {
    admin_username = "${var.aks_node_admin_username}"

    ssh_key {
      key_data = "${tls_private_key.aks_node_admin_ssh_keypair.public_key_openssh}"
    }
  }

  agent_pool_profile {
    name    = "${var.aks_agent_pool_name}"
    count   = "${var.aks_agent_pool_count}"
    vm_size = "${var.aks_agent_pool_vm_size}"
    os_type = "${var.aks_agent_pool_os_type}"

    os_disk_size_gb = "${var.aks_os_disk_size_gb}"
    vnet_subnet_id = "${azurerm_subnet.aks.id}"
  }

  network_profile {
    network_plugin = "azure"
    pod_cidr = "${var.aks_virtual_network_pod_cidr}"
    service_cidr = "${var.aks_virtual_network_service_cidr}"
    dns_service_ip = "${var.aks_dns_service_ip}"
    docker_bridge_cidr = "${var.aks_docker_bridge_cidr}"
  }

  service_principal {
    client_id     = "${azurerm_azuread_service_principal.aks.application_id}"
    client_secret = "${azurerm_azuread_service_principal_password.aks.value}"
  }


}

Debug Output

Running the first time:

Terraform will perform the following actions:

  + azurerm_kubernetes_cluster.aks
      id:                                         <computed>
      agent_pool_profile.#:                       "1"
      agent_pool_profile.0.count:                 "3"
      agent_pool_profile.0.dns_prefix:            <computed>
      agent_pool_profile.0.fqdn:                  <computed>
      agent_pool_profile.0.name:                  "main"
      agent_pool_profile.0.os_disk_size_gb:       "30"
      agent_pool_profile.0.os_type:               "Linux"
      agent_pool_profile.0.vm_size:               "Standard_B2ms"
      agent_pool_profile.0.vnet_subnet_id:        "${azurerm_subnet.aks.id}"
      fqdn:                                       <computed>
      kube_config.#:                              <computed>
      kube_config_raw:                            <computed>
      kubernetes_version:                         "1.10.6"
      location:                                   "westeurope"
      network_profile.#:                          "1"
      network_profile.0.dns_service_ip:           "10.240.64.10"
      network_profile.0.docker_bridge_cidr:       "10.240.96.1/20"
      network_profile.0.network_plugin:           "azure"
      network_profile.0.pod_cidr:                 "10.240.32.0/20"
      network_profile.0.service_cidr:             "10.240.64.0/20"
      node_resource_group:                        <computed>

  + azurerm_subnet.aks
      id:                                         <computed>
      address_prefix:                             "10.240.0.0/20"
      ip_configurations.#:                        <computed>
      name:                                       "<redacted>"
      resource_group_name:                        "<redacted>"
      virtual_network_name:                       "<redacted>"


azurerm_kubernetes_cluster.aks: Creating...
  agent_pool_profile.#:                       "" => "1"
  agent_pool_profile.0.count:                 "" => "3"
  agent_pool_profile.0.dns_prefix:            "" => "<computed>"
  agent_pool_profile.0.fqdn:                  "" => "<computed>"
  agent_pool_profile.0.name:                  "" => "main"
  agent_pool_profile.0.os_disk_size_gb:       "" => "30"
  agent_pool_profile.0.os_type:               "" => "Linux"
  agent_pool_profile.0.vm_size:               "" => "Standard_B2ms"
  agent_pool_profile.0.vnet_subnet_id:        "" => "/subscriptions/<uuid-redacted>/resourceGroups/<name-redacted>/providers/Microsoft.Network/virtualNetworks/<name-redacted>/subnets/<name-redacted>"
  fqdn:                                       "" => "<computed>"
  kube_config.#:                              "" => "<computed>"
  kube_config_raw:                            "<sensitive>" => "<sensitive>"
  kubernetes_version:                         "" => "1.10.6"
  linux_profile.#:                            "" => "1"
  location:                                   "" => "westeurope"
  network_profile.#:                          "" => "1"
  network_profile.0.dns_service_ip:           "" => "10.240.64.10"
  network_profile.0.docker_bridge_cidr:       "" => "10.240.96.1/20"
  network_profile.0.network_plugin:           "" => "azure"
  network_profile.0.pod_cidr:                 "" => "10.240.32.0/20"
  network_profile.0.service_cidr:             "" => "10.240.64.0/20"
  node_resource_group:                        "" => "<computed>"

Running the second time:

-/+ azurerm_kubernetes_cluster.aks (new resource required)
      id:                                         "/subscriptions/<uuid-redacted>/resourcegroups/<name-redacted>/providers/Microsoft.ContainerService/managedClusters/<name-redacted>" => <computed> (forces new resource)
      agent_pool_profile.#:                       "1" => "1"
      agent_pool_profile.0.count:                 "3" => "3"
      agent_pool_profile.0.dns_prefix:            "" => <computed>
      agent_pool_profile.0.name:                  "main" => "main"
      agent_pool_profile.0.os_disk_size_gb:       "30" => "30"
      agent_pool_profile.0.os_type:               "Linux" => "Linux"
      agent_pool_profile.0.vm_size:               "Standard_B2ms" => "Standard_B2ms"
      agent_pool_profile.0.vnet_subnet_id:        "/subscriptions/<uuid-redacted>6/resourceGroups/<name-redacted>/providers/Microsoft.Network/virtualNetworks/<name-redacted>/subnets/<name-redacted>" => "/subscriptions/<uuid-redacted>/resourceGroups/<name-redacted>/providers/Microsoft.Network/virtualNetworks/<name-redacted>/subnets/<name-redacted>"
      kube_config.#:                              "1" => <computed>
      kube_config_raw:                            <sensitive> => <computed> (attribute changed)
      kubernetes_version:                         "1.10.6" => "1.10.6"
      linux_profile.#:                            "1" => "1"    
      location:                                   "westeurope" => "westeurope"
      network_profile.#:                          "1" => "1"
      network_profile.0.dns_service_ip:           "10.240.64.10" => "10.240.64.10"
      network_profile.0.docker_bridge_cidr:       "10.240.96.1/20" => "10.240.96.1/20"
      network_profile.0.network_plugin:           "azure" => "azure"
      network_profile.0.pod_cidr:                 "" => "10.240.32.0/20" (forces new resource)
      network_profile.0.service_cidr:             "10.240.64.0/20" => "10.240.64.0/20"

Expected Behavior

Pods should be created in the given network.

Actual Behavior

In fact the pods are not in the defined pod network.

Steps to Reproduce

Create an AKS resource with completely configured network profile and investigate the ip range given to the pods.

  network_profile {
    network_plugin = "azure"
    pod_cidr = "${var.aks_virtual_network_pod_cidr}"
    service_cidr = "${var.aks_virtual_network_service_cidr}"
    dns_service_ip = "${var.aks_dns_service_ip}"
    docker_bridge_cidr = "${var.aks_docker_bridge_cidr}"
  }

[lfshr]

pod_cidr is not used when "azure" plugin is specified. The subnet range is used instead. This is by Microsoft design. This is a validation issue.

I suggest this should error along the lines of "pod_cidr can only be specified when using kubenet network plugin".

[tombuildsstuff]

hey @lfshr @yves-vogl

Just to let you know that we've just shipped support for this as a part of v1.13.0 of the AzureRM Provider :)

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!