-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aks forces to destroy #4356
Comments
I recognized that you are using a terraform version more than one year old (from 16.08.2018) Can you try a current version and report back if the error still persist? |
The same problem occurs to me when trying to upgrade the provider version to 1.34.0. I am using Terraform 0.12.9. From the output of |
@dominik-lekse do you use a SP as well? Or do you use it without. I'm really curious why that would recreate it, while it didn't change. Maybe it's a wrongly marked as so. |
@MichaelMeelisRTL I am using the explicit service principal. Could this issue be related to #4339? |
Yes it looks like it... Exactly the problem I faced and could see. |
Can confirm this problem applies to 0.12.9 and 0.12.6 as well. Does also not seem to matter what kind of provider version that last touched your state (tested with both 1.29 and 1.33.1). |
For what it's worth, I'm seeing the same issue. With Terraform 0.12.8 and AzureRM provider 1.33.1, no changes to code = no changes to AKS cluster. As soon as I upgrade the AzureRM provider to 1.34, it wants to recreate the cluster. Terraform code:
Oddly, like @dominik-lekse, there's nothing listed in Terraform's output to hint as to which field triggered the requirement to recreate:
|
We experience the same problem running Terraform v0.11.13 and azurerm v1.30.1. It wants to replace the existing cluster with a new one due to Interestingly enough we see the problem when running terraform from our gitlab build pipe (image: hashicorp/terraform:0.11.13), but everything works fine when running it from my machine (os: Pop!_OS 18.04 LTS) After this bug showed up we have set up a new cluster and it worked just fine running the script several times from our gitlab build pipe, but all of a sudden this problem showed up again. We can reproduce the differing behaviors from gitlab and from local machine repeatedly. |
We just upgraded to terraform 0.12.7 (from 0.11.13) and azurerm 1.34.0, other than that we changed nothing. Now it forces aks recreation. # module.aks.azurerm_kubernetes_cluster.k8s must be replaced
-/+ resource "azurerm_kubernetes_cluster" "k8s" {
- api_server_authorized_ip_ranges = [] -> null
dns_prefix = "[REDACTED]"
~ enable_pod_security_policy = false -> (known after apply)
~ fqdn = "[REDACTED]" -> (known after apply)
~ id = "[REDACTED]" -> (known after apply)
~ kube_admin_config = [] -> (known after apply)
+ kube_admin_config_raw = (sensitive value)
~ kube_config = [
- {
- client_certificate = "[REDACTED]"
- client_key = "[REDACTED]"
- cluster_ca_certificate = "[REDACTED]"
- host = "[REDACTED]"
- password = "[REDACTED]"
- username = "[REDACTED]"
},
] -> (known after apply)
~ kube_config_raw = (sensitive value)
kubernetes_version = "1.12.8"
location = "westus2"
name = "[REDACTED]"
~ node_resource_group = "[REDACTED]" -> (known after apply)
resource_group_name = "[REDACTED]"
tags = {
"env" = "ops"
}
~ addon_profile {
+ http_application_routing { # forces replacement
+ enabled = false # forces replacement
+ http_application_routing_zone_name = (known after apply)
}
~ oms_agent {
enabled = true
+ log_analytics_workspace_id = "[REDACTED]"
}
}
~ agent_pool_profile {
- availability_zones = [] -> null
count = 6
+ dns_prefix = (known after apply)
- enable_auto_scaling = false -> null
~ fqdn = "[REDACTED]" -> (known after apply)
- max_count = 0 -> null
~ max_pods = 30 -> (known after apply)
- min_count = 0 -> null
name = "minion"
- node_taints = [] -> null
os_disk_size_gb = 200
os_type = "Linux"
type = "AvailabilitySet"
vm_size = "Standard_DS2_v2"
vnet_subnet_id = "[REDACTED]"
}
linux_profile {
admin_username = "ubuntu"
ssh_key {
key_data = "[REDACTED]"
}
}
~ network_profile {
~ dns_service_ip = "10.0.0.10" -> (known after apply)
~ docker_bridge_cidr = "172.17.0.1/16" -> (known after apply)
~ load_balancer_sku = "Basic" -> "basic"
network_plugin = "azure"
+ network_policy = (known after apply)
+ pod_cidr = (known after apply)
~ service_cidr = "10.0.0.0/16" -> (known after apply)
}
role_based_access_control {
enabled = true
}
~ service_principal {
client_id = "[REDACTED]"
+ client_secret = (sensitive value)
}
}
EDIT: Maybe i'm wrong but it looks like, the following parameters are causing this behavior, tho + http_application_routing { # forces replacement
+ enabled = false # forces replacement
+ http_application_routing_zone_name = (known after apply)
} |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
One more example on our side. So, looks like running apply with provider |
👋 I've been taking a look into this issue and have opened #4469 which includes a couple of fixes for this:
As such this will no longer force recreation of the Kubernetes Cluster - first by retrieving the existing Client Secret from the configuration during a Read to ensure that this is set; and secondary (should a diff show) by allowing the Service Principal to be rotated. With regards to upgrading - I've confirmed the following 3 upgrade paths work successfully:
The fix for this is in #4469 which has been merged and will ship in v1.35.0 of the Azure Provider which we're planning to release in the next couple of days - but since the fix for this has been merged I'm going to close this issue for the moment. Thanks! |
This has been released in version 1.35.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example: provider "azurerm" {
version = "~> 1.35.0"
}
# ... other configuration ... |
Looks solved, when I plan terraform, it now only changes the following:
Can I safely run apply? Thanks |
@FrankGNL since that fields obscured, if you check your statefile and |
I am still facing this issue - AKS forces to destroy when running terraform plan - Given that no changes to the cluster have been made. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks! |
Community Note
Terraform (and AzureRM Provider) Version
0.11.8 and terraform azurerm 1.34.0
Affected Resource(s)
azurerm_kubernetes_cluster
Terraform Configuration Files
Expected Behavior
No changes to the aks cluster
Actual Behavior
Wants to destroy the aks cluster and rebuild.
service_principal.0.client_secret: => (forces new resource)
The Service Principle secret and key have never changed in our CI pipelines and after reverting back to 1.33.1 it works.
Steps to Reproduce
terraform apply
on any old cluster with the new versionThe text was updated successfully, but these errors were encountered: