azurerm_kubernetes_flux_configuration is removing secrets after adding kustomization blocks

[naimadswdn]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.54

AzureRM Provider Version

3.75.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_flux_configuration

Terraform Configuration Files

terraform {
  backend "local" {}
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.75.0"
    }
  }
}

provider "azurerm" {
  features {}
}

## data
data "azurerm_client_config" "current" {}

## variables
variable "https_key" {
  type = string
}

variable "https_repo_url" {
  type = string
}

## resources
resource "azurerm_resource_group" "this" {
  name = "flux-debug"
  location = "centralus"
}

resource "azurerm_kubernetes_cluster" "this" {
  name                       = "flux-debug"
  location                   = "centralus"
  resource_group_name        = azurerm_resource_group.this.name
  kubernetes_version         = "1.26.6"
  sku_tier                   = "Standard"
  dns_prefix                 = "flux-debug"
  automatic_channel_upgrade  = "node-image"
  node_os_channel_upgrade    = "NodeImage"

  local_account_disabled    = true
  workload_identity_enabled = true
  oidc_issuer_enabled       = true

  default_node_pool {
    name                = "default"
    node_count          = 1
    enable_auto_scaling = false
    vm_size             = "standard_d4_v4"
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin    = "azure"
    outbound_type     = "loadBalancer"
    load_balancer_sku = "standard"
  }

  role_based_access_control_enabled = true
  azure_active_directory_role_based_access_control {
    azure_rbac_enabled = true
    managed            = true
    tenant_id          = data.azurerm_client_config.current.tenant_id
  }
}

resource "azurerm_role_assignment" "this" {
  scope                = azurerm_kubernetes_cluster.this.id
  role_definition_name = "Azure Kubernetes Service RBAC Cluster Admin"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_kubernetes_cluster_extension" "flux" {
  name           = "flux-extension"
  cluster_id     = azurerm_kubernetes_cluster.this.id
  extension_type = "microsoft.flux"
}

resource "azurerm_kubernetes_flux_configuration" "this" {
  name       = "flux-debug-repo"
  cluster_id = azurerm_kubernetes_cluster.this.id
  namespace  = "flux-system"
  scope      = "cluster"

  git_repository {
    url                      = var.https_repo_url
    reference_type           = "branch"
    reference_value          = "master"
    https_user               = "build"
    https_key_base64         = base64encode(var.https_key)
    sync_interval_in_seconds = 60
  }

  kustomizations {
    name                     = "flux-debug"
    path                     = "flux-debug"
    sync_interval_in_seconds = 60
  }

#   kustomizations {
#     name                     = "flux-debug-update"
#     path                     = "flux-debug-update"
#     sync_interval_in_seconds = 60
#   }

  depends_on = [
    azurerm_kubernetes_cluster_extension.flux
  ]
}

Debug Output/Panic Output

None

Expected Behaviour

GitRepository is configured with secret even after adding second (and more) kustomization blocks.

Actual Behaviour

After adding second kustomization block to the azurerm_kubernetes_flux_configuration resource (that is already applied), the GitRepository object on the AKS is missing secretRef and secrets are containing 0 data.

I did some investigation in the code and that is what I found:

On the create, the configurationProtectedSettings are correctly returned by the Create function.

Anyway, on the Update function there is a bug.
Line 644 is setting Properties.ConfigurationProtectedSettings to nil and it is still nil even after the update, because this if condition is not applied - git_repository DID NOT CHANGE, but the kustomizations DID change. Anyway, the if condition for kustomizations is not returning configurationProtectedSettings.

Steps to Reproduce

1. Terraform plan - out plan
2. Terraform apply "plan"
3. Connect to the AKS and check GitRepository status with the following command:
   kubectl get gitrepository flux-debug-repo -n flux-system
   Status Ready is True.
   kubectl get secret -n flux-system flux-debug-repo-auth flux-debug-repo-protected-parameters
   Both secrets contains data.
   Check secret refrence on the GitRepository:
   kubectl get gitrepository flux-debug-repo -n flux-system -o yaml | grep secretRef -A1
   secretRef is present.
4. Uncomment second kustomization from the azurerm_kubernetes_flux_configuration resource and apply changes (steps 1 and 2).
5. After applying, check GitRepository status on the AKS once again:
   kubecl get gitrepository flux-debug-repo -n flux-system
   Status Ready is False with error:
   failed to checkout and determine revision: unable to list remote for 'HTTPS_URL_REPO': authentication required
   Check secrets:
   kubectl get secret -n flux-system flux-debug-repo-auth flux-debug-repo-protected-parameters
   Both secrets does not contains data (DATA field equal to 0).
   Check secret refrence on the GitRepository:
   kubectl get gitrepository flux-debug-repo -n flux-system -o yaml | grep secretRef -A1
   No results.

Important Factoids

No response

References

No response

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.