Add ability to terraform Azure Active Directory Apps for AKS

[btai24]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Currently the only way to use AKS with RBAC enabled is integrating with Azure Active Directory (AAD). You can do this by following the steps in this link: https://docs.microsoft.com/en-us/azure/aks/aad-integration

However, I am not sure if we are currently able to terraform what is necessary in AAD for the integration with an RBAC-enabled AKS cluster. It would be nice to be able to terraform the two AAD apps needed (client, server) and their necessary configurations (generating server app secret, adding permissions).

New or Affected Resource(s)

• azurerm_azuread_application

[tombuildsstuff]

hi @btai24

Thanks for opening this issue :)

Support for managing Azure Active Directory applications is available in Terraform using the azurerm_azuread_application, azurerm_azuread_service_principal](https://www.terraform.io/docs/providers/azurerm/r/azuread_service_principal.html) and azurerm_azuread_service_principal_password resources. Would you be able to take a look and see if that works for you?

We have plans to split the AzureAD resources out into their own Provider to allow us to support additional functionality in the near future too - as part of which we're re-thinking the design of these resources such that they may be combined together.

Since this is a question about Terraform Configuration rather than a bug in Terraform, I'm going to close this issue for the moment (but we'll continue responding 😄)

Thanks!

[jpetitte]

This may not be the right place for this, but I'll give it a go.

@tombuildsstuff I'm hitting this same challenge, but I'm struggling to implement what is found https://docs.microsoft.com/en-us/azure/aks/aad-integration#create-server-application, as @btai24 mentioned, using the azurerm_azuread_application and azurerm_azuread_service_principal.

I get the following error:

Error: Error applying plan:

3 error(s) occurred:

* kubernetes_cluster_role_binding.tiller: 1 error(s) occurred:

* kubernetes_cluster_role_binding.tiller: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:anonymous" cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope
* kubernetes_service_account.tiller: 1 error(s) occurred:

* kubernetes_service_account.tiller: serviceaccounts is forbidden: User "system:anonymous" cannot create serviceaccounts in the namespace "kube-system"
* helm_release.ingress: 1 error(s) occurred:

* helm_release.ingress: error installing: deployments.extensions is forbidden: User "system:anonymous" cannot create deployments.extensions in the namespace "kube-system"

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

My Terraform looks like the following:

# Configure the Azure Provider
provider "azurerm" {}

resource "azurerm_resource_group" "rg" {
  name     = "${var.resource_group_name}"
  location = "${var.location}"

  # tags = {
  #   workspace = "myapp"
  # }
}

resource "azurerm_log_analytics_workspace" "la_workspace" {
  name                = "${var.log_analytics_workspace_name}"
  location            = "${var.log_analytics_workspace_location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  sku                 = "${var.log_analytics_workspace_sku}"
}

resource "azurerm_log_analytics_solution" "la_solution" {
  solution_name         = "ContainerInsights"
  location              = "${azurerm_log_analytics_workspace.la_workspace.location}"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  workspace_resource_id = "${azurerm_log_analytics_workspace.la_workspace.id}"
  workspace_name        = "${azurerm_log_analytics_workspace.la_workspace.name}"

  plan {
    publisher = "Microsoft"
    product   = "OMSGallery/ContainerInsights"
  }
}

# acr
resource "azurerm_storage_account" "storage" {
  name                     = "myappacrstorage"
  resource_group_name      = "${azurerm_resource_group.rg.name}"
  location                 = "${azurerm_resource_group.rg.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_container_registry" "acr" {
  name                = "myappacr"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  location            = "${azurerm_resource_group.rg.location}"

  # admin_enabled       = true
  sku                = "Classic"
  storage_account_id = "${azurerm_storage_account.storage.id}"
}

# rbac
resource "azurerm_azuread_application" "sp" {
  name                       = "myapp-sp"
  homepage                   = "https://myapp-sp"
  identifier_uris            = ["https://myapp-sp"]
  reply_urls                 = ["https://myapp-sp"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_application" "server" {
  name                       = "myapp-server"
  homepage                   = "https://myapp-server"
  identifier_uris            = ["https://myapp-server"]
  reply_urls                 = ["https://myapp-server"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_application" "client" {
  name                       = "myapp-client"
  homepage                   = "https://myapp-client"
  identifier_uris            = ["https://myapp-client"]
  reply_urls                 = ["https://myapp-client"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_service_principal" "sp" {
  application_id = "${azurerm_azuread_application.sp.application_id}"
}

resource "azurerm_azuread_service_principal" "server" {
  application_id = "${azurerm_azuread_application.server.application_id}"
}

resource "azurerm_azuread_service_principal" "client" {
  application_id = "${azurerm_azuread_application.client.application_id}"
}

resource "azurerm_azuread_service_principal_password" "sp" {
  service_principal_id = "${azurerm_azuread_service_principal.sp.id}"
  value                = "password1"
  end_date             = "2025-01-01T01:02:03Z"
}

resource "azurerm_azuread_service_principal_password" "server" {
  service_principal_id = "${azurerm_azuread_service_principal.server.id}"
  value                = "password2"
  end_date             = "2025-01-01T01:02:03Z"
}

# data "azurerm_subscription" "primary" {}

resource "azurerm_role_assignment" "ra" {
  # scope                = "${data.azurerm_subscription.primary.id}"
  scope                = "${azurerm_container_registry.acr.id}"
  role_definition_name = "Reader"
  principal_id         = "${azurerm_azuread_service_principal.sp.id}"
}

# cluster
resource "azurerm_kubernetes_cluster" "aks" {
  name                = "${var.cluster_name}"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  dns_prefix          = "${var.dns_prefix}"

  agent_pool_profile {
    name    = "myapp"
    count   = "${var.agent_count}"
    vm_size = "Standard_B2s"
    os_type = "Linux"
  }

  addon_profile {
    oms_agent {
      enabled                    = true
      log_analytics_workspace_id = "${azurerm_log_analytics_workspace.la_workspace.id}"
    }
  }

  service_principal {
    client_id     = "${azurerm_azuread_application.sp.application_id}"
    client_secret = "password1"
  }

  role_based_access_control {
    azure_active_directory {
      client_app_id     = "${azurerm_azuread_application.client.application_id}"
      server_app_id     = "${azurerm_azuread_application.server.application_id}"
      server_app_secret = "password2"
    }
  }

  tags {
    Environment = "Production"
  }
}


provider "kubernetes" {
  host                   = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
  username               = "${azurerm_kubernetes_cluster.aks.kube_config.0.username}"
  password               = "${azurerm_kubernetes_cluster.aks.kube_config.0.password}"
  client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
  client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
  cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  subject {
    api_group = ""
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }
}

provider "helm" {
  kubernetes {
    host                   = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
    username               = "${azurerm_kubernetes_cluster.aks.kube_config.0.username}"
    password               = "${azurerm_kubernetes_cluster.aks.kube_config.0.password}"
    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
  }
}

resource "helm_release" "ingress" {
  name  = "myapp-nginx-ingress"
  chart = "stable/nginx-ingress"

  set = [
    {
      name  = "rbac.create"
      value = true
    },
  ]
}

I found some information on system:anonymous in the [https://kubernetes.io/docs/reference/access-authn-authz/authentication/#anonymous-requests](K8s docs), but I'm struggling to understand how to apply this as a potential fix in Terraform with Azurerm. What do you think?

[tombuildsstuff]

@jpetitte unfortunately I'm unfamiliar with the requirements for configuring this with Kubernetes - but I'd suggest opening a ticket on the Kubernetes Provider :)

[jpetitte]

@tombuildsstuff Yeah there could be something going on with the Kubernetes provider (I'm looking into that as well). In this case, I'd like to double check that I'm using the Azurerm provider correctly, specifically azurerm_azuread_application and it's related parts, to implement the guide found here to establish RBAC correctly in AKS. There are parts of the guide such as:

Select Manifest and edit the groupMembershipClaims value to "All".

that don't have an obvious method of implementation using Azurerm. This may be because they need to somehow be implemented using the Kubernetes provider, but I haven't been able to weed out if that's the case. There are quite a number of moving parts that go into all this, so understanding what part of the RBAC configuration is in the scope of each provider is proving difficult for me. Maybe it's all in Azurerm scope, and I've set it up correctly, which would be great. I'm just hoping someone can confirm that. I think the original poster of this issue is struggling with the same thing.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!