âš This module is deprecated in favor of aks-light module.
This terraform module creates an Azure Kubernetes Service and its associated Azure Application Gateway as ingress controller.
Inside the cluster default node pool, velero and cert-manager are installed.
Inside each node pool, Kured is installed as a daemonset.
This module also configures logging to a Log Analytics Workspace, deploys the Azure Active Directory Pod Identity and creates some Storage Classes with different types of Azure managed disks (Standard HDD retain and delete, Premium SSD retain and delete).
Module version | Terraform version | AzureRM version |
---|---|---|
>= 5.x.x | 0.15.x & 1.0.x | >= 2.10 |
>= 4.x.x | 0.13.x | >= 2.10 |
>= 3.x.x | 0.12.x | >= 2.10 |
>= 2.x.x | 0.12.x | < 2.0 |
< 2.x.x | 0.11.x | < 2.0 |
This module is optimized to work with the Claranet terraform-wrapper too which set some terraform variables in the environment needed by this module.
More details about variables set by the terraform wrapper
available in the documentation.
The helm and kubernetes providers must be defined at the root level and then passed to the module via the provider block as in the examples.
Module version | Terraform version | AzureRM version |
---|---|---|
>= 7.x.x | 1.3.x | >= 3.0 |
>= 6.x.x | 1.x | >= 3.0 |
>= 5.x.x | 0.15.x | >= 2.0 |
>= 4.x.x | 0.13.x / 0.14.x | >= 2.0 |
>= 3.x.x | 0.12.x | >= 2.0 |
>= 2.x.x | 0.12.x | < 2.0 |
< 2.x.x | 0.11.x | < 2.0 |
If you want to contribute to this repository, feel free to use our pre-commit git hook configuration which will help you automatically update and format some files for you by enforcing our Terraform code module best-practices.
More details are available in the CONTRIBUTING.md file.
This module is optimized to work with the Claranet terraform-wrapper tool
which set some terraform variables in the environment needed by this module.
More details about variables set by the terraform-wrapper
available in the documentation.
locals {
allowed_cidrs = ["x.x.x.x", "y.y.y.y"]
}
module "azure_region" {
source = "claranet/regions/azurerm"
version = "x.x.x"
azure_region = var.azure_region
}
module "rg" {
source = "claranet/rg/azurerm"
version = "x.x.x"
location = module.azure_region.location
client_name = var.client_name
environment = var.environment
stack = var.stack
}
module "azure_virtual_network" {
source = "claranet/vnet/azurerm"
version = "x.x.x"
environment = var.environment
location = module.azure_region.location
location_short = module.azure_region.location_short
client_name = var.client_name
stack = var.stack
resource_group_name = module.rg.resource_group_name
vnet_cidr = ["10.0.0.0/19"]
}
module "node_network_subnet" {
source = "claranet/subnet/azurerm"
version = "x.x.x"
environment = var.environment
location_short = module.azure_region.location_short
client_name = var.client_name
stack = var.stack
resource_group_name = module.rg.resource_group_name
virtual_network_name = module.azure_virtual_network.virtual_network_name
name_suffix = "nodes"
subnet_cidr_list = ["10.0.0.0/20"]
service_endpoints = ["Microsoft.Storage"]
}
module "appgw_network_subnet" {
source = "claranet/subnet/azurerm"
version = "x.x.x"
environment = var.environment
location_short = module.azure_region.location_short
client_name = var.client_name
stack = var.stack
resource_group_name = module.rg.resource_group_name
virtual_network_name = module.azure_virtual_network.virtual_network_name
name_suffix = "appgw"
subnet_cidr_list = ["10.0.20.0/24"]
}
module "global_run" {
source = "claranet/run/azurerm"
version = "x.x.x"
client_name = var.client_name
location = module.azure_region.location
location_short = module.azure_region.location_short
environment = var.environment
stack = var.stack
monitoring_function_splunk_token = var.monitoring_function_splunk_token
resource_group_name = module.rg.resource_group_name
tenant_id = var.azure_tenant_id
}
resource "tls_private_key" "key" {
algorithm = "RSA"
}
module "aks" {
source = "claranet/aks/azurerm"
version = "x.x.x"
providers = {
kubernetes = kubernetes.aks-module
helm = helm.aks-module
}
client_name = var.client_name
environment = var.environment
stack = var.stack
resource_group_name = module.rg.resource_group_name
location = module.azure_region.location
location_short = module.azure_region.location_short
private_cluster_enabled = false
service_cidr = "10.0.16.0/22"
kubernetes_version = "1.19.7"
vnet_id = module.azure_virtual_network.virtual_network_id
nodes_subnet_id = module.node_network_subnet.subnet_id
nodes_pools = [
{
name = "pool1"
count = 1
vm_size = "Standard_D1_v2"
os_type = "Linux"
os_disk_type = "Ephemeral"
os_disk_size_gb = 30
vnet_subnet_id = module.node_network_subnet.subnet_id
},
{
name = "bigpool1"
count = 3
vm_size = "Standard_F8s_v2"
os_type = "Linux"
os_disk_size_gb = 30
vnet_subnet_id = module.node_network_subnet.subnet_id
enable_auto_scaling = true
min_count = 3
max_count = 9
}
]
linux_profile = {
username = "nodeadmin"
ssh_key = tls_private_key.key.public_key_openssh
}
oms_log_analytics_workspace_id = module.global_run.log_analytics_workspace_id
azure_policy_enabled = false
logs_destinations_ids = [module.global_run.log_analytics_workspace_id]
appgw_subnet_id = module.appgw_network_subnet.subnet_id
appgw_ingress_controller_values = { "verbosityLevel" = 5, "appgw.shared" = true }
cert_manager_settings = { "cainjector.nodeSelector.agentpool" = "default", "nodeSelector.agentpool" = "default", "webhook.nodeSelector.agentpool" = "default" }
velero_storage_settings = { allowed_cidrs = local.allowed_cidrs }
container_registries_id = [module.acr.acr_id]
key_vault_secrets_provider = {
secret_rotation_enabled = true
secret_rotation_interval = "2m"
}
}
module "acr" {
source = "claranet/acr/azurerm"
version = "x.x.x"
location = module.azure_region.location
location_short = module.azure_region.location_short
resource_group_name = module.rg.resource_group_name
sku = "Standard"
client_name = var.client_name
environment = var.environment
stack = var.stack
logs_destinations_ids = [module.global_run.log_analytics_workspace_id]
}
Name | Version |
---|---|
azuread | ~> 2.31 |
azurecaf | ~> 1.2, >= 1.2.22 |
azurerm | ~> 3.39 |
Name | Source | Version |
---|---|---|
appgw | ./tools/agic | n/a |
certmanager | ./tools/cert-manager | n/a |
diagnostic_settings | claranet/diagnostic-settings/azurerm | ~> 6.5.0 |
infra | ./modules/infra | n/a |
kured | ./tools/kured | n/a |
velero | ./tools/velero | n/a |
Name | Description | Type | Default | Required |
---|---|---|---|---|
aadpodidentity_chart_repository | AAD Pod Identity Helm chart repository URL | string |
"https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts" |
no |
aadpodidentity_chart_version | AAD Pod Identity helm chart version to use | string |
"4.1.9" |
no |
aadpodidentity_custom_name | Custom name for aad pod identity MSI | string |
"aad-pod-identity" |
no |
aadpodidentity_extra_tags | Extra Tags to add to aad pod identity MSI | map(string) |
{} |
no |
aadpodidentity_kubenet_policy_enabled | Boolean to wether deploy or not a built-in Azure Policy at the cluster level to mitigate potential security issue with aadpodidentity used with kubenet : https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity#using-kubenet-network-plugin-with-azure-active-directory-pod-managed-identities " |
bool |
false |
no |
aadpodidentity_namespace | Kubernetes namespace in which to deploy AAD Pod Identity | string |
"system-aadpodid" |
no |
aadpodidentity_values | Settings for AAD Pod identity helm Chart:map(object({ |
map(string) |
{} |
no |
aci_subnet_id | Optional subnet Id used for ACI virtual-nodes | string |
null |
no |
agic_chart_repository | Helm chart repository URL | string |
"https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/" |
no |
agic_chart_version | Version of the Helm chart | string |
"1.5.2" |
no |
agic_enabled | Enable Application gateway ingress controller | bool |
true |
no |
agic_helm_version | [DEPRECATED] Version of Helm chart to deploy | string |
null |
no |
aks_http_proxy_settings | AKS HTTP proxy settings. URLs must be in format http(s)://fqdn:port/ . When setting the no_proxy_url_list parameter, the AKS Private Endpoint domain name and the AKS VNet CIDR must be added to the URLs list. |
object({ |
null |
no |
aks_network_plugin | AKS network plugin to use. Possible values are azure and kubenet . Changing this forces a new resource to be created |
string |
"azure" |
no |
aks_network_policy | AKS network policy to use. | string |
"calico" |
no |
aks_pod_cidr | CIDR used by pods when network plugin is set to kubenet . https://docs.microsoft.com/en-us/azure/aks/configure-kubenet |
string |
"172.17.0.0/16" |
no |
aks_route_table_id | Provide an existing route table when outbound_type variable is set to userdefinedrouting with kubenet : https://docs.microsoft.com/fr-fr/azure/aks/configure-kubenet#bring-your-own-subnet-and-route-table-with-kubenet |
string |
null |
no |
aks_sku_tier | aks sku tier. Possible values are Free ou Paid | string |
"Free" |
no |
aks_user_assigned_identity_custom_name | Custom name for the aks user assigned identity resource | string |
null |
no |
aks_user_assigned_identity_resource_group_name | Resource Group where to deploy the aks user assigned identity resource. Used when private cluster is enabled and when Azure private dns zone is not managed by aks | string |
null |
no |
aks_user_assigned_identity_tags | Tags to add to AKS MSI | map(string) |
{} |
no |
api_server_authorized_ip_ranges | Ip ranges allowed to interract with Kubernetes API. Default no restrictions | list(string) |
[] |
no |
appgw_identity_enabled | Configure a managed service identity for Application gateway used with AGIC (useful to configure ssl cert into appgw from keyvault) | bool |
false |
no |
appgw_ingress_controller_values | Application Gateway Ingress Controller settings | map(string) |
{} |
no |
appgw_private_ip | Private IP for Application Gateway. Used when variable private_ingress is set to true . |
string |
null |
no |
appgw_settings | Application gateway configuration settings. Default dummy configuration | map(any) |
{} |
no |
appgw_ssl_certificates_configs | Application gateway ssl certificates configuration | list(map(string)) |
[] |
no |
appgw_subnet_id | Application gateway subnet id | string |
"" |
no |
appgw_user_assigned_identity_custom_name | Custom name for the application gateway user assigned identity resource | string |
null |
no |
appgw_user_assigned_identity_resource_group_name | Resource Group where to deploy the Application Gateway User Assigned Identity resource. | string |
null |
no |
application_gateway_id | ID of an existing Application Gateway to use as an AGIC. use_existing_application_gateway must be set to true . |
string |
null |
no |
auto_scaler_profile | Configuration of auto_scaler_profile block object |
object({ |
null |
no |
azure_policy_enabled | Should the Azure Policy Add-On be enabled? | bool |
false |
no |
cert_manager_chart_repository | Helm chart repository URL | string |
"https://charts.jetstack.io" |
no |
cert_manager_chart_version | Cert Manager helm chart version to use | string |
"v1.8.0" |
no |
cert_manager_namespace | Kubernetes namespace in which to deploy Cert Manager | string |
"system-cert-manager" |
no |
cert_manager_settings | Settings for cert-manager helm chart | map(string) |
{} |
no |
client_name | Client name/account used in naming | string |
n/a | yes |
container_registries_id | List of Azure Container Registries ids where AKS needs pull access. | list(string) |
[] |
no |
custom_aks_name | Custom AKS name | string |
"" |
no |
custom_appgw_name | Custom name for AKS ingress application gateway | string |
"" |
no |
custom_diagnostic_settings_name | Custom name of the diagnostics settings, name will be 'default' if not set. | string |
"default" |
no |
default_node_pool | Default node pool configuration | object({ |
{} |
no |
default_node_pool_tags | Specific tags for default node pool | map(string) |
{} |
no |
default_tags_enabled | Option to enable or disable default tags | bool |
true |
no |
docker_bridge_cidr | IP address for docker with Network CIDR. | string |
"172.16.0.1/16" |
no |
enable_cert_manager | Enable cert-manager on AKS cluster | bool |
true |
no |
enable_kured | Enable kured daemon on AKS cluster | bool |
true |
no |
enable_pod_security_policy | Enable pod security policy or not. https://docs.microsoft.com/fr-fr/azure/AKS/use-pod-security-policies | bool |
false |
no |
enable_velero | Enable velero on AKS cluster | bool |
true |
no |
environment | Project environment | string |
n/a | yes |
extra_tags | Extra tags to add | map(string) |
{} |
no |
http_application_routing_enabled | Whether HTTP Application Routing is enabled. | bool |
false |
no |
key_vault_secrets_provider | Enable AKS built-in Key Vault secrets provider. If enabled, an identity is created by the AKS itself and exported from this module. | object({ |
null |
no |
kubernetes_version | Version of Kubernetes to deploy | string |
"1.17.9" |
no |
kured_chart_repository | Helm chart repository URL | string |
"https://kubereboot.github.io/charts" |
no |
kured_chart_version | Version of the Helm chart | string |
"2.2.0" |
no |
kured_settings | Settings for kured helm chart:map(object({ |
map(string) |
{} |
no |
linux_profile | Username and ssh key for accessing AKS Linux nodes with ssh. | object({ |
null |
no |
location | Azure region to use | string |
n/a | yes |
location_short | Short name of Azure regions to use | string |
n/a | yes |
logs_categories | Log categories to send to destinations. | list(string) |
null |
no |
logs_destinations_ids | List of destination resources IDs for logs diagnostic destination. Can be Storage Account , Log Analytics Workspace and Event Hub . No more than one of each can be set.If you want to specify an Azure EventHub to send logs and metrics to, you need to provide a formated string with both the EventHub Namespace authorization send ID and the EventHub name (name of the queue to use in the Namespace) separated by the ` |
` character. | list(string) |
n/a |
logs_kube_audit_enabled | Whether to include kube-audit and kube-audit-admin logs from diagnostics settings collection. Enabling this can increase your Azure billing. |
bool |
false |
no |
logs_metrics_categories | Metrics categories to send to destinations. | list(string) |
null |
no |
name_prefix | Optional prefix for the generated name | string |
"" |
no |
name_suffix | Optional suffix for the generated name | string |
"" |
no |
node_pool_tags | Specific tags for node pool | map(string) |
{} |
no |
node_resource_group | Name of the resource group in which to put AKS nodes. If null default to MC_ | string |
null |
no |
nodes_pools | A list of nodes pools to create, each item supports same properties as local.default_agent_profile |
list(any) |
[] |
no |
nodes_subnet_id | ID of the subnet used for nodes | string |
n/a | yes |
oidc_issuer_enabled | Whether to enable OpenID Connect issuer or not. https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer | bool |
false |
no |
oms_log_analytics_workspace_id | The ID of the Log Analytics Workspace used to send OMS logs | string |
n/a | yes |
outbound_type | The outbound (egress) routing method which should be used for this Kubernetes Cluster. Possible values are loadBalancer and userDefinedRouting . |
string |
"loadBalancer" |
no |
private_cluster_enabled | Configure AKS as a Private Cluster: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_cluster_enabled | bool |
true |
no |
private_dns_zone_id | Id of the private DNS Zone when <private_dns_zone_type> is custom | string |
null |
no |
private_dns_zone_role_assignment_enabled | Option to enable or disable Private DNS Zone role assignment. | bool |
true |
no |
private_dns_zone_type | Set AKS private dns zone if needed and if private cluster is enabled (privatelink..azmk8s.io) - "Custom" : You will have to deploy a private Dns Zone on your own and pass the id with <private_dns_zone_id> variable If this settings is used, aks user assigned identity will be "userassigned" instead of "systemassigned" and the aks user must have "Private DNS Zone Contributor" role on the private DNS Zone - "System" : AKS will manage the private zone and create it in the same resource group as the Node Resource Group - "None" : In case of None you will need to bring your own DNS server and set up resolving, otherwise cluster will have issues after provisioning. https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_dns_zone_id |
string |
"System" |
no |
private_ingress | Private ingress boolean variable. When true , the default http listener will listen on private IP instead of the public IP. |
bool |
false |
no |
resource_group_name | Name of the AKS resource group | string |
n/a | yes |
service_cidr | CIDR used by kubernetes services (kubectl get svc). | string |
n/a | yes |
stack | Project stack name | string |
n/a | yes |
use_caf_naming | Use the Azure CAF naming provider to generate default resource name. custom_aks_name override this if set. Legacy default name is used if this is set to false . |
bool |
true |
no |
use_existing_application_gateway | True to use an existing Application Gateway instead of creating a new one. If true you may use appgw_ingress_controller_values = { appgw.shared = true } to tell AGIC to not erase the whole Application Gateway configuration with its own configuration.You also have to deploy AzureIngressProhibitedTarget CRD. https://github.com/Azure/application-gateway-kubernetes-ingress/blob/072626cb4e37f7b7a1b0c4578c38d1eadc3e8701/docs/setup/install-existing.md#multi-cluster--shared-app-gateway |
bool |
false |
no |
velero_chart_repository | URL of the Helm chart repository | string |
"https://vmware-tanzu.github.io/helm-charts" |
no |
velero_chart_version | Velero helm chart version to use | string |
"2.29.5" |
no |
velero_identity_custom_name | Name of the Velero MSI | string |
"velero" |
no |
velero_identity_extra_tags | Extra tags to add to velero MSI | map(string) |
{} |
no |
velero_namespace | Kubernetes namespace in which to deploy Velero | string |
"system-velero" |
no |
velero_storage_settings | Settings for Storage account and blob container for Velero | object({ |
{} |
no |
velero_values | Settings for Velero helm chart:map(object({ |
map(string) |
{} |
no |
vnet_id | Vnet id that Aks MSI should be network contributor in a private cluster | string |
null |
no |
Name | Description |
---|---|
aad_pod_identity_azure_identity | Identity object for AAD Pod Identity |
aad_pod_identity_namespace | Namespace used for AAD Pod Identity |
agic_namespace | Namespace used for AGIC |
aks_id | AKS resource id |
aks_kube_config | Kube configuration of AKS Cluster |
aks_kube_config_raw | Raw kube config to be used by kubectl command |
aks_kubelet_user_managed_identity | The Kubelet User Managed Identity used by the AKS cluster. |
aks_name | Name of the AKS cluster |
aks_nodes_pools_ids | Ids of AKS nodes pools |
aks_nodes_pools_names | Names of AKS nodes pools |
aks_nodes_rg | Name of the resource group in which AKS nodes are deployed |
aks_user_managed_identity | The User Managed Identity used by the AKS cluster. |
application_gateway_id | Id of the application gateway used by AKS |
application_gateway_identity_principal_id | Id of the managed service identity of the application gateway used by AKS |
application_gateway_name | Name of the application gateway used by AKS |
cert_manager_namespace | Namespace used for Cert Manager |
key_vault_secrets_provider_identity | The User Managed Identity used by the Key Vault secrets provider. |
kured_namespace | Namespace used for Kured |
oidc_issuer_url | The URL of the OpenID Connect issuer. |
public_ip_id | Id of the public ip used by AKS application gateway |
public_ip_name | Name of the public ip used by AKS application gateway |
velero_identity | Azure Identity used for Velero pods |
velero_namespace | Namespace used for Velero |
velero_storage_account | Storage Account on which Velero data is stored. |
velero_storage_account_container | Container in Storage Account on which Velero data is stored. |
- Azure Kubernetes Service documentation : docs.microsoft.com/en-us/azure/aks/
- Azure Kubernetes Service MSI Usage : docs.microsoft.com/en-us/azure/aks/use-managed-identity
- Azure Kubernetes Service User-Defined Route usage : docs.microsoft.com/en-us/azure/aks/egress-outboundtype
- Terraform Kubernetes provider documentation: www.terraform.io/docs/providers/kubernetes/index.html
- Terraform Helm provider documentation: www.terraform.io/docs/providers/helm/index.html
- Kured documentation: github.com/weaveworks/kured
- Velero documentation: velero.io/docs/v1.2.0/
- Velero Azure specific documentation: github.com/vmware-tanzu/velero-plugin-for-microsoft-azure
- cert-manager documentation : cert-manager.io/docs/