6m3s v1.29.3-eks-ae9a62a
+```
+
+```sh
+kubectl get pods -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName
+```
+
+```text
+NAME NODE
+inflate-75d744d4c6-nqwz8 ip-10-0-16-155.eu-west-1.compute.internal
+inflate-75d744d4c6-nrqnn ip-10-0-16-155.eu-west-1.compute.internal
+inflate-75d744d4c6-sp4dx ip-10-0-16-155.eu-west-1.compute.internal
+inflate-75d744d4c6-xqzd9 ip-10-0-16-155.eu-west-1.compute.internal
+inflate-75d744d4c6-xr6p5 ip-10-0-16-155.eu-west-1.compute.internal
+aws-node-mnn7r ip-10-0-3-23.eu-west-1.compute.internal
+aws-node-rkmvm ip-10-0-16-155.eu-west-1.compute.internal
+aws-node-s4slh ip-10-0-41-2.eu-west-1.compute.internal
+coredns-68bd859788-7rcfq ip-10-0-3-23.eu-west-1.compute.internal
+coredns-68bd859788-l78hw ip-10-0-41-2.eu-west-1.compute.internal
+eks-pod-identity-agent-gbx8l ip-10-0-41-2.eu-west-1.compute.internal
+eks-pod-identity-agent-s7vt7 ip-10-0-16-155.eu-west-1.compute.internal
+eks-pod-identity-agent-xwgqw ip-10-0-3-23.eu-west-1.compute.internal
+karpenter-79f59bdfdc-9q5ff ip-10-0-41-2.eu-west-1.compute.internal
+karpenter-79f59bdfdc-cxvhr ip-10-0-3-23.eu-west-1.compute.internal
+kube-proxy-7crbl ip-10-0-41-2.eu-west-1.compute.internal
+kube-proxy-jtzds ip-10-0-16-155.eu-west-1.compute.internal
+kube-proxy-sm42c ip-10-0-3-23.eu-west-1.compute.internal
+```
### Tear Down & Clean-Up
@@ -51,21 +88,19 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.47 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
| [helm](#requirement\_helm) | >= 2.7 |
-| [kubectl](#requirement\_kubectl) | >= 1.14 |
-| [kubernetes](#requirement\_kubernetes) | >= 2.10 |
-| [null](#requirement\_null) | >= 3.0 |
+| [kubectl](#requirement\_kubectl) | >= 2.0 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.47 |
-| [aws.virginia](#provider\_aws.virginia) | >= 4.47 |
+| [aws](#provider\_aws) | >= 5.40 |
+| [aws.virginia](#provider\_aws.virginia) | >= 5.40 |
| [helm](#provider\_helm) | >= 2.7 |
-| [kubectl](#provider\_kubectl) | >= 1.14 |
+| [kubectl](#provider\_kubectl) | >= 2.0 |
## Modules
@@ -73,16 +108,17 @@ Note that this example may create resources which cost money. Run `terraform des
|------|--------|---------|
| [eks](#module\_eks) | ../.. | n/a |
| [karpenter](#module\_karpenter) | ../../modules/karpenter | n/a |
-| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 4.0 |
+| [karpenter\_disabled](#module\_karpenter\_disabled) | ../../modules/karpenter | n/a |
+| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |
## Resources
| Name | Type |
|------|------|
| [helm_release.karpenter](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource |
-| [kubectl_manifest.karpenter_example_deployment](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
-| [kubectl_manifest.karpenter_node_template](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
-| [kubectl_manifest.karpenter_provisioner](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
+| [kubectl_manifest.karpenter_example_deployment](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
+| [kubectl_manifest.karpenter_node_class](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
+| [kubectl_manifest.karpenter_node_pool](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws_ecrpublic_authorization_token.token](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ecrpublic_authorization_token) | data source |
@@ -94,7 +130,7 @@ No inputs.
| Name | Description |
|------|-------------|
-| [aws\_auth\_configmap\_yaml](#output\_aws\_auth\_configmap\_yaml) | Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles |
+| [access\_entries](#output\_access\_entries) | Map of access entries created and their attributes |
| [cloudwatch\_log\_group\_arn](#output\_cloudwatch\_log\_group\_arn) | Arn of cloudwatch log group created |
| [cloudwatch\_log\_group\_name](#output\_cloudwatch\_log\_group\_name) | Name of cloudwatch log group created |
| [cluster\_addons](#output\_cluster\_addons) | Map of attribute maps for all EKS cluster addons enabled |
@@ -106,31 +142,33 @@ No inputs.
| [cluster\_iam\_role\_unique\_id](#output\_cluster\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
| [cluster\_id](#output\_cluster\_id) | The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts |
| [cluster\_identity\_providers](#output\_cluster\_identity\_providers) | Map of attribute maps for all EKS identity providers enabled |
+| [cluster\_ip\_family](#output\_cluster\_ip\_family) | The IP family used by the cluster (e.g. `ipv4` or `ipv6`) |
| [cluster\_name](#output\_cluster\_name) | The name of the EKS cluster |
| [cluster\_oidc\_issuer\_url](#output\_cluster\_oidc\_issuer\_url) | The URL on the EKS cluster for the OpenID Connect identity provider |
| [cluster\_platform\_version](#output\_cluster\_platform\_version) | Platform version for the cluster |
| [cluster\_primary\_security\_group\_id](#output\_cluster\_primary\_security\_group\_id) | Cluster security group that was created by Amazon EKS for the cluster. Managed node groups use this security group for control-plane-to-data-plane communication. Referred to as 'Cluster security group' in the EKS console |
| [cluster\_security\_group\_arn](#output\_cluster\_security\_group\_arn) | Amazon Resource Name (ARN) of the cluster security group |
| [cluster\_security\_group\_id](#output\_cluster\_security\_group\_id) | ID of the cluster security group |
+| [cluster\_service\_cidr](#output\_cluster\_service\_cidr) | The CIDR block where Kubernetes pod and service IP addresses are assigned from |
| [cluster\_status](#output\_cluster\_status) | Status of the EKS cluster. One of `CREATING`, `ACTIVE`, `DELETING`, `FAILED` |
| [cluster\_tls\_certificate\_sha1\_fingerprint](#output\_cluster\_tls\_certificate\_sha1\_fingerprint) | The SHA1 fingerprint of the public key of the cluster's certificate |
| [eks\_managed\_node\_groups](#output\_eks\_managed\_node\_groups) | Map of attribute maps for all EKS managed node groups created |
| [eks\_managed\_node\_groups\_autoscaling\_group\_names](#output\_eks\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by EKS managed node groups |
| [fargate\_profiles](#output\_fargate\_profiles) | Map of attribute maps for all EKS Fargate Profiles created |
| [karpenter\_event\_rules](#output\_karpenter\_event\_rules) | Map of the event rules created and their attributes |
+| [karpenter\_iam\_role\_arn](#output\_karpenter\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the controller IAM role |
+| [karpenter\_iam\_role\_name](#output\_karpenter\_iam\_role\_name) | The name of the controller IAM role |
+| [karpenter\_iam\_role\_unique\_id](#output\_karpenter\_iam\_role\_unique\_id) | Stable and unique string identifying the controller IAM role |
| [karpenter\_instance\_profile\_arn](#output\_karpenter\_instance\_profile\_arn) | ARN assigned by AWS to the instance profile |
| [karpenter\_instance\_profile\_id](#output\_karpenter\_instance\_profile\_id) | Instance profile's ID |
| [karpenter\_instance\_profile\_name](#output\_karpenter\_instance\_profile\_name) | Name of the instance profile |
| [karpenter\_instance\_profile\_unique](#output\_karpenter\_instance\_profile\_unique) | Stable and unique string identifying the IAM instance profile |
-| [karpenter\_irsa\_arn](#output\_karpenter\_irsa\_arn) | The Amazon Resource Name (ARN) specifying the IAM role for service accounts |
-| [karpenter\_irsa\_name](#output\_karpenter\_irsa\_name) | The name of the IAM role for service accounts |
-| [karpenter\_irsa\_unique\_id](#output\_karpenter\_irsa\_unique\_id) | Stable and unique string identifying the IAM role for service accounts |
+| [karpenter\_node\_iam\_role\_arn](#output\_karpenter\_node\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
+| [karpenter\_node\_iam\_role\_name](#output\_karpenter\_node\_iam\_role\_name) | The name of the IAM role |
+| [karpenter\_node\_iam\_role\_unique\_id](#output\_karpenter\_node\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
| [karpenter\_queue\_arn](#output\_karpenter\_queue\_arn) | The ARN of the SQS queue |
| [karpenter\_queue\_name](#output\_karpenter\_queue\_name) | The name of the created Amazon SQS queue |
| [karpenter\_queue\_url](#output\_karpenter\_queue\_url) | The URL for the created Amazon SQS queue |
-| [karpenter\_role\_arn](#output\_karpenter\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
-| [karpenter\_role\_name](#output\_karpenter\_role\_name) | The name of the IAM role |
-| [karpenter\_role\_unique\_id](#output\_karpenter\_role\_unique\_id) | Stable and unique string identifying the IAM role |
| [node\_security\_group\_arn](#output\_node\_security\_group\_arn) | Amazon Resource Name (ARN) of the node shared security group |
| [node\_security\_group\_id](#output\_node\_security\_group\_id) | ID of the node shared security group |
| [oidc\_provider](#output\_oidc\_provider) | The OpenID Connect identity provider (issuer URL without leading `https://`) |
diff --git a/examples/karpenter/main.tf b/examples/karpenter/main.tf
index 86fc0a4f3b..b3db54453e 100644
--- a/examples/karpenter/main.tf
+++ b/examples/karpenter/main.tf
@@ -7,18 +7,6 @@ provider "aws" {
alias = "virginia"
}
-provider "kubernetes" {
- host = module.eks.cluster_endpoint
- cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
-
- exec {
- api_version = "client.authentication.k8s.io/v1beta1"
- command = "aws"
- # This requires the awscli to be installed locally where Terraform is executed
- args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
- }
-}
-
provider "helm" {
kubernetes {
host = module.eks.cluster_endpoint
@@ -53,9 +41,8 @@ data "aws_ecrpublic_authorization_token" "token" {
}
locals {
- name = "ex-${replace(basename(path.cwd), "_", "-")}"
- cluster_version = "1.27"
- region = "eu-west-1"
+ name = "ex-${basename(path.cwd)}"
+ region = "eu-west-1"
vpc_cidr = "10.0.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 3)
@@ -74,67 +61,42 @@ locals {
module "eks" {
source = "../.."
- cluster_name = local.name
- cluster_version = local.cluster_version
- cluster_endpoint_public_access = true
+ cluster_name = local.name
+ cluster_version = "1.29"
+
+ # Gives Terraform identity admin access to cluster which will
+ # allow deploying resources (Karpenter) into the cluster
+ enable_cluster_creator_admin_permissions = true
+ cluster_endpoint_public_access = true
cluster_addons = {
- kube-proxy = {}
- vpc-cni = {}
- coredns = {
- configuration_values = jsonencode({
- computeType = "Fargate"
- # Ensure that we fully utilize the minimum amount of resources that are supplied by
- # Fargate https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html
- # Fargate adds 256 MB to each pod's memory reservation for the required Kubernetes
- # components (kubelet, kube-proxy, and containerd). Fargate rounds up to the following
- # compute configuration that most closely matches the sum of vCPU and memory requests in
- # order to ensure pods always have the resources that they need to run.
- resources = {
- limits = {
- cpu = "0.25"
- # We are targeting the smallest Task size of 512Mb, so we subtract 256Mb from the
- # request/limit to ensure we can fit within that task
- memory = "256M"
- }
- requests = {
- cpu = "0.25"
- # We are targeting the smallest Task size of 512Mb, so we subtract 256Mb from the
- # request/limit to ensure we can fit within that task
- memory = "256M"
- }
- }
- })
- }
+ coredns = {}
+ eks-pod-identity-agent = {}
+ kube-proxy = {}
+ vpc-cni = {}
}
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
control_plane_subnet_ids = module.vpc.intra_subnets
- manage_aws_auth_configmap = true
- aws_auth_roles = [
- # We need to add in the Karpenter node IAM role for nodes launched by Karpenter
- {
- rolearn = module.karpenter.role_arn
- username = "system:node:{{EC2PrivateDNSName}}"
- groups = [
- "system:bootstrappers",
- "system:nodes",
- ]
- },
- ]
-
- fargate_profiles = {
+ eks_managed_node_groups = {
karpenter = {
- selectors = [
- { namespace = "karpenter" }
- ]
- }
- kube-system = {
- selectors = [
- { namespace = "kube-system" }
- ]
+ instance_types = ["m5.large"]
+
+ min_size = 2
+ max_size = 3
+ desired_size = 2
+
+ taints = {
+ # This Taint aims to keep just EKS Addons and Karpenter running on this MNG
+ # The pods that do not tolerate this taint should run on nodes created by Karpenter
+ addons = {
+ key = "CriticalAddonsOnly"
+ value = "true"
+ effect = "NO_SCHEDULE"
+ },
+ }
}
}
@@ -153,70 +115,69 @@ module "eks" {
module "karpenter" {
source = "../../modules/karpenter"
- cluster_name = module.eks.cluster_name
- irsa_oidc_provider_arn = module.eks.oidc_provider_arn
+ cluster_name = module.eks.cluster_name
+
+ enable_pod_identity = true
+ create_pod_identity_association = true
- policies = {
+ # Used to attach additional IAM policies to the Karpenter node IAM role
+ node_iam_role_additional_policies = {
AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
tags = local.tags
}
-resource "helm_release" "karpenter" {
- namespace = "karpenter"
- create_namespace = true
+module "karpenter_disabled" {
+ source = "../../modules/karpenter"
+
+ create = false
+}
+
+################################################################################
+# Karpenter Helm chart & manifests
+# Not required; just to demonstrate functionality of the sub-module
+################################################################################
+resource "helm_release" "karpenter" {
+ namespace = "kube-system"
name = "karpenter"
repository = "oci://public.ecr.aws/karpenter"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
chart = "karpenter"
- version = "v0.19.3"
-
- set {
- name = "settings.aws.clusterName"
- value = module.eks.cluster_name
- }
-
- set {
- name = "settings.aws.clusterEndpoint"
- value = module.eks.cluster_endpoint
- }
-
- set {
- name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
- value = module.karpenter.irsa_arn
- }
-
- set {
- name = "settings.aws.defaultInstanceProfile"
- value = module.karpenter.instance_profile_name
- }
-
- set {
- name = "settings.aws.interruptionQueueName"
- value = module.karpenter.queue_name
- }
+ version = "0.36.1"
+ wait = false
+
+ values = [
+ <<-EOT
+ serviceAccount:
+ name: ${module.karpenter.service_account}
+ settings:
+ clusterName: ${module.eks.cluster_name}
+ clusterEndpoint: ${module.eks.cluster_endpoint}
+ interruptionQueue: ${module.karpenter.queue_name}
+ EOT
+ ]
}
-resource "kubectl_manifest" "karpenter_provisioner" {
+resource "kubectl_manifest" "karpenter_node_class" {
yaml_body = <<-YAML
- apiVersion: karpenter.sh/v1alpha5
- kind: Provisioner
+ apiVersion: karpenter.k8s.aws/v1beta1
+ kind: EC2NodeClass
metadata:
name: default
spec:
- requirements:
- - key: karpenter.sh/capacity-type
- operator: In
- values: ["spot"]
- limits:
- resources:
- cpu: 1000
- providerRef:
- name: default
- ttlSecondsAfterEmpty: 30
+ amiFamily: AL2
+ role: ${module.karpenter.node_iam_role_name}
+ subnetSelectorTerms:
+ - tags:
+ karpenter.sh/discovery: ${module.eks.cluster_name}
+ securityGroupSelectorTerms:
+ - tags:
+ karpenter.sh/discovery: ${module.eks.cluster_name}
+ tags:
+ karpenter.sh/discovery: ${module.eks.cluster_name}
YAML
depends_on = [
@@ -224,23 +185,39 @@ resource "kubectl_manifest" "karpenter_provisioner" {
]
}
-resource "kubectl_manifest" "karpenter_node_template" {
+resource "kubectl_manifest" "karpenter_node_pool" {
yaml_body = <<-YAML
- apiVersion: karpenter.k8s.aws/v1alpha1
- kind: AWSNodeTemplate
+ apiVersion: karpenter.sh/v1beta1
+ kind: NodePool
metadata:
name: default
spec:
- subnetSelector:
- karpenter.sh/discovery: ${module.eks.cluster_name}
- securityGroupSelector:
- karpenter.sh/discovery: ${module.eks.cluster_name}
- tags:
- karpenter.sh/discovery: ${module.eks.cluster_name}
+ template:
+ spec:
+ nodeClassRef:
+ name: default
+ requirements:
+ - key: "karpenter.k8s.aws/instance-category"
+ operator: In
+ values: ["c", "m", "r"]
+ - key: "karpenter.k8s.aws/instance-cpu"
+ operator: In
+ values: ["4", "8", "16", "32"]
+ - key: "karpenter.k8s.aws/instance-hypervisor"
+ operator: In
+ values: ["nitro"]
+ - key: "karpenter.k8s.aws/instance-generation"
+ operator: Gt
+ values: ["2"]
+ limits:
+ cpu: 1000
+ disruption:
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 30s
YAML
depends_on = [
- helm_release.karpenter
+ kubectl_manifest.karpenter_node_class
]
}
@@ -282,7 +259,7 @@ resource "kubectl_manifest" "karpenter_example_deployment" {
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
- version = "~> 4.0"
+ version = "~> 5.0"
name = local.name
cidr = local.vpc_cidr
diff --git a/examples/karpenter/outputs.tf b/examples/karpenter/outputs.tf
index f0ad50bd6a..de0e2e6a28 100644
--- a/examples/karpenter/outputs.tf
+++ b/examples/karpenter/outputs.tf
@@ -47,6 +47,25 @@ output "cluster_primary_security_group_id" {
value = module.eks.cluster_primary_security_group_id
}
+output "cluster_service_cidr" {
+ description = "The CIDR block where Kubernetes pod and service IP addresses are assigned from"
+ value = module.eks.cluster_service_cidr
+}
+
+output "cluster_ip_family" {
+ description = "The IP family used by the cluster (e.g. `ipv4` or `ipv6`)"
+ value = module.eks.cluster_ip_family
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+output "access_entries" {
+ description = "Map of access entries created and their attributes"
+ value = module.eks.access_entries
+}
+
################################################################################
# Security Group
################################################################################
@@ -183,31 +202,22 @@ output "self_managed_node_groups_autoscaling_group_names" {
}
################################################################################
-# Additional
-################################################################################
-
-output "aws_auth_configmap_yaml" {
- description = "Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles"
- value = module.eks.aws_auth_configmap_yaml
-}
-
-################################################################################
-# IAM Role for Service Account (IRSA)
+# Karpenter controller IAM Role
################################################################################
-output "karpenter_irsa_name" {
- description = "The name of the IAM role for service accounts"
- value = module.karpenter.irsa_name
+output "karpenter_iam_role_name" {
+ description = "The name of the controller IAM role"
+ value = module.karpenter.iam_role_name
}
-output "karpenter_irsa_arn" {
- description = "The Amazon Resource Name (ARN) specifying the IAM role for service accounts"
- value = module.karpenter.irsa_arn
+output "karpenter_iam_role_arn" {
+ description = "The Amazon Resource Name (ARN) specifying the controller IAM role"
+ value = module.karpenter.iam_role_arn
}
-output "karpenter_irsa_unique_id" {
- description = "Stable and unique string identifying the IAM role for service accounts"
- value = module.karpenter.irsa_unique_id
+output "karpenter_iam_role_unique_id" {
+ description = "Stable and unique string identifying the controller IAM role"
+ value = module.karpenter.iam_role_unique_id
}
################################################################################
@@ -242,19 +252,19 @@ output "karpenter_event_rules" {
# Node IAM Role
################################################################################
-output "karpenter_role_name" {
+output "karpenter_node_iam_role_name" {
description = "The name of the IAM role"
- value = module.karpenter.role_name
+ value = module.karpenter.node_iam_role_name
}
-output "karpenter_role_arn" {
+output "karpenter_node_iam_role_arn" {
description = "The Amazon Resource Name (ARN) specifying the IAM role"
- value = module.karpenter.role_arn
+ value = module.karpenter.node_iam_role_arn
}
-output "karpenter_role_unique_id" {
+output "karpenter_node_iam_role_unique_id" {
description = "Stable and unique string identifying the IAM role"
- value = module.karpenter.role_unique_id
+ value = module.karpenter.node_iam_role_unique_id
}
################################################################################
diff --git a/examples/karpenter/versions.tf b/examples/karpenter/versions.tf
index cab7b21121..a43b64f4c4 100644
--- a/examples/karpenter/versions.tf
+++ b/examples/karpenter/versions.tf
@@ -1,26 +1,18 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
- }
- kubernetes = {
- source = "hashicorp/kubernetes"
- version = ">= 2.10"
+ version = ">= 5.40"
}
helm = {
source = "hashicorp/helm"
version = ">= 2.7"
}
kubectl = {
- source = "gavinbunney/kubectl"
- version = ">= 1.14"
- }
- null = {
- source = "hashicorp/null"
- version = ">= 3.0"
+ source = "alekc/kubectl"
+ version = ">= 2.0"
}
}
}
diff --git a/examples/outposts/README.md b/examples/outposts/README.md
index 33684940ac..a454fd2c90 100644
--- a/examples/outposts/README.md
+++ b/examples/outposts/README.md
@@ -36,21 +36,25 @@ $ terraform apply
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
+```bash
+terraform destroy
+```
+
## Requirements
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.47 |
-| [kubernetes](#requirement\_kubernetes) | >= 2.10 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
+| [kubernetes](#requirement\_kubernetes) | >= 2.20 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.47 |
-| [kubernetes](#provider\_kubernetes) | >= 2.10 |
+| [aws](#provider\_aws) | >= 5.40 |
+| [kubernetes](#provider\_kubernetes) | >= 2.20 |
## Modules
@@ -80,7 +84,7 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Description |
|------|-------------|
-| [aws\_auth\_configmap\_yaml](#output\_aws\_auth\_configmap\_yaml) | Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles |
+| [access\_entries](#output\_access\_entries) | Map of access entries created and their attributes |
| [cloudwatch\_log\_group\_arn](#output\_cloudwatch\_log\_group\_arn) | Arn of cloudwatch log group created |
| [cloudwatch\_log\_group\_name](#output\_cloudwatch\_log\_group\_name) | Name of cloudwatch log group created |
| [cluster\_addons](#output\_cluster\_addons) | Map of attribute maps for all EKS cluster addons enabled |
@@ -92,12 +96,14 @@ Note that this example may create resources which cost money. Run `terraform des
| [cluster\_iam\_role\_unique\_id](#output\_cluster\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
| [cluster\_id](#output\_cluster\_id) | The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts |
| [cluster\_identity\_providers](#output\_cluster\_identity\_providers) | Map of attribute maps for all EKS identity providers enabled |
+| [cluster\_ip\_family](#output\_cluster\_ip\_family) | The IP family used by the cluster (e.g. `ipv4` or `ipv6`) |
| [cluster\_name](#output\_cluster\_name) | The name of the EKS cluster |
| [cluster\_oidc\_issuer\_url](#output\_cluster\_oidc\_issuer\_url) | The URL on the EKS cluster for the OpenID Connect identity provider |
| [cluster\_platform\_version](#output\_cluster\_platform\_version) | Platform version for the cluster |
| [cluster\_primary\_security\_group\_id](#output\_cluster\_primary\_security\_group\_id) | Cluster security group that was created by Amazon EKS for the cluster. Managed node groups use this security group for control-plane-to-data-plane communication. Referred to as 'Cluster security group' in the EKS console |
| [cluster\_security\_group\_arn](#output\_cluster\_security\_group\_arn) | Amazon Resource Name (ARN) of the cluster security group |
| [cluster\_security\_group\_id](#output\_cluster\_security\_group\_id) | ID of the cluster security group |
+| [cluster\_service\_cidr](#output\_cluster\_service\_cidr) | The CIDR block where Kubernetes pod and service IP addresses are assigned from |
| [cluster\_status](#output\_cluster\_status) | Status of the EKS cluster. One of `CREATING`, `ACTIVE`, `DELETING`, `FAILED` |
| [cluster\_tls\_certificate\_sha1\_fingerprint](#output\_cluster\_tls\_certificate\_sha1\_fingerprint) | The SHA1 fingerprint of the public key of the cluster's certificate |
| [eks\_managed\_node\_groups](#output\_eks\_managed\_node\_groups) | Map of attribute maps for all EKS managed node groups created |
diff --git a/examples/outposts/main.tf b/examples/outposts/main.tf
index 1964520ab8..7b4068e0e1 100644
--- a/examples/outposts/main.tf
+++ b/examples/outposts/main.tf
@@ -2,21 +2,9 @@ provider "aws" {
region = var.region
}
-provider "kubernetes" {
- host = module.eks.cluster_endpoint
- cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
-
- exec {
- api_version = "client.authentication.k8s.io/v1beta1"
- command = "aws"
- # Note: `cluster_id` is used with Outposts for auth
- args = ["eks", "get-token", "--cluster-id", module.eks.cluster_id, "--region", var.region]
- }
-}
-
locals {
name = "ex-${basename(path.cwd)}"
- cluster_version = "1.27" # Required by EKS on Outposts
+ cluster_version = "1.29"
outpost_arn = element(tolist(data.aws_outposts_outposts.this.arns), 0)
instance_type = element(tolist(data.aws_outposts_outpost_instance_types.this.instance_types), 0)
@@ -41,6 +29,10 @@ module "eks" {
cluster_endpoint_public_access = false # Not available on Outpost
cluster_endpoint_private_access = true
+ # Gives Terraform identity admin access to cluster which will
+ # allow deploying resources (EBS storage class) into the cluster
+ enable_cluster_creator_admin_permissions = true
+
vpc_id = data.aws_vpc.this.id
subnet_ids = data.aws_subnets.this.ids
@@ -49,9 +41,6 @@ module "eks" {
outpost_arns = [local.outpost_arn]
}
- # Local clusters will automatically add the node group IAM role to the aws-auth configmap
- manage_aws_auth_configmap = true
-
# Extend cluster security group rules
cluster_security_group_additional_rules = {
ingress_vpc_https = {
diff --git a/examples/outposts/outputs.tf b/examples/outposts/outputs.tf
index 43334ecc0a..24183fd207 100644
--- a/examples/outposts/outputs.tf
+++ b/examples/outposts/outputs.tf
@@ -47,6 +47,25 @@ output "cluster_primary_security_group_id" {
value = module.eks.cluster_primary_security_group_id
}
+output "cluster_service_cidr" {
+ description = "The CIDR block where Kubernetes pod and service IP addresses are assigned from"
+ value = module.eks.cluster_service_cidr
+}
+
+output "cluster_ip_family" {
+ description = "The IP family used by the cluster (e.g. `ipv4` or `ipv6`)"
+ value = module.eks.cluster_ip_family
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+output "access_entries" {
+ description = "Map of access entries created and their attributes"
+ value = module.eks.access_entries
+}
+
################################################################################
# KMS Key
################################################################################
@@ -200,12 +219,3 @@ output "self_managed_node_groups_autoscaling_group_names" {
description = "List of the autoscaling group names created by self-managed node groups"
value = module.eks.self_managed_node_groups_autoscaling_group_names
}
-
-################################################################################
-# Additional
-################################################################################
-
-output "aws_auth_configmap_yaml" {
- description = "Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles"
- value = module.eks.aws_auth_configmap_yaml
-}
diff --git a/examples/outposts/prerequisites/main.tf b/examples/outposts/prerequisites/main.tf
index 014418121d..66ab2a4e29 100644
--- a/examples/outposts/prerequisites/main.tf
+++ b/examples/outposts/prerequisites/main.tf
@@ -23,7 +23,7 @@ locals {
module "ssm_bastion_ec2" {
source = "terraform-aws-modules/ec2-instance/aws"
- version = "~> 4.2"
+ version = "~> 5.5"
name = "${local.name}-bastion"
@@ -56,7 +56,7 @@ module "ssm_bastion_ec2" {
rm terraform_${local.terraform_version}_linux_amd64.zip 2> /dev/null
# Install kubectl
- curl -LO https://dl.k8s.io/release/v1.27.0/bin/linux/amd64/kubectl
+ curl -LO https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# Remove default awscli which is v1 - we want latest v2
@@ -80,7 +80,7 @@ module "ssm_bastion_ec2" {
module "bastion_security_group" {
source = "terraform-aws-modules/security-group/aws"
- version = "~> 4.0"
+ version = "~> 5.0"
name = "${local.name}-bastion"
description = "Security group to allow provisioning ${local.name} EKS local cluster on Outposts"
diff --git a/examples/outposts/prerequisites/versions.tf b/examples/outposts/prerequisites/versions.tf
index 5f058b4c11..6f83215f50 100644
--- a/examples/outposts/prerequisites/versions.tf
+++ b/examples/outposts/prerequisites/versions.tf
@@ -1,10 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.34"
+ version = ">= 5.40"
}
}
}
diff --git a/examples/outposts/versions.tf b/examples/outposts/versions.tf
index aeb892f359..2ac7910678 100644
--- a/examples/outposts/versions.tf
+++ b/examples/outposts/versions.tf
@@ -1,14 +1,14 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
+ version = ">= 5.40"
}
kubernetes = {
source = "hashicorp/kubernetes"
- version = ">= 2.10"
+ version = ">= 2.20"
}
}
}
diff --git a/examples/self_managed_node_group/README.md b/examples/self_managed_node_group/README.md
index c5ddbc325c..f6f063f693 100644
--- a/examples/self_managed_node_group/README.md
+++ b/examples/self_managed_node_group/README.md
@@ -25,24 +25,25 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.47 |
-| [kubernetes](#requirement\_kubernetes) | >= 2.10 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.47 |
+| [aws](#provider\_aws) | >= 5.40 |
## Modules
| Name | Source | Version |
|------|--------|---------|
-| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 1.5 |
+| [disabled\_self\_managed\_node\_group](#module\_disabled\_self\_managed\_node\_group) | ../../modules/self-managed-node-group | n/a |
+| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 2.0 |
| [eks](#module\_eks) | ../.. | n/a |
| [key\_pair](#module\_key\_pair) | terraform-aws-modules/key-pair/aws | ~> 2.0 |
-| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 4.0 |
+| [kms](#module\_kms) | terraform-aws-modules/kms/aws | ~> 2.1 |
+| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |
## Resources
@@ -62,7 +63,7 @@ No inputs.
| Name | Description |
|------|-------------|
-| [aws\_auth\_configmap\_yaml](#output\_aws\_auth\_configmap\_yaml) | Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles |
+| [access\_entries](#output\_access\_entries) | Map of access entries created and their attributes |
| [cloudwatch\_log\_group\_arn](#output\_cloudwatch\_log\_group\_arn) | Arn of cloudwatch log group created |
| [cloudwatch\_log\_group\_name](#output\_cloudwatch\_log\_group\_name) | Name of cloudwatch log group created |
| [cluster\_addons](#output\_cluster\_addons) | Map of attribute maps for all EKS cluster addons enabled |
@@ -74,12 +75,14 @@ No inputs.
| [cluster\_iam\_role\_unique\_id](#output\_cluster\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
| [cluster\_id](#output\_cluster\_id) | The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts |
| [cluster\_identity\_providers](#output\_cluster\_identity\_providers) | Map of attribute maps for all EKS identity providers enabled |
+| [cluster\_ip\_family](#output\_cluster\_ip\_family) | The IP family used by the cluster (e.g. `ipv4` or `ipv6`) |
| [cluster\_name](#output\_cluster\_name) | The name of the EKS cluster |
| [cluster\_oidc\_issuer\_url](#output\_cluster\_oidc\_issuer\_url) | The URL on the EKS cluster for the OpenID Connect identity provider |
| [cluster\_platform\_version](#output\_cluster\_platform\_version) | Platform version for the cluster |
| [cluster\_primary\_security\_group\_id](#output\_cluster\_primary\_security\_group\_id) | Cluster security group that was created by Amazon EKS for the cluster. Managed node groups use this security group for control-plane-to-data-plane communication. Referred to as 'Cluster security group' in the EKS console |
| [cluster\_security\_group\_arn](#output\_cluster\_security\_group\_arn) | Amazon Resource Name (ARN) of the cluster security group |
| [cluster\_security\_group\_id](#output\_cluster\_security\_group\_id) | ID of the cluster security group |
+| [cluster\_service\_cidr](#output\_cluster\_service\_cidr) | The CIDR block where Kubernetes pod and service IP addresses are assigned from |
| [cluster\_status](#output\_cluster\_status) | Status of the EKS cluster. One of `CREATING`, `ACTIVE`, `DELETING`, `FAILED` |
| [cluster\_tls\_certificate\_sha1\_fingerprint](#output\_cluster\_tls\_certificate\_sha1\_fingerprint) | The SHA1 fingerprint of the public key of the cluster's certificate |
| [eks\_managed\_node\_groups](#output\_eks\_managed\_node\_groups) | Map of attribute maps for all EKS managed node groups created |
diff --git a/examples/self_managed_node_group/main.tf b/examples/self_managed_node_group/main.tf
index 87be519086..1d0e80b2fd 100644
--- a/examples/self_managed_node_group/main.tf
+++ b/examples/self_managed_node_group/main.tf
@@ -2,24 +2,12 @@ provider "aws" {
region = local.region
}
-provider "kubernetes" {
- host = module.eks.cluster_endpoint
- cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
-
- exec {
- api_version = "client.authentication.k8s.io/v1beta1"
- command = "aws"
- # This requires the awscli to be installed locally where Terraform is executed
- args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
- }
-}
-
data "aws_caller_identity" "current" {}
data "aws_availability_zones" "available" {}
locals {
name = "ex-${replace(basename(path.cwd), "_", "-")}"
- cluster_version = "1.27"
+ cluster_version = "1.29"
region = "eu-west-1"
vpc_cidr = "10.0.0.0/16"
@@ -43,6 +31,12 @@ module "eks" {
cluster_version = local.cluster_version
cluster_endpoint_public_access = true
+ enable_cluster_creator_admin_permissions = true
+
+ # Enable EFA support by adding necessary security group rules
+ # to the shared node security group
+ enable_efa_support = true
+
cluster_addons = {
coredns = {
most_recent = true
@@ -59,9 +53,12 @@ module "eks" {
subnet_ids = module.vpc.private_subnets
control_plane_subnet_ids = module.vpc.intra_subnets
- # Self managed node groups will not automatically create the aws-auth configmap so we need to
- create_aws_auth_configmap = true
- manage_aws_auth_configmap = true
+ # External encryption key
+ create_kms_key = false
+ cluster_encryption_config = {
+ resources = ["secrets"]
+ provider_key_arn = module.kms.key_arn
+ }
self_managed_node_group_defaults = {
# enable discovery of autoscaling groups by cluster-autoscaler
@@ -75,11 +72,34 @@ module "eks" {
# Default node group - as provisioned by the module defaults
default_node_group = {}
+ # AL2023 node group utilizing new user data format which utilizes nodeadm
+ # to join nodes to the cluster (instead of /etc/eks/bootstrap.sh)
+ al2023_nodeadm = {
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ cloudinit_pre_nodeadm = [
+ {
+ content_type = "application/node.eks.aws"
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ }
+ ]
+ }
+
# Bottlerocket node group
bottlerocket = {
name = "bottlerocket-self-mng"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
ami_id = data.aws_ami.eks_default_bottlerocket.id
instance_type = "m5.large"
desired_size = 2
@@ -141,36 +161,6 @@ module "eks" {
}
}
- efa = {
- min_size = 1
- max_size = 2
- desired_size = 1
-
- # aws ec2 describe-instance-types --region eu-west-1 --filters Name=network-info.efa-supported,Values=true --query "InstanceTypes[*].[InstanceType]" --output text | sort
- instance_type = "c5n.9xlarge"
-
- post_bootstrap_user_data = <<-EOT
- # Install EFA
- curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz
- tar -xf aws-efa-installer-latest.tar.gz && cd aws-efa-installer
- ./efa_installer.sh -y --minimal
- fi_info -p efa -t FI_EP_RDM
-
- # Disable ptrace
- sysctl -w kernel.yama.ptrace_scope=0
- EOT
-
- network_interfaces = [
- {
- description = "EFA interface example"
- delete_on_termination = true
- device_index = 0
- associate_public_ip_address = false
- interface_type = "efa"
- }
- ]
- }
-
# Complete
complete = {
name = "complete-self-mng"
@@ -216,6 +206,58 @@ module "eks" {
}
}
+ instance_attributes = {
+ name = "instance-attributes"
+
+ min_size = 1
+ max_size = 2
+ desired_size = 1
+
+ bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
+
+ instance_type = null
+
+ # launch template configuration
+ instance_requirements = {
+ cpu_manufacturers = ["intel"]
+ instance_generations = ["current", "previous"]
+ spot_max_price_percentage_over_lowest_price = 100
+
+ vcpu_count = {
+ min = 1
+ }
+
+ allowed_instance_types = ["t*", "m*"]
+ }
+
+ use_mixed_instances_policy = true
+ mixed_instances_policy = {
+ instances_distribution = {
+ on_demand_base_capacity = 0
+ on_demand_percentage_above_base_capacity = 0
+ on_demand_allocation_strategy = "lowest-price"
+ spot_allocation_strategy = "price-capacity-optimized"
+ }
+
+ # ASG configuration
+ override = [
+ {
+ instance_requirements = {
+ cpu_manufacturers = ["intel"]
+ instance_generations = ["current", "previous"]
+ spot_max_price_percentage_over_lowest_price = 100
+
+ vcpu_count = {
+ min = 1
+ }
+
+ allowed_instance_types = ["t*", "m*"]
+ }
+ }
+ ]
+ }
+ }
+
metadata_options = {
http_endpoint = "enabled"
http_tokens = "required"
@@ -235,28 +277,51 @@ module "eks" {
additional = aws_iam_policy.additional.arn
}
- timeouts = {
- create = "80m"
- update = "80m"
- delete = "80m"
- }
-
tags = {
ExtraTag = "Self managed node group complete example"
}
}
+
+ efa = {
+ # Disabling automatic creation due to instance type/quota availability
+ # Can be enabled when appropriate for testing/validation
+ create = false
+
+ ami_type = "AL2_x86_64_GPU"
+ instance_type = "trn1n.32xlarge"
+
+ enable_efa_support = true
+ pre_bootstrap_user_data = <<-EOT
+ # Mount NVME instance store volumes since they are typically
+ # available on instances that support EFA
+ setup-local-disks raid0
+ EOT
+
+ min_size = 2
+ max_size = 2
+ desired_size = 2
+ }
}
tags = local.tags
}
+module "disabled_self_managed_node_group" {
+ source = "../../modules/self-managed-node-group"
+
+ create = false
+
+ # Hard requirement
+ cluster_service_cidr = ""
+}
+
################################################################################
# Supporting Resources
################################################################################
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
- version = "~> 4.0"
+ version = "~> 5.0"
name = local.name
cidr = local.vpc_cidr
@@ -312,7 +377,7 @@ module "key_pair" {
module "ebs_kms_key" {
source = "terraform-aws-modules/kms/aws"
- version = "~> 1.5"
+ version = "~> 2.0"
description = "Customer managed key to encrypt EKS managed node group volumes"
@@ -334,6 +399,18 @@ module "ebs_kms_key" {
tags = local.tags
}
+module "kms" {
+ source = "terraform-aws-modules/kms/aws"
+ version = "~> 2.1"
+
+ aliases = ["eks/${local.name}"]
+ description = "${local.name} cluster encryption key"
+ enable_default_policy = true
+ key_owners = [data.aws_caller_identity.current.arn]
+
+ tags = local.tags
+}
+
resource "aws_iam_policy" "additional" {
name = "${local.name}-additional"
description = "Example usage of node additional policy"
diff --git a/examples/self_managed_node_group/outputs.tf b/examples/self_managed_node_group/outputs.tf
index 43334ecc0a..24183fd207 100644
--- a/examples/self_managed_node_group/outputs.tf
+++ b/examples/self_managed_node_group/outputs.tf
@@ -47,6 +47,25 @@ output "cluster_primary_security_group_id" {
value = module.eks.cluster_primary_security_group_id
}
+output "cluster_service_cidr" {
+ description = "The CIDR block where Kubernetes pod and service IP addresses are assigned from"
+ value = module.eks.cluster_service_cidr
+}
+
+output "cluster_ip_family" {
+ description = "The IP family used by the cluster (e.g. `ipv4` or `ipv6`)"
+ value = module.eks.cluster_ip_family
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+output "access_entries" {
+ description = "Map of access entries created and their attributes"
+ value = module.eks.access_entries
+}
+
################################################################################
# KMS Key
################################################################################
@@ -200,12 +219,3 @@ output "self_managed_node_groups_autoscaling_group_names" {
description = "List of the autoscaling group names created by self-managed node groups"
value = module.eks.self_managed_node_groups_autoscaling_group_names
}
-
-################################################################################
-# Additional
-################################################################################
-
-output "aws_auth_configmap_yaml" {
- description = "Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles"
- value = module.eks.aws_auth_configmap_yaml
-}
diff --git a/examples/self_managed_node_group/versions.tf b/examples/self_managed_node_group/versions.tf
index aeb892f359..6f83215f50 100644
--- a/examples/self_managed_node_group/versions.tf
+++ b/examples/self_managed_node_group/versions.tf
@@ -1,14 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
- }
- kubernetes = {
- source = "hashicorp/kubernetes"
- version = ">= 2.10"
+ version = ">= 5.40"
}
}
}
diff --git a/examples/user_data/README.md b/examples/user_data/README.md
index cea7dce755..de9b419bb4 100644
--- a/examples/user_data/README.md
+++ b/examples/user_data/README.md
@@ -17,37 +17,85 @@ $ terraform apply
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [local](#requirement\_local) | >= 2.4 |
## Providers
-No providers.
+| Name | Version |
+|------|---------|
+| [local](#provider\_local) | >= 2.4 |
## Modules
| Name | Source | Version |
|------|--------|---------|
+| [eks\_mng\_al2023\_additional](#module\_eks\_mng\_al2023\_additional) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2023\_custom\_ami](#module\_eks\_mng\_al2023\_custom\_ami) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2023\_custom\_template](#module\_eks\_mng\_al2023\_custom\_template) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2023\_no\_op](#module\_eks\_mng\_al2023\_no\_op) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2\_additional](#module\_eks\_mng\_al2\_additional) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2\_custom\_ami](#module\_eks\_mng\_al2\_custom\_ami) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2\_custom\_ami\_ipv6](#module\_eks\_mng\_al2\_custom\_ami\_ipv6) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2\_custom\_template](#module\_eks\_mng\_al2\_custom\_template) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2\_disabled](#module\_eks\_mng\_al2\_disabled) | ../../modules/_user_data | n/a |
+| [eks\_mng\_al2\_no\_op](#module\_eks\_mng\_al2\_no\_op) | ../../modules/_user_data | n/a |
| [eks\_mng\_bottlerocket\_additional](#module\_eks\_mng\_bottlerocket\_additional) | ../../modules/_user_data | n/a |
| [eks\_mng\_bottlerocket\_custom\_ami](#module\_eks\_mng\_bottlerocket\_custom\_ami) | ../../modules/_user_data | n/a |
| [eks\_mng\_bottlerocket\_custom\_template](#module\_eks\_mng\_bottlerocket\_custom\_template) | ../../modules/_user_data | n/a |
| [eks\_mng\_bottlerocket\_no\_op](#module\_eks\_mng\_bottlerocket\_no\_op) | ../../modules/_user_data | n/a |
-| [eks\_mng\_linux\_additional](#module\_eks\_mng\_linux\_additional) | ../../modules/_user_data | n/a |
-| [eks\_mng\_linux\_custom\_ami](#module\_eks\_mng\_linux\_custom\_ami) | ../../modules/_user_data | n/a |
-| [eks\_mng\_linux\_custom\_template](#module\_eks\_mng\_linux\_custom\_template) | ../../modules/_user_data | n/a |
-| [eks\_mng\_linux\_no\_op](#module\_eks\_mng\_linux\_no\_op) | ../../modules/_user_data | n/a |
+| [eks\_mng\_windows\_additional](#module\_eks\_mng\_windows\_additional) | ../../modules/_user_data | n/a |
+| [eks\_mng\_windows\_custom\_ami](#module\_eks\_mng\_windows\_custom\_ami) | ../../modules/_user_data | n/a |
+| [eks\_mng\_windows\_custom\_template](#module\_eks\_mng\_windows\_custom\_template) | ../../modules/_user_data | n/a |
+| [eks\_mng\_windows\_no\_op](#module\_eks\_mng\_windows\_no\_op) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2023\_bootstrap](#module\_self\_mng\_al2023\_bootstrap) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2023\_custom\_template](#module\_self\_mng\_al2023\_custom\_template) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2023\_no\_op](#module\_self\_mng\_al2023\_no\_op) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2\_bootstrap](#module\_self\_mng\_al2\_bootstrap) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2\_bootstrap\_ipv6](#module\_self\_mng\_al2\_bootstrap\_ipv6) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2\_custom\_template](#module\_self\_mng\_al2\_custom\_template) | ../../modules/_user_data | n/a |
+| [self\_mng\_al2\_no\_op](#module\_self\_mng\_al2\_no\_op) | ../../modules/_user_data | n/a |
| [self\_mng\_bottlerocket\_bootstrap](#module\_self\_mng\_bottlerocket\_bootstrap) | ../../modules/_user_data | n/a |
| [self\_mng\_bottlerocket\_custom\_template](#module\_self\_mng\_bottlerocket\_custom\_template) | ../../modules/_user_data | n/a |
| [self\_mng\_bottlerocket\_no\_op](#module\_self\_mng\_bottlerocket\_no\_op) | ../../modules/_user_data | n/a |
-| [self\_mng\_linux\_bootstrap](#module\_self\_mng\_linux\_bootstrap) | ../../modules/_user_data | n/a |
-| [self\_mng\_linux\_custom\_template](#module\_self\_mng\_linux\_custom\_template) | ../../modules/_user_data | n/a |
-| [self\_mng\_linux\_no\_op](#module\_self\_mng\_linux\_no\_op) | ../../modules/_user_data | n/a |
| [self\_mng\_windows\_bootstrap](#module\_self\_mng\_windows\_bootstrap) | ../../modules/_user_data | n/a |
| [self\_mng\_windows\_custom\_template](#module\_self\_mng\_windows\_custom\_template) | ../../modules/_user_data | n/a |
| [self\_mng\_windows\_no\_op](#module\_self\_mng\_windows\_no\_op) | ../../modules/_user_data | n/a |
## Resources
-No resources.
+| Name | Type |
+|------|------|
+| [local_file.eks_mng_al2023_additional](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2023_custom_ami](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2023_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2023_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2_additional](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2_custom_ami](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2_custom_ami_ipv6](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_al2_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_bottlerocket_additional](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_bottlerocket_custom_ami](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_bottlerocket_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_bottlerocket_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_windows_additional](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_windows_custom_ami](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_windows_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.eks_mng_windows_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2023_bootstrap](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2023_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2023_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2_bootstrap](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2_bootstrap_ipv6](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_al2_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_bottlerocket_bootstrap](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_bottlerocket_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_bottlerocket_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_windows_bootstrap](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_windows_custom_template](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [local_file.self_mng_windows_no_op](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
## Inputs
@@ -55,23 +103,5 @@ No inputs.
## Outputs
-| Name | Description |
-|------|-------------|
-| [eks\_mng\_bottlerocket\_additional](#output\_eks\_mng\_bottlerocket\_additional) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_bottlerocket\_custom\_ami](#output\_eks\_mng\_bottlerocket\_custom\_ami) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_bottlerocket\_custom\_template](#output\_eks\_mng\_bottlerocket\_custom\_template) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_bottlerocket\_no\_op](#output\_eks\_mng\_bottlerocket\_no\_op) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_linux\_additional](#output\_eks\_mng\_linux\_additional) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_linux\_custom\_ami](#output\_eks\_mng\_linux\_custom\_ami) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_linux\_custom\_template](#output\_eks\_mng\_linux\_custom\_template) | Base64 decoded user data rendered for the provided inputs |
-| [eks\_mng\_linux\_no\_op](#output\_eks\_mng\_linux\_no\_op) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_bottlerocket\_bootstrap](#output\_self\_mng\_bottlerocket\_bootstrap) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_bottlerocket\_custom\_template](#output\_self\_mng\_bottlerocket\_custom\_template) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_bottlerocket\_no\_op](#output\_self\_mng\_bottlerocket\_no\_op) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_linux\_bootstrap](#output\_self\_mng\_linux\_bootstrap) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_linux\_custom\_template](#output\_self\_mng\_linux\_custom\_template) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_linux\_no\_op](#output\_self\_mng\_linux\_no\_op) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_windows\_bootstrap](#output\_self\_mng\_windows\_bootstrap) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_windows\_custom\_template](#output\_self\_mng\_windows\_custom\_template) | Base64 decoded user data rendered for the provided inputs |
-| [self\_mng\_windows\_no\_op](#output\_self\_mng\_windows\_no\_op) | Base64 decoded user data rendered for the provided inputs |
+No outputs.
diff --git a/examples/user_data/main.tf b/examples/user_data/main.tf
index d7d513190e..9a55b3cf77 100644
--- a/examples/user_data/main.tf
+++ b/examples/user_data/main.tf
@@ -4,32 +4,45 @@ locals {
cluster_endpoint = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
cluster_auth_base64 = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="
cluster_service_ipv4_cidr = "172.16.0.0/16"
+ cluster_service_ipv6_cidr = "fdd3:7636:68bc::/108"
+ cluster_service_cidr = "192.168.0.0/16"
}
################################################################################
-# User Data Module
+# EKS managed node group - AL2
################################################################################
-# EKS managed node group - linux
-module "eks_mng_linux_no_op" {
+module "eks_mng_al2_disabled" {
source = "../../modules/_user_data"
+
+ create = false
+}
+
+module "eks_mng_al2_no_op" {
+ source = "../../modules/_user_data"
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
}
-module "eks_mng_linux_additional" {
+module "eks_mng_al2_additional" {
source = "../../modules/_user_data"
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
pre_bootstrap_user_data = <<-EOT
export USE_MAX_PODS=false
EOT
}
-module "eks_mng_linux_custom_ami" {
+module "eks_mng_al2_custom_ami" {
source = "../../modules/_user_data"
- cluster_name = local.name
- cluster_endpoint = local.cluster_endpoint
- cluster_auth_base64 = local.cluster_auth_base64
- cluster_service_ipv4_cidr = local.cluster_service_ipv4_cidr
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_ipv4_cidr
enable_bootstrap_user_data = true
@@ -44,13 +57,35 @@ module "eks_mng_linux_custom_ami" {
EOT
}
+module "eks_mng_al2_custom_ami_ipv6" {
+ source = "../../modules/_user_data"
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_ip_family = "ipv6"
+ cluster_service_cidr = local.cluster_service_ipv6_cidr
+
+ enable_bootstrap_user_data = true
-module "eks_mng_linux_custom_template" {
+ pre_bootstrap_user_data = <<-EOT
+ export FOO=bar
+ EOT
+
+ bootstrap_extra_args = "--kubelet-extra-args '--instance-type t3a.large'"
+
+ post_bootstrap_user_data = <<-EOT
+ echo "All done"
+ EOT
+}
+
+module "eks_mng_al2_custom_template" {
source = "../../modules/_user_data"
- cluster_name = local.name
- cluster_endpoint = local.cluster_endpoint
- cluster_auth_base64 = local.cluster_auth_base64
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_ipv4_cidr
user_data_template_path = "${path.module}/templates/linux_custom.tpl"
@@ -66,17 +101,132 @@ module "eks_mng_linux_custom_template" {
EOT
}
-# EKS managed node group - bottlerocket
+################################################################################
+# EKS managed node group - AL2023
+################################################################################
+
+module "eks_mng_al2023_no_op" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+}
+
+module "eks_mng_al2023_additional" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+}
+
+module "eks_mng_al2023_custom_ami" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_cidr
+
+ enable_bootstrap_user_data = true
+
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+
+ cloudinit_post_nodeadm = [{
+ content = <<-EOT
+ echo "All done"
+ EOT
+ content_type = "text/x-shellscript; charset=\"us-ascii\""
+ }]
+}
+
+module "eks_mng_al2023_custom_template" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_cidr
+
+ enable_bootstrap_user_data = true
+ user_data_template_path = "${path.module}/templates/al2023_custom.tpl"
+
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+
+ cloudinit_post_nodeadm = [{
+ content = <<-EOT
+ echo "All done"
+ EOT
+ content_type = "text/x-shellscript; charset=\"us-ascii\""
+ }]
+}
+
+################################################################################
+# EKS managed node group - Bottlerocket
+################################################################################
+
module "eks_mng_bottlerocket_no_op" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
}
module "eks_mng_bottlerocket_additional" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
+ cluster_service_cidr = local.cluster_service_cidr
bootstrap_extra_args = <<-EOT
# extra args added
@@ -88,11 +238,15 @@ module "eks_mng_bottlerocket_additional" {
module "eks_mng_bottlerocket_custom_ami" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
- cluster_name = local.name
- cluster_endpoint = local.cluster_endpoint
- cluster_auth_base64 = local.cluster_auth_base64
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_cidr
+ additional_cluster_dns_ips = [
+ "169.254.20.10"
+ ]
enable_bootstrap_user_data = true
@@ -106,11 +260,13 @@ module "eks_mng_bottlerocket_custom_ami" {
module "eks_mng_bottlerocket_custom_template" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
cluster_name = local.name
cluster_endpoint = local.cluster_endpoint
cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
user_data_template_path = "${path.module}/templates/bottlerocket_custom.tpl"
@@ -121,23 +277,107 @@ module "eks_mng_bottlerocket_custom_template" {
EOT
}
-# Self managed node group - linux
-module "self_mng_linux_no_op" {
+################################################################################
+# EKS managed node group - Windows
+################################################################################
+
+module "eks_mng_windows_no_op" {
source = "../../modules/_user_data"
- is_eks_managed_node_group = false
+ ami_type = "WINDOWS_CORE_2022_x86_64"
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
}
-module "self_mng_linux_bootstrap" {
+module "eks_mng_windows_additional" {
source = "../../modules/_user_data"
+ ami_type = "WINDOWS_CORE_2022_x86_64"
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
+ pre_bootstrap_user_data = <<-EOT
+ [string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+ EOT
+}
+
+module "eks_mng_windows_custom_ami" {
+ source = "../../modules/_user_data"
+
+ ami_type = "WINDOWS_CORE_2022_x86_64"
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
enable_bootstrap_user_data = true
- is_eks_managed_node_group = false
+
+ pre_bootstrap_user_data = <<-EOT
+ [string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+ EOT
+ # I don't know if this is the right way on Windows, but its just a string check here anyways
+ bootstrap_extra_args = "-KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot"
+
+ post_bootstrap_user_data = <<-EOT
+ [string]$Something = 'IStillDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+ EOT
+}
+
+module "eks_mng_windows_custom_template" {
+ source = "../../modules/_user_data"
+
+ ami_type = "WINDOWS_CORE_2022_x86_64"
cluster_name = local.name
cluster_endpoint = local.cluster_endpoint
cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
+ enable_bootstrap_user_data = true
+
+ user_data_template_path = "${path.module}/templates/windows_custom.tpl"
+
+ pre_bootstrap_user_data = <<-EOT
+ [string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+ EOT
+ # I don't know if this is the right way on Windows, but its just a string check here anyways
+ bootstrap_extra_args = "-KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot"
+
+ post_bootstrap_user_data = <<-EOT
+ [string]$Something = 'IStillDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+ EOT
+}
+
+################################################################################
+# Self-managed node group - AL2
+################################################################################
+
+module "self_mng_al2_no_op" {
+ source = "../../modules/_user_data"
+
+ is_eks_managed_node_group = false
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+}
+
+module "self_mng_al2_bootstrap" {
+ source = "../../modules/_user_data"
+
+ enable_bootstrap_user_data = true
+ is_eks_managed_node_group = false
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_ipv4_cidr
+
pre_bootstrap_user_data = <<-EOT
echo "foo"
export FOO=bar
@@ -150,15 +390,40 @@ module "self_mng_linux_bootstrap" {
EOT
}
-module "self_mng_linux_custom_template" {
+module "self_mng_al2_bootstrap_ipv6" {
source = "../../modules/_user_data"
enable_bootstrap_user_data = true
is_eks_managed_node_group = false
- cluster_name = local.name
- cluster_endpoint = local.cluster_endpoint
- cluster_auth_base64 = local.cluster_auth_base64
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_ip_family = "ipv6"
+ cluster_service_cidr = local.cluster_service_ipv6_cidr
+
+ pre_bootstrap_user_data = <<-EOT
+ echo "foo"
+ export FOO=bar
+ EOT
+
+ bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
+
+ post_bootstrap_user_data = <<-EOT
+ echo "All done"
+ EOT
+}
+
+module "self_mng_al2_custom_template" {
+ source = "../../modules/_user_data"
+
+ enable_bootstrap_user_data = true
+ is_eks_managed_node_group = false
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_ipv4_cidr
user_data_template_path = "${path.module}/templates/linux_custom.tpl"
@@ -174,19 +439,114 @@ module "self_mng_linux_custom_template" {
EOT
}
-# Self managed node group - bottlerocket
+################################################################################
+# Self-managed node group - AL2023
+################################################################################
+
+module "self_mng_al2023_no_op" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ is_eks_managed_node_group = false
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+}
+
+module "self_mng_al2023_bootstrap" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ enable_bootstrap_user_data = true
+ is_eks_managed_node_group = false
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_cidr
+
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+
+ cloudinit_post_nodeadm = [{
+ content = <<-EOT
+ echo "All done"
+ EOT
+ content_type = "text/x-shellscript; charset=\"us-ascii\""
+ }]
+}
+
+module "self_mng_al2023_custom_template" {
+ source = "../../modules/_user_data"
+
+ ami_type = "AL2023_x86_64_STANDARD"
+
+ enable_bootstrap_user_data = true
+ is_eks_managed_node_group = false
+
+ cluster_name = local.name
+ cluster_endpoint = local.cluster_endpoint
+ cluster_auth_base64 = local.cluster_auth_base64
+ cluster_service_cidr = local.cluster_service_cidr
+
+ user_data_template_path = "${path.module}/templates/al2023_custom.tpl"
+
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+
+ cloudinit_post_nodeadm = [{
+ content = <<-EOT
+ echo "All done"
+ EOT
+ content_type = "text/x-shellscript; charset=\"us-ascii\""
+ }]
+}
+
+################################################################################
+# Self-managed node group - Bottlerocket
+################################################################################
+
module "self_mng_bottlerocket_no_op" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
is_eks_managed_node_group = false
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
}
module "self_mng_bottlerocket_bootstrap" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
enable_bootstrap_user_data = true
is_eks_managed_node_group = false
@@ -195,6 +555,9 @@ module "self_mng_bottlerocket_bootstrap" {
cluster_endpoint = local.cluster_endpoint
cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
bootstrap_extra_args = <<-EOT
# extra args added
[settings.kernel]
@@ -205,7 +568,7 @@ module "self_mng_bottlerocket_bootstrap" {
module "self_mng_bottlerocket_custom_template" {
source = "../../modules/_user_data"
- platform = "bottlerocket"
+ ami_type = "BOTTLEROCKET_x86_64"
enable_bootstrap_user_data = true
is_eks_managed_node_group = false
@@ -214,6 +577,9 @@ module "self_mng_bottlerocket_custom_template" {
cluster_endpoint = local.cluster_endpoint
cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
user_data_template_path = "${path.module}/templates/bottlerocket_custom.tpl"
bootstrap_extra_args = <<-EOT
@@ -223,19 +589,25 @@ module "self_mng_bottlerocket_custom_template" {
EOT
}
-# Self managed node group - windows
+################################################################################
+# Self-managed node group - Windows
+################################################################################
+
module "self_mng_windows_no_op" {
source = "../../modules/_user_data"
- platform = "windows"
+ ami_type = "WINDOWS_CORE_2022_x86_64"
is_eks_managed_node_group = false
+
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
}
module "self_mng_windows_bootstrap" {
source = "../../modules/_user_data"
- platform = "windows"
+ ami_type = "WINDOWS_CORE_2022_x86_64"
enable_bootstrap_user_data = true
is_eks_managed_node_group = false
@@ -244,10 +616,13 @@ module "self_mng_windows_bootstrap" {
cluster_endpoint = local.cluster_endpoint
cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
pre_bootstrap_user_data = <<-EOT
[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
EOT
- # I don't know if this is the right way on WindowsOS, but its just a string check here anyways
+ # I don't know if this is the right way on Windows, but its just a string check here anyways
bootstrap_extra_args = "-KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot"
post_bootstrap_user_data = <<-EOT
@@ -258,7 +633,7 @@ module "self_mng_windows_bootstrap" {
module "self_mng_windows_custom_template" {
source = "../../modules/_user_data"
- platform = "windows"
+ ami_type = "WINDOWS_CORE_2022_x86_64"
enable_bootstrap_user_data = true
is_eks_managed_node_group = false
@@ -267,12 +642,15 @@ module "self_mng_windows_custom_template" {
cluster_endpoint = local.cluster_endpoint
cluster_auth_base64 = local.cluster_auth_base64
+ # Hard requirement
+ cluster_service_cidr = local.cluster_service_cidr
+
user_data_template_path = "${path.module}/templates/windows_custom.tpl"
pre_bootstrap_user_data = <<-EOT
[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
EOT
- # I don't know if this is the right way on WindowsOS, but its just a string check here anyways
+ # I don't know if this is the right way on Windows, but its just a string check here anyways
bootstrap_extra_args = "-KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot"
post_bootstrap_user_data = <<-EOT
diff --git a/examples/user_data/outputs.tf b/examples/user_data/outputs.tf
index dd2c3407e1..c40f6632d0 100644
--- a/examples/user_data/outputs.tf
+++ b/examples/user_data/outputs.tf
@@ -1,89 +1,189 @@
-# EKS managed node group - linux
-output "eks_mng_linux_no_op" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_linux_no_op.user_data)
+################################################################################
+# We are writing to local file so that we can better track diffs across changes
+#
+# Its harder to verify changes and diffs when we use the standard `output`
+# route, writing to file makes this easier and better highlights changes
+# to avoid unintended disruptions
+################################################################################
+
+################################################################################
+# EKS managed node group - AL2
+################################################################################
+
+resource "local_file" "eks_mng_al2_no_op" {
+ content = base64decode(module.eks_mng_al2_no_op.user_data)
+ filename = "${path.module}/rendered/al2/eks-mng-no-op.sh"
+}
+
+resource "local_file" "eks_mng_al2_additional" {
+ content = base64decode(module.eks_mng_al2_additional.user_data)
+ filename = "${path.module}/rendered/al2/eks-mng-additional.txt"
+}
+
+resource "local_file" "eks_mng_al2_custom_ami" {
+ content = base64decode(module.eks_mng_al2_custom_ami.user_data)
+ filename = "${path.module}/rendered/al2/eks-mng-custom-ami.sh"
+}
+
+resource "local_file" "eks_mng_al2_custom_ami_ipv6" {
+ content = base64decode(module.eks_mng_al2_custom_ami_ipv6.user_data)
+ filename = "${path.module}/rendered/al2/eks-mng-custom-ami-ipv6.sh"
+}
+
+resource "local_file" "eks_mng_al2_custom_template" {
+ content = base64decode(module.eks_mng_al2_custom_template.user_data)
+ filename = "${path.module}/rendered/al2/eks-mng-custom-template.sh"
}
-output "eks_mng_linux_additional" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_linux_additional.user_data)
+################################################################################
+# EKS managed node group - AL2023
+################################################################################
+
+resource "local_file" "eks_mng_al2023_no_op" {
+ content = base64decode(module.eks_mng_al2023_no_op.user_data)
+ filename = "${path.module}/rendered/al2023/eks-mng-no-op.txt"
}
-output "eks_mng_linux_custom_ami" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_linux_custom_ami.user_data)
+resource "local_file" "eks_mng_al2023_additional" {
+ content = base64decode(module.eks_mng_al2023_additional.user_data)
+ filename = "${path.module}/rendered/al2023/eks-mng-additional.txt"
}
-output "eks_mng_linux_custom_template" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_linux_custom_template.user_data)
+resource "local_file" "eks_mng_al2023_custom_ami" {
+ content = base64decode(module.eks_mng_al2023_custom_ami.user_data)
+ filename = "${path.module}/rendered/al2023/eks-mng-custom-ami.txt"
}
-# EKS managed node group - bottlerocket
-output "eks_mng_bottlerocket_no_op" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_bottlerocket_no_op.user_data)
+resource "local_file" "eks_mng_al2023_custom_template" {
+ content = base64decode(module.eks_mng_al2023_custom_template.user_data)
+ filename = "${path.module}/rendered/al2023/eks-mng-custom-template.txt"
}
-output "eks_mng_bottlerocket_additional" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_bottlerocket_additional.user_data)
+################################################################################
+# EKS managed node group - Bottlerocket
+################################################################################
+
+resource "local_file" "eks_mng_bottlerocket_no_op" {
+ content = base64decode(module.eks_mng_bottlerocket_no_op.user_data)
+ filename = "${path.module}/rendered/bottlerocket/eks-mng-no-op.toml"
}
-output "eks_mng_bottlerocket_custom_ami" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_bottlerocket_custom_ami.user_data)
+resource "local_file" "eks_mng_bottlerocket_additional" {
+ content = base64decode(module.eks_mng_bottlerocket_additional.user_data)
+ filename = "${path.module}/rendered/bottlerocket/eks-mng-additional.toml"
}
-output "eks_mng_bottlerocket_custom_template" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.eks_mng_bottlerocket_custom_template.user_data)
+resource "local_file" "eks_mng_bottlerocket_custom_ami" {
+ content = base64decode(module.eks_mng_bottlerocket_custom_ami.user_data)
+ filename = "${path.module}/rendered/bottlerocket/eks-mng-custom-ami.toml"
}
-# Self managed node group - linux
-output "self_mng_linux_no_op" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_linux_no_op.user_data)
+resource "local_file" "eks_mng_bottlerocket_custom_template" {
+ content = base64decode(module.eks_mng_bottlerocket_custom_template.user_data)
+ filename = "${path.module}/rendered/bottlerocket/eks-mng-custom-template.toml"
}
-output "self_mng_linux_bootstrap" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_linux_bootstrap.user_data)
+################################################################################
+# EKS managed node group - Windows
+################################################################################
+
+resource "local_file" "eks_mng_windows_no_op" {
+ content = base64decode(module.eks_mng_windows_no_op.user_data)
+ filename = "${path.module}/rendered/windows/eks-mng-no-op.ps1"
}
-output "self_mng_linux_custom_template" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_linux_custom_template.user_data)
+resource "local_file" "eks_mng_windows_additional" {
+ content = base64decode(module.eks_mng_windows_additional.user_data)
+ filename = "${path.module}/rendered/windows/eks-mng-additional.ps1"
}
-# Self managed node group - bottlerocket
-output "self_mng_bottlerocket_no_op" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_bottlerocket_no_op.user_data)
+resource "local_file" "eks_mng_windows_custom_ami" {
+ content = base64decode(module.eks_mng_windows_custom_ami.user_data)
+ filename = "${path.module}/rendered/windows/eks-mng-custom-ami.ps1"
+}
+
+resource "local_file" "eks_mng_windows_custom_template" {
+ content = base64decode(module.eks_mng_windows_custom_template.user_data)
+ filename = "${path.module}/rendered/windows/eks-mng-custom-template.ps1"
+}
+
+################################################################################
+# Self-managed node group - AL2
+################################################################################
+
+resource "local_file" "self_mng_al2_no_op" {
+ content = base64decode(module.self_mng_al2_no_op.user_data)
+ filename = "${path.module}/rendered/al2/self-mng-no-op.sh"
}
-output "self_mng_bottlerocket_bootstrap" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_bottlerocket_bootstrap.user_data)
+resource "local_file" "self_mng_al2_bootstrap" {
+ content = base64decode(module.self_mng_al2_bootstrap.user_data)
+ filename = "${path.module}/rendered/al2/self-mng-bootstrap.sh"
}
-output "self_mng_bottlerocket_custom_template" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_bottlerocket_custom_template.user_data)
+resource "local_file" "self_mng_al2_bootstrap_ipv6" {
+ content = base64decode(module.self_mng_al2_bootstrap_ipv6.user_data)
+ filename = "${path.module}/rendered/al2/self-mng-bootstrap-ipv6.sh"
}
-# Self managed node group - windows
-output "self_mng_windows_no_op" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_windows_no_op.user_data)
+resource "local_file" "self_mng_al2_custom_template" {
+ content = base64decode(module.self_mng_al2_custom_template.user_data)
+ filename = "${path.module}/rendered/al2/self-mng-custom-template.sh"
+}
+
+################################################################################
+# Self-managed node group - AL2023
+################################################################################
+
+resource "local_file" "self_mng_al2023_no_op" {
+ content = base64decode(module.self_mng_al2023_no_op.user_data)
+ filename = "${path.module}/rendered/al2023/self-mng-no-op.txt"
+}
+
+resource "local_file" "self_mng_al2023_bootstrap" {
+ content = base64decode(module.self_mng_al2023_bootstrap.user_data)
+ filename = "${path.module}/rendered/al2023/self-mng-bootstrap.txt"
+}
+
+resource "local_file" "self_mng_al2023_custom_template" {
+ content = base64decode(module.self_mng_al2023_custom_template.user_data)
+ filename = "${path.module}/rendered/al2023/self-mng-custom-template.txt"
+}
+
+################################################################################
+# Self-managed node group - Bottlerocket
+################################################################################
+
+resource "local_file" "self_mng_bottlerocket_no_op" {
+ content = base64decode(module.self_mng_bottlerocket_no_op.user_data)
+ filename = "${path.module}/rendered/bottlerocket/self-mng-no-op.toml"
+}
+
+resource "local_file" "self_mng_bottlerocket_bootstrap" {
+ content = base64decode(module.self_mng_bottlerocket_bootstrap.user_data)
+ filename = "${path.module}/rendered/bottlerocket/self-mng-bootstrap.toml"
+}
+
+resource "local_file" "self_mng_bottlerocket_custom_template" {
+ content = base64decode(module.self_mng_bottlerocket_custom_template.user_data)
+ filename = "${path.module}/rendered/bottlerocket/self-mng-custom-template.toml"
+}
+
+################################################################################
+# Self-managed node group - Windows
+################################################################################
+
+resource "local_file" "self_mng_windows_no_op" {
+ content = base64decode(module.self_mng_windows_no_op.user_data)
+ filename = "${path.module}/rendered/windows/self-mng-no-op.ps1"
}
-output "self_mng_windows_bootstrap" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_windows_bootstrap.user_data)
+resource "local_file" "self_mng_windows_bootstrap" {
+ content = base64decode(module.self_mng_windows_bootstrap.user_data)
+ filename = "${path.module}/rendered/windows/self-mng-bootstrap.ps1"
}
-output "self_mng_windows_custom_template" {
- description = "Base64 decoded user data rendered for the provided inputs"
- value = base64decode(module.self_mng_windows_custom_template.user_data)
+resource "local_file" "self_mng_windows_custom_template" {
+ content = base64decode(module.self_mng_windows_custom_template.user_data)
+ filename = "${path.module}/rendered/windows/self-mng-custom-template.ps1"
}
diff --git a/examples/user_data/rendered/al2/eks-mng-additional.txt b/examples/user_data/rendered/al2/eks-mng-additional.txt
new file mode 100755
index 0000000000..151f0cba7a
--- /dev/null
+++ b/examples/user_data/rendered/al2/eks-mng-additional.txt
@@ -0,0 +1,11 @@
+Content-Type: multipart/mixed; boundary="//"
+MIME-Version: 1.0
+
+--//
+Content-Transfer-Encoding: 7bit
+Content-Type: text/x-shellscript
+Mime-Version: 1.0
+
+export USE_MAX_PODS=false
+
+--//--
diff --git a/examples/user_data/rendered/al2/eks-mng-custom-ami-ipv6.sh b/examples/user_data/rendered/al2/eks-mng-custom-ami-ipv6.sh
new file mode 100755
index 0000000000..fceb7e3571
--- /dev/null
+++ b/examples/user_data/rendered/al2/eks-mng-custom-ami-ipv6.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+export FOO=bar
+B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+API_SERVER_URL=https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+/etc/eks/bootstrap.sh ex-user-data --kubelet-extra-args '--instance-type t3a.large' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ipv6 --service-ipv6-cidr fdd3:7636:68bc::/108
+echo "All done"
diff --git a/examples/user_data/rendered/al2/eks-mng-custom-ami.sh b/examples/user_data/rendered/al2/eks-mng-custom-ami.sh
new file mode 100755
index 0000000000..c7d92a7ce4
--- /dev/null
+++ b/examples/user_data/rendered/al2/eks-mng-custom-ami.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+export FOO=bar
+B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+API_SERVER_URL=https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+/etc/eks/bootstrap.sh ex-user-data --kubelet-extra-args '--instance-type t3a.large' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ipv4 --service-ipv4-cidr 172.16.0.0/16
+echo "All done"
diff --git a/examples/user_data/rendered/al2/eks-mng-custom-template.sh b/examples/user_data/rendered/al2/eks-mng-custom-template.sh
new file mode 100755
index 0000000000..e18460fa1d
--- /dev/null
+++ b/examples/user_data/rendered/al2/eks-mng-custom-template.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+set -ex
+
+echo "foo"
+export FOO=bar
+
+# Custom user data template provided for rendering
+B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+API_SERVER_URL=https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+/etc/eks/bootstrap.sh ex-user-data --kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ipv4 --service-ipv4-cidr 172.16.0.0/16
+echo "All done"
diff --git a/examples/complete/variables.tf b/examples/user_data/rendered/al2/eks-mng-no-op.sh
old mode 100644
new mode 100755
similarity index 100%
rename from examples/complete/variables.tf
rename to examples/user_data/rendered/al2/eks-mng-no-op.sh
diff --git a/examples/user_data/rendered/al2/self-mng-bootstrap-ipv6.sh b/examples/user_data/rendered/al2/self-mng-bootstrap-ipv6.sh
new file mode 100755
index 0000000000..b6fd557a13
--- /dev/null
+++ b/examples/user_data/rendered/al2/self-mng-bootstrap-ipv6.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+set -e
+echo "foo"
+export FOO=bar
+B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+API_SERVER_URL=https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+/etc/eks/bootstrap.sh ex-user-data --kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ipv6 --service-ipv6-cidr fdd3:7636:68bc::/108
+echo "All done"
diff --git a/examples/user_data/rendered/al2/self-mng-bootstrap.sh b/examples/user_data/rendered/al2/self-mng-bootstrap.sh
new file mode 100755
index 0000000000..7fcd81973e
--- /dev/null
+++ b/examples/user_data/rendered/al2/self-mng-bootstrap.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+set -e
+echo "foo"
+export FOO=bar
+B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+API_SERVER_URL=https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+/etc/eks/bootstrap.sh ex-user-data --kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ipv4 --service-ipv4-cidr 172.16.0.0/16
+echo "All done"
diff --git a/examples/user_data/rendered/al2/self-mng-custom-template.sh b/examples/user_data/rendered/al2/self-mng-custom-template.sh
new file mode 100755
index 0000000000..e18460fa1d
--- /dev/null
+++ b/examples/user_data/rendered/al2/self-mng-custom-template.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+set -ex
+
+echo "foo"
+export FOO=bar
+
+# Custom user data template provided for rendering
+B64_CLUSTER_CA=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+API_SERVER_URL=https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+/etc/eks/bootstrap.sh ex-user-data --kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ipv4 --service-ipv4-cidr 172.16.0.0/16
+echo "All done"
diff --git a/examples/user_data/rendered/al2/self-mng-no-op.sh b/examples/user_data/rendered/al2/self-mng-no-op.sh
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/rendered/al2023/eks-mng-additional.txt b/examples/user_data/rendered/al2023/eks-mng-additional.txt
new file mode 100755
index 0000000000..fe3c75c898
--- /dev/null
+++ b/examples/user_data/rendered/al2023/eks-mng-additional.txt
@@ -0,0 +1,19 @@
+Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
+MIME-Version: 1.0
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+
+--MIMEBOUNDARY--
diff --git a/examples/user_data/rendered/al2023/eks-mng-custom-ami.txt b/examples/user_data/rendered/al2023/eks-mng-custom-ami.txt
new file mode 100755
index 0000000000..46362c2030
--- /dev/null
+++ b/examples/user_data/rendered/al2023/eks-mng-custom-ami.txt
@@ -0,0 +1,41 @@
+Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
+MIME-Version: 1.0
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ cluster:
+ name: ex-user-data
+ apiServerEndpoint: https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+ certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+ cidr: 192.168.0.0/16
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: text/x-shellscript; charset="us-ascii"
+Mime-Version: 1.0
+
+echo "All done"
+
+--MIMEBOUNDARY--
diff --git a/examples/user_data/rendered/al2023/eks-mng-custom-template.txt b/examples/user_data/rendered/al2023/eks-mng-custom-template.txt
new file mode 100755
index 0000000000..a97e188c83
--- /dev/null
+++ b/examples/user_data/rendered/al2023/eks-mng-custom-template.txt
@@ -0,0 +1,45 @@
+Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
+MIME-Version: 1.0
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ cluster:
+ name: ex-user-data
+ apiServerEndpoint: https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+ certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+ cidr: 192.168.0.0/16
+ containerd:
+ config: |
+ [plugins."io.containerd.grpc.v1.cri".containerd]
+ discard_unpacked_layers = false
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: text/x-shellscript; charset="us-ascii"
+Mime-Version: 1.0
+
+echo "All done"
+
+--MIMEBOUNDARY--
diff --git a/examples/user_data/rendered/al2023/eks-mng-no-op.txt b/examples/user_data/rendered/al2023/eks-mng-no-op.txt
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/rendered/al2023/self-mng-bootstrap.txt b/examples/user_data/rendered/al2023/self-mng-bootstrap.txt
new file mode 100755
index 0000000000..46362c2030
--- /dev/null
+++ b/examples/user_data/rendered/al2023/self-mng-bootstrap.txt
@@ -0,0 +1,41 @@
+Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
+MIME-Version: 1.0
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ cluster:
+ name: ex-user-data
+ apiServerEndpoint: https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+ certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+ cidr: 192.168.0.0/16
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: text/x-shellscript; charset="us-ascii"
+Mime-Version: 1.0
+
+echo "All done"
+
+--MIMEBOUNDARY--
diff --git a/examples/user_data/rendered/al2023/self-mng-custom-template.txt b/examples/user_data/rendered/al2023/self-mng-custom-template.txt
new file mode 100755
index 0000000000..a97e188c83
--- /dev/null
+++ b/examples/user_data/rendered/al2023/self-mng-custom-template.txt
@@ -0,0 +1,45 @@
+Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
+MIME-Version: 1.0
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: application/node.eks.aws
+Mime-Version: 1.0
+
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ cluster:
+ name: ex-user-data
+ apiServerEndpoint: https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com
+ certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
+ cidr: 192.168.0.0/16
+ containerd:
+ config: |
+ [plugins."io.containerd.grpc.v1.cri".containerd]
+ discard_unpacked_layers = false
+
+--MIMEBOUNDARY
+Content-Transfer-Encoding: 7bit
+Content-Type: text/x-shellscript; charset="us-ascii"
+Mime-Version: 1.0
+
+echo "All done"
+
+--MIMEBOUNDARY--
diff --git a/examples/user_data/rendered/al2023/self-mng-no-op.txt b/examples/user_data/rendered/al2023/self-mng-no-op.txt
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/rendered/bottlerocket/eks-mng-additional.toml b/examples/user_data/rendered/bottlerocket/eks-mng-additional.toml
new file mode 100755
index 0000000000..7ed4affaf6
--- /dev/null
+++ b/examples/user_data/rendered/bottlerocket/eks-mng-additional.toml
@@ -0,0 +1,3 @@
+# extra args added
+[settings.kernel]
+lockdown = "integrity"
diff --git a/examples/user_data/rendered/bottlerocket/eks-mng-custom-ami.toml b/examples/user_data/rendered/bottlerocket/eks-mng-custom-ami.toml
new file mode 100755
index 0000000000..38b0c46a0b
--- /dev/null
+++ b/examples/user_data/rendered/bottlerocket/eks-mng-custom-ami.toml
@@ -0,0 +1,8 @@
+[settings.kubernetes]
+"cluster-name" = "ex-user-data"
+"api-server" = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
+"cluster-certificate" = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="
+"cluster-dns-ip" = ["192.168.0.10", "169.254.20.10"]
+# extra args added
+[settings.kernel]
+lockdown = "integrity"
diff --git a/examples/user_data/rendered/bottlerocket/eks-mng-custom-template.toml b/examples/user_data/rendered/bottlerocket/eks-mng-custom-template.toml
new file mode 100755
index 0000000000..c5c6774cfc
--- /dev/null
+++ b/examples/user_data/rendered/bottlerocket/eks-mng-custom-template.toml
@@ -0,0 +1,9 @@
+# Custom user data template provided for rendering
+[settings.kubernetes]
+"cluster-name" = "ex-user-data"
+"api-server" = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
+"cluster-certificate" = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="
+
+# extra args added
+[settings.kernel]
+lockdown = "integrity"
diff --git a/examples/user_data/rendered/bottlerocket/eks-mng-no-op.toml b/examples/user_data/rendered/bottlerocket/eks-mng-no-op.toml
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/rendered/bottlerocket/self-mng-bootstrap.toml b/examples/user_data/rendered/bottlerocket/self-mng-bootstrap.toml
new file mode 100755
index 0000000000..76f8b82dcd
--- /dev/null
+++ b/examples/user_data/rendered/bottlerocket/self-mng-bootstrap.toml
@@ -0,0 +1,8 @@
+[settings.kubernetes]
+"cluster-name" = "ex-user-data"
+"api-server" = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
+"cluster-certificate" = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="
+"cluster-dns-ip" = ["192.168.0.10"]
+# extra args added
+[settings.kernel]
+lockdown = "integrity"
diff --git a/examples/user_data/rendered/bottlerocket/self-mng-custom-template.toml b/examples/user_data/rendered/bottlerocket/self-mng-custom-template.toml
new file mode 100755
index 0000000000..c5c6774cfc
--- /dev/null
+++ b/examples/user_data/rendered/bottlerocket/self-mng-custom-template.toml
@@ -0,0 +1,9 @@
+# Custom user data template provided for rendering
+[settings.kubernetes]
+"cluster-name" = "ex-user-data"
+"api-server" = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
+"cluster-certificate" = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="
+
+# extra args added
+[settings.kernel]
+lockdown = "integrity"
diff --git a/examples/user_data/rendered/bottlerocket/self-mng-no-op.toml b/examples/user_data/rendered/bottlerocket/self-mng-no-op.toml
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/rendered/windows/eks-mng-additional.ps1 b/examples/user_data/rendered/windows/eks-mng-additional.ps1
new file mode 100755
index 0000000000..0debfcf9ad
--- /dev/null
+++ b/examples/user_data/rendered/windows/eks-mng-additional.ps1
@@ -0,0 +1 @@
+[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
diff --git a/examples/user_data/rendered/windows/eks-mng-custom-ami.ps1 b/examples/user_data/rendered/windows/eks-mng-custom-ami.ps1
new file mode 100755
index 0000000000..182195b707
--- /dev/null
+++ b/examples/user_data/rendered/windows/eks-mng-custom-ami.ps1
@@ -0,0 +1,9 @@
+
+[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
+[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
+[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
+& $EKSBootstrapScriptFile -EKSClusterName ex-user-data -APIServerEndpoint https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com -Base64ClusterCA LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ== -KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot 3>&1 4>&1 5>&1 6>&1
+$LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
+[string]$Something = 'IStillDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+
diff --git a/examples/user_data/rendered/windows/eks-mng-custom-template.ps1 b/examples/user_data/rendered/windows/eks-mng-custom-template.ps1
new file mode 100755
index 0000000000..aa4008c7e5
--- /dev/null
+++ b/examples/user_data/rendered/windows/eks-mng-custom-template.ps1
@@ -0,0 +1,10 @@
+# Custom user data template provided for rendering
+
+[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
+[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
+[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
+& $EKSBootstrapScriptFile -EKSClusterName ex-user-data -APIServerEndpoint https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com -Base64ClusterCA LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ== -KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot 3>&1 4>&1 5>&1 6>&1
+$LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
+[string]$Something = 'IStillDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+
diff --git a/examples/user_data/rendered/windows/eks-mng-no-op.ps1 b/examples/user_data/rendered/windows/eks-mng-no-op.ps1
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/rendered/windows/self-mng-bootstrap.ps1 b/examples/user_data/rendered/windows/self-mng-bootstrap.ps1
new file mode 100755
index 0000000000..182195b707
--- /dev/null
+++ b/examples/user_data/rendered/windows/self-mng-bootstrap.ps1
@@ -0,0 +1,9 @@
+
+[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
+[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
+[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
+& $EKSBootstrapScriptFile -EKSClusterName ex-user-data -APIServerEndpoint https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com -Base64ClusterCA LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ== -KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot 3>&1 4>&1 5>&1 6>&1
+$LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
+[string]$Something = 'IStillDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+
diff --git a/examples/user_data/rendered/windows/self-mng-custom-template.ps1 b/examples/user_data/rendered/windows/self-mng-custom-template.ps1
new file mode 100755
index 0000000000..aa4008c7e5
--- /dev/null
+++ b/examples/user_data/rendered/windows/self-mng-custom-template.ps1
@@ -0,0 +1,10 @@
+# Custom user data template provided for rendering
+
+[string]$Something = 'IDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
+[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
+[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
+& $EKSBootstrapScriptFile -EKSClusterName ex-user-data -APIServerEndpoint https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com -Base64ClusterCA LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ== -KubeletExtraArgs --node-labels=node.kubernetes.io/lifecycle=spot 3>&1 4>&1 5>&1 6>&1
+$LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
+[string]$Something = 'IStillDoNotKnowAnyPowerShell ¯\_(ツ)_/¯'
+
diff --git a/examples/user_data/rendered/windows/self-mng-no-op.ps1 b/examples/user_data/rendered/windows/self-mng-no-op.ps1
new file mode 100755
index 0000000000..e69de29bb2
diff --git a/examples/user_data/templates/al2023_custom.tpl b/examples/user_data/templates/al2023_custom.tpl
new file mode 100644
index 0000000000..34c566c154
--- /dev/null
+++ b/examples/user_data/templates/al2023_custom.tpl
@@ -0,0 +1,15 @@
+%{ if enable_bootstrap_user_data ~}
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ cluster:
+ name: ${cluster_name}
+ apiServerEndpoint: ${cluster_endpoint}
+ certificateAuthority: ${cluster_auth_base64}
+ cidr: ${cluster_service_cidr}
+ containerd:
+ config: |
+ [plugins."io.containerd.grpc.v1.cri".containerd]
+ discard_unpacked_layers = false
+%{ endif ~}
diff --git a/examples/user_data/templates/linux_custom.tpl b/examples/user_data/templates/linux_custom.tpl
index bfe21f117a..b3cb73a2ab 100644
--- a/examples/user_data/templates/linux_custom.tpl
+++ b/examples/user_data/templates/linux_custom.tpl
@@ -6,5 +6,6 @@ ${pre_bootstrap_user_data ~}
# Custom user data template provided for rendering
B64_CLUSTER_CA=${cluster_auth_base64}
API_SERVER_URL=${cluster_endpoint}
-/etc/eks/bootstrap.sh ${cluster_name} ${bootstrap_extra_args} --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
+/etc/eks/bootstrap.sh ${cluster_name} ${bootstrap_extra_args} --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ${cluster_ip_family} --service-${cluster_ip_family}-cidr ${cluster_service_cidr}
${post_bootstrap_user_data ~}
diff --git a/examples/user_data/versions.tf b/examples/user_data/versions.tf
index 7117131f4c..31969d6c02 100644
--- a/examples/user_data/versions.tf
+++ b/examples/user_data/versions.tf
@@ -1,3 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
+
+ required_providers {
+ local = {
+ source = "hashicorp/local"
+ version = ">= 2.4"
+ }
+ }
}
diff --git a/main.tf b/main.tf
index 52d36aa201..4cb1200327 100644
--- a/main.tf
+++ b/main.tf
@@ -12,6 +12,8 @@ data "aws_iam_session_context" "current" {
locals {
create = var.create && var.putin_khuylo
+ partition = data.aws_partition.current.partition
+
cluster_role = try(aws_iam_role.this[0].arn, var.iam_role_arn)
create_outposts_local_cluster = length(var.outpost_config) > 0
@@ -30,6 +32,17 @@ resource "aws_eks_cluster" "this" {
version = var.cluster_version
enabled_cluster_log_types = var.cluster_enabled_log_types
+ access_config {
+ authentication_mode = var.authentication_mode
+
+ # See access entries below - this is a one time operation from the EKS API.
+ # Instead, we are hardcoding this to false and if users wish to achieve this
+ # same functionality, we will do that through an access entry which can be
+ # enabled or disabled at any time of their choosing using the variable
+ # var.enable_cluster_creator_admin_permissions
+ bootstrap_cluster_creator_admin_permissions = false
+ }
+
vpc_config {
security_group_ids = compact(distinct(concat(var.cluster_additional_security_group_ids, [local.cluster_security_group_id])))
subnet_ids = coalescelist(var.control_plane_subnet_ids, var.subnet_ids)
@@ -71,14 +84,15 @@ resource "aws_eks_cluster" "this" {
}
tags = merge(
+ { terraform-aws-modules = "eks" },
var.tags,
var.cluster_tags,
)
timeouts {
- create = lookup(var.cluster_timeouts, "create", null)
- update = lookup(var.cluster_timeouts, "update", null)
- delete = lookup(var.cluster_timeouts, "delete", null)
+ create = try(var.cluster_timeouts.create, null)
+ update = try(var.cluster_timeouts.update, null)
+ delete = try(var.cluster_timeouts.delete, null)
}
depends_on = [
@@ -88,6 +102,12 @@ resource "aws_eks_cluster" "this" {
aws_cloudwatch_log_group.this,
aws_iam_policy.cni_ipv6_policy,
]
+
+ lifecycle {
+ ignore_changes = [
+ access_config[0].bootstrap_cluster_creator_admin_permissions
+ ]
+ }
}
resource "aws_ec2_tag" "cluster_primary_security_group" {
@@ -109,20 +129,102 @@ resource "aws_cloudwatch_log_group" "this" {
name = "/aws/eks/${var.cluster_name}/cluster"
retention_in_days = var.cloudwatch_log_group_retention_in_days
kms_key_id = var.cloudwatch_log_group_kms_key_id
+ log_group_class = var.cloudwatch_log_group_class
tags = merge(
var.tags,
+ var.cloudwatch_log_group_tags,
{ Name = "/aws/eks/${var.cluster_name}/cluster" }
)
}
+################################################################################
+# Access Entry
+################################################################################
+
+locals {
+ # This replaces the one time logic from the EKS API with something that can be
+ # better controlled by users through Terraform
+ bootstrap_cluster_creator_admin_permissions = {
+ cluster_creator = {
+ principal_arn = data.aws_iam_session_context.current.issuer_arn
+ type = "STANDARD"
+
+ policy_associations = {
+ admin = {
+ policy_arn = "arn:${local.partition}:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
+ access_scope = {
+ type = "cluster"
+ }
+ }
+ }
+ }
+ }
+
+ # Merge the bootstrap behavior with the entries that users provide
+ merged_access_entries = merge(
+ { for k, v in local.bootstrap_cluster_creator_admin_permissions : k => v if var.enable_cluster_creator_admin_permissions },
+ var.access_entries,
+ )
+
+ # Flatten out entries and policy associations so users can specify the policy
+ # associations within a single entry
+ flattened_access_entries = flatten([
+ for entry_key, entry_val in local.merged_access_entries : [
+ for pol_key, pol_val in lookup(entry_val, "policy_associations", {}) :
+ merge(
+ {
+ principal_arn = entry_val.principal_arn
+ entry_key = entry_key
+ pol_key = pol_key
+ },
+ { for k, v in {
+ association_policy_arn = pol_val.policy_arn
+ association_access_scope_type = pol_val.access_scope.type
+ association_access_scope_namespaces = lookup(pol_val.access_scope, "namespaces", [])
+ } : k => v if !contains(["EC2_LINUX", "EC2_WINDOWS", "FARGATE_LINUX"], lookup(entry_val, "type", "STANDARD")) },
+ )
+ ]
+ ])
+}
+
+resource "aws_eks_access_entry" "this" {
+ for_each = { for k, v in local.merged_access_entries : k => v if local.create }
+
+ cluster_name = aws_eks_cluster.this[0].name
+ kubernetes_groups = try(each.value.kubernetes_groups, null)
+ principal_arn = each.value.principal_arn
+ type = try(each.value.type, "STANDARD")
+ user_name = try(each.value.user_name, null)
+
+ tags = merge(var.tags, try(each.value.tags, {}))
+}
+
+resource "aws_eks_access_policy_association" "this" {
+ for_each = { for k, v in local.flattened_access_entries : "${v.entry_key}_${v.pol_key}" => v if local.create }
+
+ access_scope {
+ namespaces = try(each.value.association_access_scope_namespaces, [])
+ type = each.value.association_access_scope_type
+ }
+
+ cluster_name = aws_eks_cluster.this[0].name
+
+ policy_arn = each.value.association_policy_arn
+ principal_arn = each.value.principal_arn
+
+ depends_on = [
+ aws_eks_access_entry.this,
+ ]
+}
+
################################################################################
# KMS Key
################################################################################
module "kms" {
source = "terraform-aws-modules/kms/aws"
- version = "1.1.0" # Note - be mindful of Terraform/provider version compatibility between modules
+ version = "2.1.0" # Note - be mindful of Terraform/provider version compatibility between modules
create = local.create && var.create_kms_key && local.enable_cluster_encryption_config # not valid on Outposts
@@ -147,7 +249,10 @@ module "kms" {
cluster = { name = "eks/${var.cluster_name}" }
}
- tags = var.tags
+ tags = merge(
+ { terraform-aws-modules = "eks" },
+ var.tags,
+ )
}
################################################################################
@@ -220,19 +325,26 @@ resource "aws_security_group_rule" "cluster" {
# Note - this is different from EKS identity provider
################################################################################
+locals {
+ # Not available on outposts
+ create_oidc_provider = local.create && var.enable_irsa && !local.create_outposts_local_cluster
+
+ oidc_root_ca_thumbprint = local.create_oidc_provider && var.include_oidc_root_ca_thumbprint ? [data.tls_certificate.this[0].certificates[0].sha1_fingerprint] : []
+}
+
data "tls_certificate" "this" {
# Not available on outposts
- count = local.create && var.enable_irsa && !local.create_outposts_local_cluster ? 1 : 0
+ count = local.create_oidc_provider && var.include_oidc_root_ca_thumbprint ? 1 : 0
url = aws_eks_cluster.this[0].identity[0].oidc[0].issuer
}
resource "aws_iam_openid_connect_provider" "oidc_provider" {
# Not available on outposts
- count = local.create && var.enable_irsa && !local.create_outposts_local_cluster ? 1 : 0
+ count = local.create_oidc_provider ? 1 : 0
- client_id_list = distinct(compact(concat(["sts.${local.dns_suffix}"], var.openid_connect_audiences)))
- thumbprint_list = concat(data.tls_certificate.this[0].certificates[*].sha1_fingerprint, var.custom_oidc_thumbprints)
+ client_id_list = distinct(compact(concat(["sts.amazonaws.com"], var.openid_connect_audiences)))
+ thumbprint_list = concat(local.oidc_root_ca_thumbprint, var.custom_oidc_thumbprints)
url = aws_eks_cluster.this[0].identity[0].oidc[0].issuer
tags = merge(
@@ -248,13 +360,9 @@ resource "aws_iam_openid_connect_provider" "oidc_provider" {
locals {
create_iam_role = local.create && var.create_iam_role
iam_role_name = coalesce(var.iam_role_name, "${var.cluster_name}-cluster")
- iam_role_policy_prefix = "arn:${data.aws_partition.current.partition}:iam::aws:policy"
+ iam_role_policy_prefix = "arn:${local.partition}:iam::aws:policy"
cluster_encryption_policy_name = coalesce(var.cluster_encryption_policy_name, "${local.iam_role_name}-ClusterEncryption")
-
- # TODO - hopefully this can be removed once the AWS endpoint is named properly in China
- # https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1904
- dns_suffix = coalesce(var.cluster_iam_role_dns_suffix, data.aws_partition.current.dns_suffix)
}
data "aws_iam_policy_document" "assume_role_policy" {
@@ -266,7 +374,7 @@ data "aws_iam_policy_document" "assume_role_policy" {
principals {
type = "Service"
- identifiers = ["eks.${local.dns_suffix}"]
+ identifiers = ["eks.amazonaws.com"]
}
dynamic "principals" {
@@ -275,7 +383,7 @@ data "aws_iam_policy_document" "assume_role_policy" {
content {
type = "Service"
identifiers = [
- "ec2.${local.dns_suffix}",
+ "ec2.amazonaws.com",
]
}
}
@@ -379,6 +487,14 @@ resource "aws_iam_policy" "cluster_encryption" {
# EKS Addons
################################################################################
+data "aws_eks_addon_version" "this" {
+ for_each = { for k, v in var.cluster_addons : k => v if local.create && !local.create_outposts_local_cluster }
+
+ addon_name = try(each.value.name, each.key)
+ kubernetes_version = coalesce(var.cluster_version, aws_eks_cluster.this[0].version)
+ most_recent = try(each.value.most_recent, null)
+}
+
resource "aws_eks_addon" "this" {
# Not supported on outposts
for_each = { for k, v in var.cluster_addons : k => v if !try(v.before_compute, false) && local.create && !local.create_outposts_local_cluster }
@@ -386,11 +502,12 @@ resource "aws_eks_addon" "this" {
cluster_name = aws_eks_cluster.this[0].name
addon_name = try(each.value.name, each.key)
- addon_version = try(each.value.addon_version, data.aws_eks_addon_version.this[each.key].version)
- configuration_values = try(each.value.configuration_values, null)
- preserve = try(each.value.preserve, null)
- resolve_conflicts = try(each.value.resolve_conflicts, "OVERWRITE")
- service_account_role_arn = try(each.value.service_account_role_arn, null)
+ addon_version = coalesce(try(each.value.addon_version, null), data.aws_eks_addon_version.this[each.key].version)
+ configuration_values = try(each.value.configuration_values, null)
+ preserve = try(each.value.preserve, true)
+ resolve_conflicts_on_create = try(each.value.resolve_conflicts_on_create, "OVERWRITE")
+ resolve_conflicts_on_update = try(each.value.resolve_conflicts_on_update, "OVERWRITE")
+ service_account_role_arn = try(each.value.service_account_role_arn, null)
timeouts {
create = try(each.value.timeouts.create, var.cluster_addons_timeouts.create, null)
@@ -404,7 +521,7 @@ resource "aws_eks_addon" "this" {
module.self_managed_node_group,
]
- tags = var.tags
+ tags = merge(var.tags, try(each.value.tags, {}))
}
resource "aws_eks_addon" "before_compute" {
@@ -414,11 +531,12 @@ resource "aws_eks_addon" "before_compute" {
cluster_name = aws_eks_cluster.this[0].name
addon_name = try(each.value.name, each.key)
- addon_version = try(each.value.addon_version, data.aws_eks_addon_version.this[each.key].version)
- configuration_values = try(each.value.configuration_values, null)
- preserve = try(each.value.preserve, null)
- resolve_conflicts = try(each.value.resolve_conflicts, "OVERWRITE")
- service_account_role_arn = try(each.value.service_account_role_arn, null)
+ addon_version = coalesce(try(each.value.addon_version, null), data.aws_eks_addon_version.this[each.key].version)
+ configuration_values = try(each.value.configuration_values, null)
+ preserve = try(each.value.preserve, true)
+ resolve_conflicts_on_create = try(each.value.resolve_conflicts_on_create, "OVERWRITE")
+ resolve_conflicts_on_update = try(each.value.resolve_conflicts_on_update, "OVERWRITE")
+ service_account_role_arn = try(each.value.service_account_role_arn, null)
timeouts {
create = try(each.value.timeouts.create, var.cluster_addons_timeouts.create, null)
@@ -426,15 +544,7 @@ resource "aws_eks_addon" "before_compute" {
delete = try(each.value.timeouts.delete, var.cluster_addons_timeouts.delete, null)
}
- tags = var.tags
-}
-
-data "aws_eks_addon_version" "this" {
- for_each = { for k, v in var.cluster_addons : k => v if local.create && !local.create_outposts_local_cluster }
-
- addon_name = try(each.value.name, each.key)
- kubernetes_version = coalesce(var.cluster_version, aws_eks_cluster.this[0].version)
- most_recent = try(each.value.most_recent, null)
+ tags = merge(var.tags, try(each.value.tags, {}))
}
################################################################################
@@ -442,6 +552,14 @@ data "aws_eks_addon_version" "this" {
# Note - this is different from IRSA
################################################################################
+locals {
+ # Maintain current behavior for <= 1.29, remove default for >= 1.30
+ # `null` will return the latest Kubernetes version from the EKS API, which at time of writing is 1.30
+ # https://github.com/kubernetes/kubernetes/pull/123561
+ idpc_backwards_compat_version = contains(["1.21", "1.22", "1.23", "1.24", "1.25", "1.26", "1.27", "1.28", "1.29"], coalesce(var.cluster_version, "1.30"))
+ idpc_issuer_url = local.idpc_backwards_compat_version ? try(aws_eks_cluster.this[0].identity[0].oidc[0].issuer, null) : null
+}
+
resource "aws_eks_identity_provider_config" "this" {
for_each = { for k, v in var.cluster_identity_providers : k => v if local.create && !local.create_outposts_local_cluster }
@@ -452,118 +570,12 @@ resource "aws_eks_identity_provider_config" "this" {
groups_claim = lookup(each.value, "groups_claim", null)
groups_prefix = lookup(each.value, "groups_prefix", null)
identity_provider_config_name = try(each.value.identity_provider_config_name, each.key)
- issuer_url = try(each.value.issuer_url, aws_eks_cluster.this[0].identity[0].oidc[0].issuer)
- required_claims = lookup(each.value, "required_claims", null)
- username_claim = lookup(each.value, "username_claim", null)
- username_prefix = lookup(each.value, "username_prefix", null)
- }
-
- tags = var.tags
-}
-
-################################################################################
-# aws-auth configmap
-################################################################################
-
-locals {
- node_iam_role_arns_non_windows = distinct(
- compact(
- concat(
- [for group in module.eks_managed_node_group : group.iam_role_arn],
- [for group in module.self_managed_node_group : group.iam_role_arn if group.platform != "windows"],
- var.aws_auth_node_iam_role_arns_non_windows,
- )
- )
- )
-
- node_iam_role_arns_windows = distinct(
- compact(
- concat(
- [for group in module.self_managed_node_group : group.iam_role_arn if group.platform == "windows"],
- var.aws_auth_node_iam_role_arns_windows,
- )
- )
- )
-
- fargate_profile_pod_execution_role_arns = distinct(
- compact(
- concat(
- [for group in module.fargate_profile : group.fargate_profile_pod_execution_role_arn],
- var.aws_auth_fargate_profile_pod_execution_role_arns,
- )
- )
- )
-
- aws_auth_configmap_data = {
- mapRoles = yamlencode(concat(
- [for role_arn in local.node_iam_role_arns_non_windows : {
- rolearn = role_arn
- username = "system:node:{{EC2PrivateDNSName}}"
- groups = [
- "system:bootstrappers",
- "system:nodes",
- ]
- }
- ],
- [for role_arn in local.node_iam_role_arns_windows : {
- rolearn = role_arn
- username = "system:node:{{EC2PrivateDNSName}}"
- groups = [
- "eks:kube-proxy-windows",
- "system:bootstrappers",
- "system:nodes",
- ]
- }
- ],
- # Fargate profile
- [for role_arn in local.fargate_profile_pod_execution_role_arns : {
- rolearn = role_arn
- username = "system:node:{{SessionName}}"
- groups = [
- "system:bootstrappers",
- "system:nodes",
- "system:node-proxier",
- ]
- }
- ],
- var.aws_auth_roles
- ))
- mapUsers = yamlencode(var.aws_auth_users)
- mapAccounts = yamlencode(var.aws_auth_accounts)
- }
-}
-
-resource "kubernetes_config_map" "aws_auth" {
- count = var.create && var.create_aws_auth_configmap ? 1 : 0
-
- metadata {
- name = "aws-auth"
- namespace = "kube-system"
+ # TODO - make argument explicitly required on next breaking change
+ issuer_url = try(each.value.issuer_url, local.idpc_issuer_url)
+ required_claims = lookup(each.value, "required_claims", null)
+ username_claim = lookup(each.value, "username_claim", null)
+ username_prefix = lookup(each.value, "username_prefix", null)
}
- data = local.aws_auth_configmap_data
-
- lifecycle {
- # We are ignoring the data here since we will manage it with the resource below
- # This is only intended to be used in scenarios where the configmap does not exist
- ignore_changes = [data, metadata[0].labels, metadata[0].annotations]
- }
-}
-
-resource "kubernetes_config_map_v1_data" "aws_auth" {
- count = var.create && var.manage_aws_auth_configmap ? 1 : 0
-
- force = true
-
- metadata {
- name = "aws-auth"
- namespace = "kube-system"
- }
-
- data = local.aws_auth_configmap_data
-
- depends_on = [
- # Required for instances where the configmap does not exist yet to avoid race condition
- kubernetes_config_map.aws_auth,
- ]
+ tags = merge(var.tags, try(each.value.tags, {}))
}
diff --git a/modules/_user_data/README.md b/modules/_user_data/README.md
index 0853fd9e1a..e5207d9443 100644
--- a/modules/_user_data/README.md
+++ b/modules/_user_data/README.md
@@ -9,14 +9,16 @@ See [`examples/user_data/`](https://github.com/terraform-aws-modules/terraform-a
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
| [cloudinit](#requirement\_cloudinit) | >= 2.0 |
+| [null](#requirement\_null) | >= 3.0 |
## Providers
| Name | Version |
|------|---------|
| [cloudinit](#provider\_cloudinit) | >= 2.0 |
+| [null](#provider\_null) | >= 3.0 |
## Modules
@@ -26,28 +28,37 @@ No modules.
| Name | Type |
|------|------|
+| [null_resource.validate_cluster_service_cidr](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
+| [cloudinit_config.al2023_eks_managed_node_group](https://registry.terraform.io/providers/hashicorp/cloudinit/latest/docs/data-sources/config) | data source |
| [cloudinit_config.linux_eks_managed_node_group](https://registry.terraform.io/providers/hashicorp/cloudinit/latest/docs/data-sources/config) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
-| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `platform` = `bottlerocket`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
+| [additional\_cluster\_dns\_ips](#input\_additional\_cluster\_dns\_ips) | Additional DNS IP addresses to use for the cluster. Only used when `ami_type` = `BOTTLEROCKET_*` | `list(string)` | `[]` | no |
+| [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values | `string` | `null` | no |
+| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
+| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
| [cluster\_auth\_base64](#input\_cluster\_auth\_base64) | Base64 encoded CA of associated EKS cluster | `string` | `""` | no |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Endpoint of associated EKS cluster | `string` | `""` | no |
+| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
| [cluster\_name](#input\_cluster\_name) | Name of the EKS cluster | `string` | `""` | no |
-| [cluster\_service\_ipv4\_cidr](#input\_cluster\_service\_ipv4\_cidr) | The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks | `string` | `null` | no |
+| [cluster\_service\_cidr](#input\_cluster\_service\_cidr) | The CIDR block (IPv4 or IPv6) used by the cluster to assign Kubernetes service IP addresses. This is derived from the cluster itself | `string` | `""` | no |
+| [cluster\_service\_ipv4\_cidr](#input\_cluster\_service\_ipv4\_cidr) | [Deprecated] The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks | `string` | `null` | no |
| [create](#input\_create) | Determines whether to create user-data or not | `bool` | `true` | no |
| [enable\_bootstrap\_user\_data](#input\_enable\_bootstrap\_user\_data) | Determines whether the bootstrap configurations are populated within the user data template | `bool` | `false` | no |
| [is\_eks\_managed\_node\_group](#input\_is\_eks\_managed\_node\_group) | Determines whether the user data is used on nodes in an EKS managed node group. Used to determine if user data will be appended or not | `bool` | `true` | no |
-| [platform](#input\_platform) | Identifies if the OS platform is `bottlerocket`, `linux`, or `windows` based | `string` | `"linux"` | no |
-| [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no |
-| [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no |
+| [platform](#input\_platform) | [DEPRECATED - use `ami_type` instead. Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows` | `string` | `"linux"` | no |
+| [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*` | `string` | `""` | no |
+| [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*` | `string` | `""` | no |
| [user\_data\_template\_path](#input\_user\_data\_template\_path) | Path to a local, custom user data template file to use when rendering user data | `string` | `""` | no |
## Outputs
| Name | Description |
|------|-------------|
+| [platform](#output\_platform) | [DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023, or `windows |
| [user\_data](#output\_user\_data) | Base64 encoded user data rendered for the provided inputs |
diff --git a/modules/_user_data/main.tf b/modules/_user_data/main.tf
index 8ace10539d..b695ba6cca 100644
--- a/modules/_user_data/main.tf
+++ b/modules/_user_data/main.tf
@@ -1,70 +1,99 @@
+# The `cluster_service_cidr` is required when `create == true`
+# This is a hacky way to make that logic work, otherwise Terraform always wants a value
+# and supplying any old value like `""` or `null` is not valid and will silently
+# fail to join nodes to the cluster
+resource "null_resource" "validate_cluster_service_cidr" {
+ lifecycle {
+ precondition {
+ # The length 6 is currently arbitrary, but it's a safe bet that the CIDR will be longer than that
+ # The main point is that a value needs to be provided when `create = true`
+ condition = var.create ? length(local.cluster_service_cidr) > 6 : true
+ error_message = "`cluster_service_cidr` is required when `create = true`."
+ }
+ }
+}
locals {
- int_linux_default_user_data = var.create && var.platform == "linux" && (var.enable_bootstrap_user_data || var.user_data_template_path != "") ? base64encode(templatefile(
- coalesce(var.user_data_template_path, "${path.module}/../../templates/linux_user_data.tpl"),
+ # Converts AMI type into user data type that represents the underlying format (bash, toml, PS1, nodeadm)
+ # TODO - platform will be removed in v21.0 and only `ami_type` will be valid
+ ami_type_to_user_data_type = {
+ AL2_x86_64 = "linux"
+ AL2_x86_64_GPU = "linux"
+ AL2_ARM_64 = "linux"
+ BOTTLEROCKET_ARM_64 = "bottlerocket"
+ BOTTLEROCKET_x86_64 = "bottlerocket"
+ BOTTLEROCKET_ARM_64_NVIDIA = "bottlerocket"
+ BOTTLEROCKET_x86_64_NVIDIA = "bottlerocket"
+ WINDOWS_CORE_2019_x86_64 = "windows"
+ WINDOWS_FULL_2019_x86_64 = "windows"
+ WINDOWS_CORE_2022_x86_64 = "windows"
+ WINDOWS_FULL_2022_x86_64 = "windows"
+ AL2023_x86_64_STANDARD = "al2023"
+ AL2023_ARM_64_STANDARD = "al2023"
+ }
+ # Try to use `ami_type` first, but fall back to current, default behavior
+ # TODO - will be removed in v21.0
+ user_data_type = try(local.ami_type_to_user_data_type[var.ami_type], var.platform)
+
+ template_path = {
+ al2023 = "${path.module}/../../templates/al2023_user_data.tpl"
+ bottlerocket = "${path.module}/../../templates/bottlerocket_user_data.tpl"
+ linux = "${path.module}/../../templates/linux_user_data.tpl"
+ windows = "${path.module}/../../templates/windows_user_data.tpl"
+ }
+
+ cluster_service_cidr = try(coalesce(var.cluster_service_ipv4_cidr, var.cluster_service_cidr), "")
+ cluster_dns_ips = flatten(concat([try(cidrhost(local.cluster_service_cidr, 10), "")], var.additional_cluster_dns_ips))
+
+ user_data = base64encode(templatefile(
+ coalesce(var.user_data_template_path, local.template_path[local.user_data_type]),
{
# https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-custom-ami
enable_bootstrap_user_data = var.enable_bootstrap_user_data
+
# Required to bootstrap node
cluster_name = var.cluster_name
cluster_endpoint = var.cluster_endpoint
cluster_auth_base64 = var.cluster_auth_base64
+
+ cluster_service_cidr = local.cluster_service_cidr
+ cluster_ip_family = var.cluster_ip_family
+
+ # Bottlerocket
+ cluster_dns_ips = "[${join(", ", formatlist("\"%s\"", local.cluster_dns_ips))}]"
+
# Optional
- cluster_service_ipv4_cidr = var.cluster_service_ipv4_cidr != null ? var.cluster_service_ipv4_cidr : ""
- bootstrap_extra_args = var.bootstrap_extra_args
- pre_bootstrap_user_data = var.pre_bootstrap_user_data
- post_bootstrap_user_data = var.post_bootstrap_user_data
+ bootstrap_extra_args = var.bootstrap_extra_args
+ pre_bootstrap_user_data = var.pre_bootstrap_user_data
+ post_bootstrap_user_data = var.post_bootstrap_user_data
+ }
+ ))
+
+ user_data_type_to_rendered = {
+ al2023 = {
+ user_data = var.create ? try(data.cloudinit_config.al2023_eks_managed_node_group[0].rendered, local.user_data) : ""
}
- )) : ""
- platform = {
bottlerocket = {
- user_data = var.create && var.platform == "bottlerocket" && (var.enable_bootstrap_user_data || var.user_data_template_path != "" || var.bootstrap_extra_args != "") ? base64encode(templatefile(
- coalesce(var.user_data_template_path, "${path.module}/../../templates/bottlerocket_user_data.tpl"),
- {
- # https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-custom-ami
- enable_bootstrap_user_data = var.enable_bootstrap_user_data
- # Required to bootstrap node
- cluster_name = var.cluster_name
- cluster_endpoint = var.cluster_endpoint
- cluster_auth_base64 = var.cluster_auth_base64
- # Optional - is appended if using EKS managed node group without custom AMI
- # cluster_service_ipv4_cidr = var.cluster_service_ipv4_cidr # Bottlerocket pulls this automatically https://github.com/bottlerocket-os/bottlerocket/issues/1866
- bootstrap_extra_args = var.bootstrap_extra_args
- }
- )) : ""
+ user_data = var.create && local.user_data_type == "bottlerocket" && (var.enable_bootstrap_user_data || var.user_data_template_path != "" || var.bootstrap_extra_args != "") ? local.user_data : ""
}
linux = {
- user_data = try(data.cloudinit_config.linux_eks_managed_node_group[0].rendered, local.int_linux_default_user_data)
-
+ user_data = var.create ? try(data.cloudinit_config.linux_eks_managed_node_group[0].rendered, local.user_data) : ""
}
windows = {
- user_data = var.create && var.platform == "windows" && var.enable_bootstrap_user_data ? base64encode(templatefile(
- coalesce(var.user_data_template_path, "${path.module}/../../templates/windows_user_data.tpl"),
- {
- # Required to bootstrap node
- cluster_name = var.cluster_name
- cluster_endpoint = var.cluster_endpoint
- cluster_auth_base64 = var.cluster_auth_base64
- # Optional - is appended if using EKS managed node group without custom AMI
- # cluster_service_ipv4_cidr = var.cluster_service_ipv4_cidr # Not supported yet: https://github.com/awslabs/amazon-eks-ami/issues/805
- bootstrap_extra_args = var.bootstrap_extra_args
- pre_bootstrap_user_data = var.pre_bootstrap_user_data
- post_bootstrap_user_data = var.post_bootstrap_user_data
- }
- )) : ""
+ user_data = var.create && local.user_data_type == "windows" && (var.enable_bootstrap_user_data || var.user_data_template_path != "" || var.pre_bootstrap_user_data != "") ? local.user_data : ""
}
}
}
# https://github.com/aws/containers-roadmap/issues/596#issuecomment-675097667
-# An important note is that user data must in MIME multi-part archive format,
+# Managed nodegroup data must in MIME multi-part archive format,
# as by default, EKS will merge the bootstrapping command required for nodes to join the
# cluster with your user data. If you use a custom AMI in your launch template,
# this merging will NOT happen and you are responsible for nodes joining the cluster.
# See docs for more details -> https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-user-data
data "cloudinit_config" "linux_eks_managed_node_group" {
- count = var.create && var.platform == "linux" && var.is_eks_managed_node_group && !var.enable_bootstrap_user_data && var.pre_bootstrap_user_data != "" && var.user_data_template_path == "" ? 1 : 0
+ count = var.create && local.user_data_type == "linux" && var.is_eks_managed_node_group && !var.enable_bootstrap_user_data && var.pre_bootstrap_user_data != "" && var.user_data_template_path == "" ? 1 : 0
base64_encode = true
gzip = false
@@ -72,7 +101,44 @@ data "cloudinit_config" "linux_eks_managed_node_group" {
# Prepend to existing user data supplied by AWS EKS
part {
- content_type = "text/x-shellscript"
content = var.pre_bootstrap_user_data
+ content_type = "text/x-shellscript"
+ }
+}
+
+# Scenarios:
+#
+# 1. Do nothing - provide nothing
+# 2. Prepend stuff on EKS MNG (before EKS MNG adds its bit at the end)
+# 3. Own all of the stuff on self-MNG or EKS MNG w/ custom AMI
+
+locals {
+ nodeadm_cloudinit = var.enable_bootstrap_user_data ? concat(
+ var.cloudinit_pre_nodeadm,
+ [{
+ content_type = "application/node.eks.aws"
+ content = base64decode(local.user_data)
+ }],
+ var.cloudinit_post_nodeadm
+ ) : var.cloudinit_pre_nodeadm
+}
+
+data "cloudinit_config" "al2023_eks_managed_node_group" {
+ count = var.create && local.user_data_type == "al2023" && length(local.nodeadm_cloudinit) > 0 ? 1 : 0
+
+ base64_encode = true
+ gzip = false
+ boundary = "MIMEBOUNDARY"
+
+ dynamic "part" {
+ # Using the index is fine in this context since any change in user data will be a replacement
+ for_each = { for i, v in local.nodeadm_cloudinit : i => v }
+
+ content {
+ content = part.value.content
+ content_type = try(part.value.content_type, null)
+ filename = try(part.value.filename, null)
+ merge_type = try(part.value.merge_type, null)
+ }
}
}
diff --git a/modules/_user_data/outputs.tf b/modules/_user_data/outputs.tf
index 075801b233..7bebb3f218 100644
--- a/modules/_user_data/outputs.tf
+++ b/modules/_user_data/outputs.tf
@@ -1,4 +1,9 @@
output "user_data" {
description = "Base64 encoded user data rendered for the provided inputs"
- value = try(local.platform[var.platform].user_data, null)
+ value = try(local.user_data_type_to_rendered[local.user_data_type].user_data, null)
+}
+
+output "platform" {
+ description = "[DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023, or `windows`"
+ value = local.user_data_type
}
diff --git a/modules/_user_data/variables.tf b/modules/_user_data/variables.tf
index 232e1e883e..d5a1ef1bd3 100644
--- a/modules/_user_data/variables.tf
+++ b/modules/_user_data/variables.tf
@@ -5,11 +5,17 @@ variable "create" {
}
variable "platform" {
- description = "Identifies if the OS platform is `bottlerocket`, `linux`, or `windows` based"
+ description = "[DEPRECATED - use `ami_type` instead. Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows`"
type = string
default = "linux"
}
+variable "ami_type" {
+ description = "Type of Amazon Machine Image (AMI) associated with the EKS Node Group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values"
+ type = string
+ default = null
+}
+
variable "enable_bootstrap_user_data" {
description = "Determines whether the bootstrap configurations are populated within the user data template"
type = bool
@@ -40,26 +46,45 @@ variable "cluster_auth_base64" {
default = ""
}
+variable "cluster_service_cidr" {
+ description = "The CIDR block (IPv4 or IPv6) used by the cluster to assign Kubernetes service IP addresses. This is derived from the cluster itself"
+ type = string
+ default = ""
+}
+
+variable "cluster_ip_family" {
+ description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
+ type = string
+ default = "ipv4"
+}
+
+variable "additional_cluster_dns_ips" {
+ description = "Additional DNS IP addresses to use for the cluster. Only used when `ami_type` = `BOTTLEROCKET_*`"
+ type = list(string)
+ default = []
+}
+
+# TODO - remove at next breaking change
variable "cluster_service_ipv4_cidr" {
- description = "The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks"
+ description = "[Deprecated] The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks"
type = string
default = null
}
variable "pre_bootstrap_user_data" {
- description = "User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket`"
+ description = "User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*`"
type = string
default = ""
}
variable "post_bootstrap_user_data" {
- description = "User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket`"
+ description = "User data that is appended to the user data script after of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*`"
type = string
default = ""
}
variable "bootstrap_extra_args" {
- description = "Additional arguments passed to the bootstrap script. When `platform` = `bottlerocket`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data"
+ description = "Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data"
type = string
default = ""
}
@@ -69,3 +94,25 @@ variable "user_data_template_path" {
type = string
default = ""
}
+
+variable "cloudinit_pre_nodeadm" {
+ description = "Array of cloud-init document parts that are created before the nodeadm document part"
+ type = list(object({
+ content = string
+ content_type = optional(string)
+ filename = optional(string)
+ merge_type = optional(string)
+ }))
+ default = []
+}
+
+variable "cloudinit_post_nodeadm" {
+ description = "Array of cloud-init document parts that are created after the nodeadm document part"
+ type = list(object({
+ content = string
+ content_type = optional(string)
+ filename = optional(string)
+ merge_type = optional(string)
+ }))
+ default = []
+}
diff --git a/modules/_user_data/versions.tf b/modules/_user_data/versions.tf
index 2dbd12cdc0..9219addeda 100644
--- a/modules/_user_data/versions.tf
+++ b/modules/_user_data/versions.tf
@@ -1,10 +1,14 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
cloudinit = {
source = "hashicorp/cloudinit"
version = ">= 2.0"
}
+ null = {
+ source = "hashicorp/null"
+ version = ">= 3.0"
+ }
}
}
diff --git a/modules/aws-auth/README.md b/modules/aws-auth/README.md
new file mode 100644
index 0000000000..5ba490b7f1
--- /dev/null
+++ b/modules/aws-auth/README.md
@@ -0,0 +1,81 @@
+# `aws-auth` Module
+
+Configuration in this directory creates/updates the `aws-auth` ConfigMap.
+
+```hcl
+module "eks" {
+ source = "terraform-aws-modules/eks/aws//modules/aws-auth"
+ version = "~> 20.0"
+
+ manage_aws_auth_configmap = true
+
+ aws_auth_roles = [
+ {
+ rolearn = "arn:aws:iam::66666666666:role/role1"
+ username = "role1"
+ groups = ["system:masters"]
+ },
+ ]
+
+ aws_auth_users = [
+ {
+ userarn = "arn:aws:iam::66666666666:user/user1"
+ username = "user1"
+ groups = ["system:masters"]
+ },
+ {
+ userarn = "arn:aws:iam::66666666666:user/user2"
+ username = "user2"
+ groups = ["system:masters"]
+ },
+ ]
+
+ aws_auth_accounts = [
+ "777777777777",
+ "888888888888",
+ ]
+}
+```
+
+## Usage
+
+
+## Requirements
+
+| Name | Version |
+|------|---------|
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [kubernetes](#requirement\_kubernetes) | >= 2.20 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| [kubernetes](#provider\_kubernetes) | >= 2.20 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [kubernetes_config_map.aws_auth](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/config_map) | resource |
+| [kubernetes_config_map_v1_data.aws_auth](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/config_map_v1_data) | resource |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| [aws\_auth\_accounts](#input\_aws\_auth\_accounts) | List of account maps to add to the aws-auth configmap | `list(any)` | `[]` | no |
+| [aws\_auth\_roles](#input\_aws\_auth\_roles) | List of role maps to add to the aws-auth configmap | `list(any)` | `[]` | no |
+| [aws\_auth\_users](#input\_aws\_auth\_users) | List of user maps to add to the aws-auth configmap | `list(any)` | `[]` | no |
+| [create](#input\_create) | Controls if resources should be created (affects all resources) | `bool` | `true` | no |
+| [create\_aws\_auth\_configmap](#input\_create\_aws\_auth\_configmap) | Determines whether to create the aws-auth configmap. NOTE - this is only intended for scenarios where the configmap does not exist (i.e. - when using only self-managed node groups). Most users should use `manage_aws_auth_configmap` | `bool` | `false` | no |
+| [manage\_aws\_auth\_configmap](#input\_manage\_aws\_auth\_configmap) | Determines whether to manage the aws-auth configmap | `bool` | `true` | no |
+
+## Outputs
+
+No outputs.
+
diff --git a/modules/aws-auth/main.tf b/modules/aws-auth/main.tf
new file mode 100644
index 0000000000..2f7e9694a7
--- /dev/null
+++ b/modules/aws-auth/main.tf
@@ -0,0 +1,47 @@
+
+################################################################################
+# aws-auth configmap
+################################################################################
+
+locals {
+ aws_auth_configmap_data = {
+ mapRoles = yamlencode(var.aws_auth_roles)
+ mapUsers = yamlencode(var.aws_auth_users)
+ mapAccounts = yamlencode(var.aws_auth_accounts)
+ }
+}
+
+resource "kubernetes_config_map" "aws_auth" {
+ count = var.create && var.create_aws_auth_configmap ? 1 : 0
+
+ metadata {
+ name = "aws-auth"
+ namespace = "kube-system"
+ }
+
+ data = local.aws_auth_configmap_data
+
+ lifecycle {
+ # We are ignoring the data here since we will manage it with the resource below
+ # This is only intended to be used in scenarios where the configmap does not exist
+ ignore_changes = [data, metadata[0].labels, metadata[0].annotations]
+ }
+}
+
+resource "kubernetes_config_map_v1_data" "aws_auth" {
+ count = var.create && var.manage_aws_auth_configmap ? 1 : 0
+
+ force = true
+
+ metadata {
+ name = "aws-auth"
+ namespace = "kube-system"
+ }
+
+ data = local.aws_auth_configmap_data
+
+ depends_on = [
+ # Required for instances where the configmap does not exist yet to avoid race condition
+ kubernetes_config_map.aws_auth,
+ ]
+}
diff --git a/modules/aws-auth/outputs.tf b/modules/aws-auth/outputs.tf
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/modules/aws-auth/variables.tf b/modules/aws-auth/variables.tf
new file mode 100644
index 0000000000..3aaeb023e3
--- /dev/null
+++ b/modules/aws-auth/variables.tf
@@ -0,0 +1,39 @@
+variable "create" {
+ description = "Controls if resources should be created (affects all resources)"
+ type = bool
+ default = true
+}
+
+################################################################################
+# aws-auth ConfigMap
+################################################################################
+
+variable "create_aws_auth_configmap" {
+ description = "Determines whether to create the aws-auth configmap. NOTE - this is only intended for scenarios where the configmap does not exist (i.e. - when using only self-managed node groups). Most users should use `manage_aws_auth_configmap`"
+ type = bool
+ default = false
+}
+
+variable "manage_aws_auth_configmap" {
+ description = "Determines whether to manage the aws-auth configmap"
+ type = bool
+ default = true
+}
+
+variable "aws_auth_roles" {
+ description = "List of role maps to add to the aws-auth configmap"
+ type = list(any)
+ default = []
+}
+
+variable "aws_auth_users" {
+ description = "List of user maps to add to the aws-auth configmap"
+ type = list(any)
+ default = []
+}
+
+variable "aws_auth_accounts" {
+ description = "List of account maps to add to the aws-auth configmap"
+ type = list(any)
+ default = []
+}
diff --git a/modules/aws-auth/versions.tf b/modules/aws-auth/versions.tf
new file mode 100644
index 0000000000..f330045476
--- /dev/null
+++ b/modules/aws-auth/versions.tf
@@ -0,0 +1,10 @@
+terraform {
+ required_version = ">= 1.3.2"
+
+ required_providers {
+ kubernetes = {
+ source = "hashicorp/kubernetes"
+ version = ">= 2.20"
+ }
+ }
+}
diff --git a/modules/eks-managed-node-group/README.md b/modules/eks-managed-node-group/README.md
index bf3a35976a..9cc5680b84 100644
--- a/modules/eks-managed-node-group/README.md
+++ b/modules/eks-managed-node-group/README.md
@@ -63,14 +63,14 @@ module "eks_managed_node_group" {
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.47 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.47 |
+| [aws](#provider\_aws) | >= 5.40 |
## Modules
@@ -88,32 +88,41 @@ module "eks_managed_node_group" {
| [aws_iam_role_policy_attachment.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_launch_template.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template) | resource |
+| [aws_placement_group.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/placement_group) | resource |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
+| [aws_ec2_instance_type.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type) | data source |
+| [aws_ec2_instance_type_offerings.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type_offerings) | data source |
| [aws_iam_policy_document.assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [aws_ssm_parameter.ami](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |
+| [aws_subnets.efa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| [ami\_id](#input\_ami\_id) | The AMI from which to launch the instance. If not supplied, EKS will use its own default image | `string` | `""` | no |
-| [ami\_release\_version](#input\_ami\_release\_version) | AMI version of the EKS Node Group. Defaults to latest version for Kubernetes version | `string` | `null` | no |
-| [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group. Valid values are `AL2_x86_64`, `AL2_x86_64_GPU`, `AL2_ARM_64`, `CUSTOM`, `BOTTLEROCKET_ARM_64`, `BOTTLEROCKET_x86_64` | `string` | `null` | no |
+| [ami\_release\_version](#input\_ami\_release\_version) | The AMI version. Defaults to latest AMI release version for the given Kubernetes version and AMI type | `string` | `null` | no |
+| [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values | `string` | `null` | no |
| [block\_device\_mappings](#input\_block\_device\_mappings) | Specify volumes to attach to the instance besides the volumes specified by the AMI | `any` | `{}` | no |
-| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `platform` = `bottlerocket`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
+| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
| [capacity\_reservation\_specification](#input\_capacity\_reservation\_specification) | Targeting for EC2 capacity reservations | `any` | `{}` | no |
| [capacity\_type](#input\_capacity\_type) | Type of capacity associated with the EKS Node Group. Valid values: `ON_DEMAND`, `SPOT` | `string` | `"ON_DEMAND"` | no |
+| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
| [cluster\_auth\_base64](#input\_cluster\_auth\_base64) | Base64 encoded CA of associated EKS cluster | `string` | `""` | no |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Endpoint of associated EKS cluster | `string` | `""` | no |
-| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `null` | no |
+| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
| [cluster\_name](#input\_cluster\_name) | Name of associated EKS cluster | `string` | `null` | no |
| [cluster\_primary\_security\_group\_id](#input\_cluster\_primary\_security\_group\_id) | The ID of the EKS cluster primary security group to associate with the instance(s). This is the security group that is automatically created by the EKS service | `string` | `null` | no |
-| [cluster\_service\_ipv4\_cidr](#input\_cluster\_service\_ipv4\_cidr) | The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks | `string` | `null` | no |
+| [cluster\_service\_cidr](#input\_cluster\_service\_cidr) | The CIDR block (IPv4 or IPv6) used by the cluster to assign Kubernetes service IP addresses. This is derived from the cluster itself | `string` | `""` | no |
+| [cluster\_service\_ipv4\_cidr](#input\_cluster\_service\_ipv4\_cidr) | [Deprecated] The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks | `string` | `null` | no |
| [cluster\_version](#input\_cluster\_version) | Kubernetes version. Defaults to EKS Cluster Kubernetes version | `string` | `null` | no |
| [cpu\_options](#input\_cpu\_options) | The CPU options for the instance | `map(string)` | `{}` | no |
| [create](#input\_create) | Determines whether to create EKS managed node group or not | `bool` | `true` | no |
| [create\_iam\_role](#input\_create\_iam\_role) | Determines whether an IAM role is created or to use an existing IAM role | `bool` | `true` | no |
| [create\_launch\_template](#input\_create\_launch\_template) | Determines whether to create a launch template or not. If set to `false`, EKS will use its own default launch template | `bool` | `true` | no |
+| [create\_placement\_group](#input\_create\_placement\_group) | Determines whether a placement group is created & used by the nodegroup | `bool` | `false` | no |
| [create\_schedule](#input\_create\_schedule) | Determines whether to create autoscaling group schedule or not | `bool` | `true` | no |
| [credit\_specification](#input\_credit\_specification) | Customize the credit specification of the instance | `map(string)` | `{}` | no |
| [desired\_size](#input\_desired\_size) | Desired number of instances/nodes | `number` | `1` | no |
@@ -123,6 +132,7 @@ module "eks_managed_node_group" {
| [elastic\_gpu\_specifications](#input\_elastic\_gpu\_specifications) | The elastic GPU to attach to the instance | `any` | `{}` | no |
| [elastic\_inference\_accelerator](#input\_elastic\_inference\_accelerator) | Configuration block containing an Elastic Inference Accelerator to attach to the instance | `map(string)` | `{}` | no |
| [enable\_bootstrap\_user\_data](#input\_enable\_bootstrap\_user\_data) | Determines whether the bootstrap configurations are populated within the user data template. Only valid when using a custom AMI via `ami_id` | `bool` | `false` | no |
+| [enable\_efa\_support](#input\_enable\_efa\_support) | Determines whether to enable Elastic Fabric Adapter (EFA) support | `bool` | `false` | no |
| [enable\_monitoring](#input\_enable\_monitoring) | Enables/disables detailed monitoring | `bool` | `true` | no |
| [enclave\_options](#input\_enclave\_options) | Enable Nitro Enclaves on launched instances | `map(string)` | `{}` | no |
| [force\_update\_version](#input\_force\_update\_version) | Force version update if existing pods are unable to be drained due to a pod disruption budget issue | `bool` | `null` | no |
@@ -155,9 +165,10 @@ module "eks_managed_node_group" {
| [name](#input\_name) | Name of the EKS managed node group | `string` | `""` | no |
| [network\_interfaces](#input\_network\_interfaces) | Customize network interfaces to be attached at instance boot time | `list(any)` | `[]` | no |
| [placement](#input\_placement) | The placement of the instance | `map(string)` | `{}` | no |
-| [platform](#input\_platform) | Identifies if the OS platform is `bottlerocket` or `linux` based; `windows` is not supported | `string` | `"linux"` | no |
-| [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no |
-| [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no |
+| [placement\_group\_strategy](#input\_placement\_group\_strategy) | The placement group strategy | `string` | `"cluster"` | no |
+| [platform](#input\_platform) | [DEPRECATED - use `ami_type` instead. Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows` | `string` | `"linux"` | no |
+| [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*` | `string` | `""` | no |
+| [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*` | `string` | `""` | no |
| [private\_dns\_name\_options](#input\_private\_dns\_name\_options) | The options for the instance hostname. The default values are inherited from the subnet | `map(string)` | `{}` | no |
| [ram\_disk\_id](#input\_ram\_disk\_id) | The ID of the ram disk | `string` | `null` | no |
| [remote\_access](#input\_remote\_access) | Configuration block with remote access settings. Only valid when `use_custom_launch_template` = `false` | `any` | `{}` | no |
@@ -170,6 +181,7 @@ module "eks_managed_node_group" {
| [update\_config](#input\_update\_config) | Configuration block of settings for max unavailable resources during node group updates | `map(string)` | {
"max_unavailable_percentage": 33
} | no |
| [update\_launch\_template\_default\_version](#input\_update\_launch\_template\_default\_version) | Whether to update the launch templates default version on each update. Conflicts with `launch_template_default_version` | `bool` | `true` | no |
| [use\_custom\_launch\_template](#input\_use\_custom\_launch\_template) | Determines whether to use a custom launch template or not. If set to `false`, EKS will use its own default launch template | `bool` | `true` | no |
+| [use\_latest\_ami\_release\_version](#input\_use\_latest\_ami\_release\_version) | Determines whether to use the latest AMI release version for the given `ami_type` (except for `CUSTOM`). Note: `ami_type` and `cluster_version` must be supplied in order to enable this feature | `bool` | `false` | no |
| [use\_name\_prefix](#input\_use\_name\_prefix) | Determines whether to use `name` as is or create a unique name beginning with the `name` as the prefix | `bool` | `true` | no |
| [user\_data\_template\_path](#input\_user\_data\_template\_path) | Path to a local, custom user data template file to use when rendering user data | `string` | `""` | no |
| [vpc\_security\_group\_ids](#input\_vpc\_security\_group\_ids) | A list of security group IDs to associate | `list(string)` | `[]` | no |
@@ -193,4 +205,5 @@ module "eks_managed_node_group" {
| [node\_group\_resources](#output\_node\_group\_resources) | List of objects containing information about underlying resources |
| [node\_group\_status](#output\_node\_group\_status) | Status of the EKS Node Group |
| [node\_group\_taints](#output\_node\_group\_taints) | List of objects containing information about taints applied to the node group |
+| [platform](#output\_platform) | [DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows` |
diff --git a/modules/eks-managed-node-group/main.tf b/modules/eks-managed-node-group/main.tf
index 8ca23782dc..82fe075204 100644
--- a/modules/eks-managed-node-group/main.tf
+++ b/modules/eks-managed-node-group/main.tf
@@ -10,18 +10,49 @@ module "user_data" {
create = var.create
platform = var.platform
+ ami_type = var.ami_type
- cluster_name = var.cluster_name
- cluster_endpoint = var.cluster_endpoint
- cluster_auth_base64 = var.cluster_auth_base64
-
- cluster_service_ipv4_cidr = var.cluster_service_ipv4_cidr
+ cluster_name = var.cluster_name
+ cluster_endpoint = var.cluster_endpoint
+ cluster_auth_base64 = var.cluster_auth_base64
+ cluster_ip_family = var.cluster_ip_family
+ cluster_service_cidr = try(coalesce(var.cluster_service_cidr, var.cluster_service_ipv4_cidr), "")
enable_bootstrap_user_data = var.enable_bootstrap_user_data
pre_bootstrap_user_data = var.pre_bootstrap_user_data
post_bootstrap_user_data = var.post_bootstrap_user_data
bootstrap_extra_args = var.bootstrap_extra_args
user_data_template_path = var.user_data_template_path
+
+ cloudinit_pre_nodeadm = var.cloudinit_pre_nodeadm
+ cloudinit_post_nodeadm = var.cloudinit_post_nodeadm
+}
+
+################################################################################
+# EFA Support
+################################################################################
+
+data "aws_ec2_instance_type" "this" {
+ count = var.create && var.enable_efa_support ? 1 : 0
+
+ instance_type = local.efa_instance_type
+}
+
+locals {
+ efa_instance_type = try(element(var.instance_types, 0), "")
+ num_network_cards = try(data.aws_ec2_instance_type.this[0].maximum_network_cards, 0)
+
+ efa_network_interfaces = [
+ for i in range(local.num_network_cards) : {
+ associate_public_ip_address = false
+ delete_on_termination = true
+ device_index = i == 0 ? 0 : 1
+ network_card_index = i
+ interface_type = "efa"
+ }
+ ]
+
+ network_interfaces = var.enable_efa_support ? local.efa_network_interfaces : var.network_interfaces
}
################################################################################
@@ -31,6 +62,8 @@ module "user_data" {
locals {
launch_template_name = coalesce(var.launch_template_name, var.name)
security_group_ids = compact(concat([var.cluster_primary_security_group_id], var.vpc_security_group_ids))
+
+ placement = var.create && (var.enable_efa_support || var.create_placement_group) ? { group_name = aws_placement_group.this[0].name } : var.placement
}
resource "aws_launch_template" "this" {
@@ -179,7 +212,7 @@ resource "aws_launch_template" "this" {
for_each = length(var.license_specifications) > 0 ? var.license_specifications : {}
content {
- license_configuration_arn = license_specifications.value.license_configuration_arn
+ license_configuration_arn = license_specification.value.license_configuration_arn
}
}
@@ -215,7 +248,8 @@ resource "aws_launch_template" "this" {
name_prefix = var.launch_template_use_name_prefix ? "${local.launch_template_name}-" : null
dynamic "network_interfaces" {
- for_each = var.network_interfaces
+ for_each = local.network_interfaces
+
content {
associate_carrier_ip_address = try(network_interfaces.value.associate_carrier_ip_address, null)
associate_public_ip_address = try(network_interfaces.value.associate_public_ip_address, null)
@@ -243,14 +277,14 @@ resource "aws_launch_template" "this" {
}
dynamic "placement" {
- for_each = length(var.placement) > 0 ? [var.placement] : []
+ for_each = length(local.placement) > 0 ? [local.placement] : []
content {
affinity = try(placement.value.affinity, null)
- availability_zone = try(placement.value.availability_zone, null)
- group_name = try(placement.value.group_name, null)
- host_id = try(placement.value.host_id, null)
- host_resource_group_arn = try(placement.value.host_resource_group_arn, null)
+ availability_zone = lookup(placement.value, "availability_zone", null)
+ group_name = lookup(placement.value, "group_name", null)
+ host_id = lookup(placement.value, "host_id", null)
+ host_resource_group_arn = lookup(placement.value, "host_resource_group_arn", null)
partition_number = try(placement.value.partition_number, null)
spread_domain = try(placement.value.spread_domain, null)
tenancy = try(placement.value.tenancy, null)
@@ -280,9 +314,12 @@ resource "aws_launch_template" "this" {
update_default_version = var.update_launch_template_default_version
user_data = module.user_data.user_data
- vpc_security_group_ids = length(var.network_interfaces) > 0 ? [] : local.security_group_ids
+ vpc_security_group_ids = length(local.network_interfaces) > 0 ? [] : local.security_group_ids
- tags = var.tags
+ tags = merge(
+ var.tags,
+ var.launch_template_tags,
+ )
# Prevent premature access of policies by pods that
# require permissions on create/destroy that depend on nodes
@@ -295,6 +332,45 @@ resource "aws_launch_template" "this" {
}
}
+################################################################################
+# AMI SSM Parameter
+################################################################################
+
+locals {
+ # Just to ensure templating doesn't fail when values are not provided
+ ssm_cluster_version = var.cluster_version != null ? var.cluster_version : ""
+ ssm_ami_type = var.ami_type != null ? var.ami_type : ""
+
+ # Map the AMI type to the respective SSM param path
+ ssm_ami_type_to_ssm_param = {
+ AL2_x86_64 = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2/recommended/release_version"
+ AL2_x86_64_GPU = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2-gpu/recommended/release_version"
+ AL2_ARM_64 = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2-arm64/recommended/release_version"
+ CUSTOM = "NONE"
+ BOTTLEROCKET_ARM_64 = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}/arm64/latest/image_version"
+ BOTTLEROCKET_x86_64 = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}/x86_64/latest/image_version"
+ BOTTLEROCKET_ARM_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}-nvidia/arm64/latest/image_version"
+ BOTTLEROCKET_x86_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}-nvidia/x86_64/latest/image_version"
+ WINDOWS_CORE_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Full-EKS_Optimized-${local.ssm_cluster_version}"
+ WINDOWS_FULL_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-${local.ssm_cluster_version}"
+ WINDOWS_CORE_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-EKS_Optimized-${local.ssm_cluster_version}"
+ WINDOWS_FULL_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-${local.ssm_cluster_version}"
+ AL2023_x86_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/standard/recommended/release_version"
+ AL2023_ARM_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/arm64/standard/recommended/release_version"
+ }
+
+ # The Windows SSM params currently do not have a release version, so we have to get the full output JSON blob and parse out the release version
+ windows_latest_ami_release_version = var.create && var.use_latest_ami_release_version && startswith(local.ssm_ami_type, "WINDOWS") ? nonsensitive(jsondecode(data.aws_ssm_parameter.ami[0].value)["release_version"]) : null
+ # Based on the steps above, try to get an AMI release version - if not, `null` is returned
+ latest_ami_release_version = startswith(local.ssm_ami_type, "WINDOWS") ? local.windows_latest_ami_release_version : try(nonsensitive(data.aws_ssm_parameter.ami[0].value), null)
+}
+
+data "aws_ssm_parameter" "ami" {
+ count = var.create && var.use_latest_ami_release_version ? 1 : 0
+
+ name = local.ssm_ami_type_to_ssm_param[var.ami_type]
+}
+
################################################################################
# Node Group
################################################################################
@@ -311,7 +387,7 @@ resource "aws_eks_node_group" "this" {
# Required
cluster_name = var.cluster_name
node_role_arn = var.create_iam_role ? aws_iam_role.this[0].arn : var.iam_role_arn
- subnet_ids = var.subnet_ids
+ subnet_ids = var.enable_efa_support ? data.aws_subnets.efa[0].ids : var.subnet_ids
scaling_config {
min_size = var.min_size
@@ -325,7 +401,7 @@ resource "aws_eks_node_group" "this" {
# https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-custom-ami
ami_type = var.ami_id != "" ? null : var.ami_type
- release_version = var.ami_id != "" ? null : var.ami_release_version
+ release_version = var.ami_id != "" ? null : var.use_latest_ami_release_version ? local.latest_ami_release_version : var.ami_release_version
version = var.ami_id != "" ? null : var.cluster_version
capacity_type = var.capacity_type
@@ -395,13 +471,21 @@ resource "aws_eks_node_group" "this" {
################################################################################
locals {
- iam_role_name = coalesce(var.iam_role_name, var.name)
+ create_iam_role = var.create && var.create_iam_role
+
+ iam_role_name = coalesce(var.iam_role_name, "${var.name}-eks-node-group")
iam_role_policy_prefix = "arn:${data.aws_partition.current.partition}:iam::aws:policy"
- cni_policy = var.cluster_ip_family == "ipv6" ? "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy" : "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+
+ ipv4_cni_policy = { for k, v in {
+ AmazonEKS_CNI_Policy = "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+ } : k => v if var.iam_role_attach_cni_policy && var.cluster_ip_family == "ipv4" }
+ ipv6_cni_policy = { for k, v in {
+ AmazonEKS_CNI_IPv6_Policy = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy"
+ } : k => v if var.iam_role_attach_cni_policy && var.cluster_ip_family == "ipv6" }
}
data "aws_iam_policy_document" "assume_role_policy" {
- count = var.create && var.create_iam_role ? 1 : 0
+ count = local.create_iam_role ? 1 : 0
statement {
sid = "EKSNodeAssumeRole"
@@ -409,13 +493,13 @@ data "aws_iam_policy_document" "assume_role_policy" {
principals {
type = "Service"
- identifiers = ["ec2.${data.aws_partition.current.dns_suffix}"]
+ identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "this" {
- count = var.create && var.create_iam_role ? 1 : 0
+ count = local.create_iam_role ? 1 : 0
name = var.iam_role_use_name_prefix ? null : local.iam_role_name
name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
@@ -431,23 +515,76 @@ resource "aws_iam_role" "this" {
# Policies attached ref https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group
resource "aws_iam_role_policy_attachment" "this" {
- for_each = { for k, v in toset(compact([
- "${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy",
- "${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly",
- var.iam_role_attach_cni_policy ? local.cni_policy : "",
- ])) : k => v if var.create && var.create_iam_role }
+ for_each = { for k, v in merge(
+ {
+ AmazonEKSWorkerNodePolicy = "${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy"
+ AmazonEC2ContainerRegistryReadOnly = "${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly"
+ },
+ local.ipv4_cni_policy,
+ local.ipv6_cni_policy
+ ) : k => v if local.create_iam_role }
policy_arn = each.value
role = aws_iam_role.this[0].name
}
resource "aws_iam_role_policy_attachment" "additional" {
- for_each = { for k, v in var.iam_role_additional_policies : k => v if var.create && var.create_iam_role }
+ for_each = { for k, v in var.iam_role_additional_policies : k => v if local.create_iam_role }
policy_arn = each.value
role = aws_iam_role.this[0].name
}
+################################################################################
+# Placement Group
+################################################################################
+
+resource "aws_placement_group" "this" {
+ count = var.create && (var.enable_efa_support || var.create_placement_group) ? 1 : 0
+
+ name = "${var.cluster_name}-${var.name}"
+ strategy = var.placement_group_strategy
+
+ tags = var.tags
+}
+
+################################################################################
+# Instance AZ Lookup
+
+# Instances usually used in placement groups w/ EFA are only available in
+# select availability zones. These data sources will cross reference the availability
+# zones supported by the instance type with the subnets provided to ensure only
+# AZs/subnets that are supported are used.
+################################################################################
+
+# Find the availability zones supported by the instance type
+data "aws_ec2_instance_type_offerings" "this" {
+ count = var.create && var.enable_efa_support ? 1 : 0
+
+ filter {
+ name = "instance-type"
+ values = [local.efa_instance_type]
+ }
+
+ location_type = "availability-zone-id"
+}
+
+# Reverse the lookup to find one of the subnets provided based on the availability
+# availability zone ID of the queried instance type (supported)
+data "aws_subnets" "efa" {
+ count = var.create && var.enable_efa_support ? 1 : 0
+
+ filter {
+ name = "subnet-id"
+ values = var.subnet_ids
+ }
+
+ filter {
+ name = "availability-zone-id"
+ values = data.aws_ec2_instance_type_offerings.this[0].locations
+ }
+}
+
################################################################################
# Autoscaling Group Schedule
################################################################################
diff --git a/modules/eks-managed-node-group/migrations.tf b/modules/eks-managed-node-group/migrations.tf
new file mode 100644
index 0000000000..5d51a7208a
--- /dev/null
+++ b/modules/eks-managed-node-group/migrations.tf
@@ -0,0 +1,20 @@
+################################################################################
+# Migrations: v20.7 -> v20.8
+################################################################################
+
+# Node IAM role policy attachment
+# Commercial partition only - `moved` does now allow multiple moves to same target
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"]
+ to = aws_iam_role_policy_attachment.this["AmazonEKSWorkerNodePolicy"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"]
+ to = aws_iam_role_policy_attachment.this["AmazonEC2ContainerRegistryReadOnly"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"]
+ to = aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"]
+}
diff --git a/modules/eks-managed-node-group/outputs.tf b/modules/eks-managed-node-group/outputs.tf
index 012cd46f4e..8cab6e2c13 100644
--- a/modules/eks-managed-node-group/outputs.tf
+++ b/modules/eks-managed-node-group/outputs.tf
@@ -88,3 +88,12 @@ output "iam_role_unique_id" {
description = "Stable and unique string identifying the IAM role"
value = try(aws_iam_role.this[0].unique_id, null)
}
+
+################################################################################
+# Additional
+################################################################################
+
+output "platform" {
+ description = "[DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows`"
+ value = module.user_data.platform
+}
diff --git a/modules/eks-managed-node-group/variables.tf b/modules/eks-managed-node-group/variables.tf
index 197cd28c95..344b346ad1 100644
--- a/modules/eks-managed-node-group/variables.tf
+++ b/modules/eks-managed-node-group/variables.tf
@@ -11,7 +11,7 @@ variable "tags" {
}
variable "platform" {
- description = "Identifies if the OS platform is `bottlerocket` or `linux` based; `windows` is not supported"
+ description = "[DEPRECATED - use `ami_type` instead. Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows`"
type = string
default = "linux"
}
@@ -44,26 +44,33 @@ variable "cluster_auth_base64" {
default = ""
}
+variable "cluster_service_cidr" {
+ description = "The CIDR block (IPv4 or IPv6) used by the cluster to assign Kubernetes service IP addresses. This is derived from the cluster itself"
+ type = string
+ default = ""
+}
+
+# TODO - remove at next breaking change
variable "cluster_service_ipv4_cidr" {
- description = "The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks"
+ description = "[Deprecated] The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks"
type = string
default = null
}
variable "pre_bootstrap_user_data" {
- description = "User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket`"
+ description = "User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*`"
type = string
default = ""
}
variable "post_bootstrap_user_data" {
- description = "User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket`"
+ description = "User data that is appended to the user data script after of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*`"
type = string
default = ""
}
variable "bootstrap_extra_args" {
- description = "Additional arguments passed to the bootstrap script. When `platform` = `bottlerocket`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data"
+ description = "Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data"
type = string
default = ""
}
@@ -74,6 +81,28 @@ variable "user_data_template_path" {
default = ""
}
+variable "cloudinit_pre_nodeadm" {
+ description = "Array of cloud-init document parts that are created before the nodeadm document part"
+ type = list(object({
+ content = string
+ content_type = optional(string)
+ filename = optional(string)
+ merge_type = optional(string)
+ }))
+ default = []
+}
+
+variable "cloudinit_post_nodeadm" {
+ description = "Array of cloud-init document parts that are created after the nodeadm document part"
+ type = list(object({
+ content = string
+ content_type = optional(string)
+ filename = optional(string)
+ merge_type = optional(string)
+ }))
+ default = []
+}
+
################################################################################
# Launch template
################################################################################
@@ -250,6 +279,12 @@ variable "enable_monitoring" {
default = true
}
+variable "enable_efa_support" {
+ description = "Determines whether to enable Elastic Fabric Adapter (EFA) support"
+ type = bool
+ default = false
+}
+
variable "network_interfaces" {
description = "Customize network interfaces to be attached at instance boot time"
type = list(any)
@@ -262,6 +297,18 @@ variable "placement" {
default = {}
}
+variable "create_placement_group" {
+ description = "Determines whether a placement group is created & used by the nodegroup"
+ type = bool
+ default = false
+}
+
+variable "placement_group_strategy" {
+ description = "The placement group strategy"
+ type = string
+ default = "cluster"
+}
+
variable "private_dns_name_options" {
description = "The options for the instance hostname. The default values are inherited from the subnet"
type = map(string)
@@ -321,17 +368,23 @@ variable "use_name_prefix" {
}
variable "ami_type" {
- description = "Type of Amazon Machine Image (AMI) associated with the EKS Node Group. Valid values are `AL2_x86_64`, `AL2_x86_64_GPU`, `AL2_ARM_64`, `CUSTOM`, `BOTTLEROCKET_ARM_64`, `BOTTLEROCKET_x86_64`"
+ description = "Type of Amazon Machine Image (AMI) associated with the EKS Node Group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values"
type = string
default = null
}
variable "ami_release_version" {
- description = "AMI version of the EKS Node Group. Defaults to latest version for Kubernetes version"
+ description = "The AMI version. Defaults to latest AMI release version for the given Kubernetes version and AMI type"
type = string
default = null
}
+variable "use_latest_ami_release_version" {
+ description = "Determines whether to use the latest AMI release version for the given `ami_type` (except for `CUSTOM`). Note: `ami_type` and `cluster_version` must be supplied in order to enable this feature"
+ type = bool
+ default = false
+}
+
variable "capacity_type" {
description = "Type of capacity associated with the EKS Node Group. Valid values: `ON_DEMAND`, `SPOT`"
type = string
@@ -413,7 +466,7 @@ variable "create_iam_role" {
variable "cluster_ip_family" {
description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
type = string
- default = null
+ default = "ipv4"
}
variable "iam_role_arn" {
diff --git a/modules/eks-managed-node-group/versions.tf b/modules/eks-managed-node-group/versions.tf
index 55eff62b09..6f83215f50 100644
--- a/modules/eks-managed-node-group/versions.tf
+++ b/modules/eks-managed-node-group/versions.tf
@@ -1,10 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
+ version = ">= 5.40"
}
}
}
diff --git a/modules/fargate-profile/README.md b/modules/fargate-profile/README.md
index cc0bab2a5b..072c2f2e33 100644
--- a/modules/fargate-profile/README.md
+++ b/modules/fargate-profile/README.md
@@ -28,14 +28,14 @@ module "fargate_profile" {
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.47 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.47 |
+| [aws](#provider\_aws) | >= 5.40 |
## Modules
@@ -52,12 +52,13 @@ No modules.
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws_iam_policy_document.assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
-| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `null` | no |
+| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
| [cluster\_name](#input\_cluster\_name) | Name of the EKS cluster | `string` | `null` | no |
| [create](#input\_create) | Determines whether to create Fargate profile or not | `bool` | `true` | no |
| [create\_iam\_role](#input\_create\_iam\_role) | Determines whether an IAM role is created or to use an existing IAM role | `bool` | `true` | no |
diff --git a/modules/fargate-profile/main.tf b/modules/fargate-profile/main.tf
index de9dd2d754..1e2cf60024 100644
--- a/modules/fargate-profile/main.tf
+++ b/modules/fargate-profile/main.tf
@@ -1,10 +1,19 @@
data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}
+data "aws_region" "current" {}
locals {
+ create_iam_role = var.create && var.create_iam_role
+
iam_role_name = coalesce(var.iam_role_name, var.name, "fargate-profile")
iam_role_policy_prefix = "arn:${data.aws_partition.current.partition}:iam::aws:policy"
- cni_policy = var.cluster_ip_family == "ipv6" ? "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy" : "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+
+ ipv4_cni_policy = { for k, v in {
+ AmazonEKS_CNI_Policy = "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+ } : k => v if var.iam_role_attach_cni_policy && var.cluster_ip_family == "ipv4" }
+ ipv6_cni_policy = { for k, v in {
+ AmazonEKS_CNI_IPv6_Policy = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy"
+ } : k => v if var.iam_role_attach_cni_policy && var.cluster_ip_family == "ipv6" }
}
################################################################################
@@ -12,7 +21,7 @@ locals {
################################################################################
data "aws_iam_policy_document" "assume_role_policy" {
- count = var.create && var.create_iam_role ? 1 : 0
+ count = local.create_iam_role ? 1 : 0
statement {
effect = "Allow"
@@ -22,11 +31,20 @@ data "aws_iam_policy_document" "assume_role_policy" {
type = "Service"
identifiers = ["eks-fargate-pods.amazonaws.com"]
}
+
+ condition {
+ test = "ArnLike"
+ variable = "aws:SourceArn"
+
+ values = [
+ "arn:${data.aws_partition.current.partition}:eks:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:fargateprofile/${var.cluster_name}/*",
+ ]
+ }
}
}
resource "aws_iam_role" "this" {
- count = var.create && var.create_iam_role ? 1 : 0
+ count = local.create_iam_role ? 1 : 0
name = var.iam_role_use_name_prefix ? null : local.iam_role_name
name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
@@ -41,17 +59,20 @@ resource "aws_iam_role" "this" {
}
resource "aws_iam_role_policy_attachment" "this" {
- for_each = { for k, v in toset(compact([
- "${local.iam_role_policy_prefix}/AmazonEKSFargatePodExecutionRolePolicy",
- var.iam_role_attach_cni_policy ? local.cni_policy : "",
- ])) : k => v if var.create && var.create_iam_role }
+ for_each = { for k, v in merge(
+ {
+ AmazonEKSFargatePodExecutionRolePolicy = "${local.iam_role_policy_prefix}/AmazonEKSFargatePodExecutionRolePolicy"
+ },
+ local.ipv4_cni_policy,
+ local.ipv6_cni_policy
+ ) : k => v if local.create_iam_role }
policy_arn = each.value
role = aws_iam_role.this[0].name
}
resource "aws_iam_role_policy_attachment" "additional" {
- for_each = { for k, v in var.iam_role_additional_policies : k => v if var.create && var.create_iam_role }
+ for_each = { for k, v in var.iam_role_additional_policies : k => v if local.create_iam_role }
policy_arn = each.value
role = aws_iam_role.this[0].name
diff --git a/modules/fargate-profile/migrations.tf b/modules/fargate-profile/migrations.tf
new file mode 100644
index 0000000000..02494f6893
--- /dev/null
+++ b/modules/fargate-profile/migrations.tf
@@ -0,0 +1,15 @@
+################################################################################
+# Migrations: v20.8 -> v20.9
+################################################################################
+
+# Node IAM role policy attachment
+# Commercial partition only - `moved` does now allow multiple moves to same target
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy"]
+ to = aws_iam_role_policy_attachment.this["AmazonEKSFargatePodExecutionRolePolicy"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"]
+ to = aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"]
+}
diff --git a/modules/fargate-profile/variables.tf b/modules/fargate-profile/variables.tf
index e22279dc6b..75816b0af8 100644
--- a/modules/fargate-profile/variables.tf
+++ b/modules/fargate-profile/variables.tf
@@ -23,7 +23,7 @@ variable "create_iam_role" {
variable "cluster_ip_family" {
description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
type = string
- default = null
+ default = "ipv4"
}
variable "iam_role_arn" {
diff --git a/modules/fargate-profile/versions.tf b/modules/fargate-profile/versions.tf
index 55eff62b09..6f83215f50 100644
--- a/modules/fargate-profile/versions.tf
+++ b/modules/fargate-profile/versions.tf
@@ -1,10 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
+ version = ">= 5.40"
}
}
}
diff --git a/modules/karpenter/README.md b/modules/karpenter/README.md
index 8e9b6dce99..ec819ed256 100644
--- a/modules/karpenter/README.md
+++ b/modules/karpenter/README.md
@@ -7,30 +7,16 @@ Configuration in this directory creates the AWS resources required by Karpenter
### All Resources (Default)
In the following example, the Karpenter module will create:
-- An IAM role for service accounts (IRSA) with a narrowly scoped IAM policy for the Karpenter controller to utilize
-- An IAM role and instance profile for the nodes created by Karpenter to utilize
- - Note: This IAM role ARN will need to be added to the `aws-auth` configmap for nodes to join the cluster successfully
-- An SQS queue and Eventbridge event rules for Karpenter to utilize for spot termination handling, capacity rebalancing, etc.
-
-This setup is great for running Karpenter on EKS Fargate:
+- An IAM role for use with Pod Identity and a scoped IAM policy for the Karpenter controller
+- A Pod Identity association to grant Karpenter controller access provided by the IAM Role
+- A Node IAM role that Karpenter will use to create an Instance Profile for the nodes to receive IAM permissions
+- An access entry for the Node IAM role to allow nodes to join the cluster
+- SQS queue and EventBridge event rules for Karpenter to utilize for spot termination handling, capacity re-balancing, etc.
```hcl
module "eks" {
- source = "terraform-aws-modules/eks"
+ source = "terraform-aws-modules/eks/aws"
- # Shown just for connection between cluster and Karpenter sub-module below
- manage_aws_auth_configmap = true
- aws_auth_roles = [
- # We need to add in the Karpenter node IAM role for nodes launched by Karpenter
- {
- rolearn = module.karpenter.role_arn
- username = "system:node:{{EC2PrivateDNSName}}"
- groups = [
- "system:bootstrappers",
- "system:nodes",
- ]
- },
- ]
...
}
@@ -39,8 +25,10 @@ module "karpenter" {
cluster_name = module.eks.cluster_name
- irsa_oidc_provider_arn = module.eks.oidc_provider_arn
- irsa_namespace_service_accounts = ["karpenter:karpenter"]
+ # Attach additional IAM policies to the Karpenter node IAM role
+ node_iam_role_additional_policies = {
+ AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
+ }
tags = {
Environment = "dev"
@@ -49,15 +37,13 @@ module "karpenter" {
}
```
-### External Node IAM Role (Default)
+### Re-Use Existing Node IAM Role
In the following example, the Karpenter module will create:
-- An IAM role for service accounts (IRSA) with a narrowly scoped IAM policy for the Karpenter controller to utilize
-- An IAM instance profile for the nodes created by Karpenter to utilize
- - Note: This setup will utilize the existing IAM role created by the EKS Managed Node group which means the role is already populated in the `aws-auth` configmap and no further updates are required.
-- An SQS queue and Eventbridge event rules for Karpenter to utilize for spot termination handling, capacity rebalancing, etc.
+- An IAM role for use with Pod Identity and a scoped IAM policy for the Karpenter controller
+- SQS queue and EventBridge event rules for Karpenter to utilize for spot termination handling, capacity re-balancing, etc.
-In this scenario, Karpenter would run atop the EKS Managed Node group and scale out nodes as needed from there:
+In this scenario, Karpenter will re-use an existing Node IAM role from the EKS managed nodegroup which already has the necessary access entry permissions:
```hcl
module "eks" {
@@ -81,11 +67,11 @@ module "karpenter" {
cluster_name = module.eks.cluster_name
- irsa_oidc_provider_arn = module.eks.oidc_provider_arn
- irsa_namespace_service_accounts = ["karpenter:karpenter"]
+ create_node_iam_role = false
+ node_iam_role_arn = module.eks.eks_managed_node_groups["initial"].iam_role_arn
- create_iam_role = false
- iam_role_arn = module.eks.eks_managed_node_groups["initial"].iam_role_arn
+ # Since the nodegroup role will already have an access entry
+ create_access_entry = false
tags = {
Environment = "dev"
@@ -99,14 +85,14 @@ module "karpenter" {
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.47 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.47 |
+| [aws](#provider\_aws) | >= 5.40 |
## Modules
@@ -118,65 +104,75 @@ No modules.
|------|------|
| [aws_cloudwatch_event_rule.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule) | resource |
| [aws_cloudwatch_event_target.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_target) | resource |
+| [aws_eks_access_entry.node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry) | resource |
+| [aws_eks_pod_identity_association.karpenter](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_pod_identity_association) | resource |
| [aws_iam_instance_profile.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource |
-| [aws_iam_policy.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
-| [aws_iam_role.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
-| [aws_iam_role.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
-| [aws_iam_role_policy_attachment.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
-| [aws_iam_role_policy_attachment.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
-| [aws_iam_role_policy_attachment.irsa_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
-| [aws_iam_role_policy_attachment.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_policy.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
+| [aws_iam_role.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
+| [aws_iam_role.node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
+| [aws_iam_role_policy_attachment.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.controller_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.node](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.node_additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_sqs_queue.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource |
| [aws_sqs_queue_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy) | resource |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
-| [aws_iam_policy_document.assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
-| [aws_iam_policy_document.irsa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
-| [aws_iam_policy_document.irsa_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.controller](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.controller_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.node_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.queue](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
-| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `null` | no |
+| [access\_entry\_type](#input\_access\_entry\_type) | Type of the access entry. `EC2_LINUX`, `FARGATE_LINUX`, or `EC2_WINDOWS`; defaults to `EC2_LINUX` | `string` | `"EC2_LINUX"` | no |
+| [ami\_id\_ssm\_parameter\_arns](#input\_ami\_id\_ssm\_parameter\_arns) | List of SSM Parameter ARNs that Karpenter controller is allowed read access (for retrieving AMI IDs) | `list(string)` | `[]` | no |
+| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`. Note: If `ipv6` is specified, the `AmazonEKS_CNI_IPv6_Policy` must exist in the account. This policy is created by the EKS module with `create_cni_ipv6_iam_policy = true` | `string` | `"ipv4"` | no |
| [cluster\_name](#input\_cluster\_name) | The name of the EKS cluster | `string` | `""` | no |
-| [create](#input\_create) | Determines whether to create EKS managed node group or not | `bool` | `true` | no |
-| [create\_iam\_role](#input\_create\_iam\_role) | Determines whether an IAM role is created or to use an existing IAM role | `bool` | `true` | no |
-| [create\_instance\_profile](#input\_create\_instance\_profile) | Whether to create an IAM instance profile | `bool` | `true` | no |
-| [create\_irsa](#input\_create\_irsa) | Determines whether an IAM role for service accounts is created | `bool` | `true` | no |
+| [create](#input\_create) | Controls if resources should be created (affects nearly all resources) | `bool` | `true` | no |
+| [create\_access\_entry](#input\_create\_access\_entry) | Determines whether an access entry is created for the IAM role used by the node IAM role | `bool` | `true` | no |
+| [create\_iam\_role](#input\_create\_iam\_role) | Determines whether an IAM role is created | `bool` | `true` | no |
+| [create\_instance\_profile](#input\_create\_instance\_profile) | Whether to create an IAM instance profile | `bool` | `false` | no |
+| [create\_node\_iam\_role](#input\_create\_node\_iam\_role) | Determines whether an IAM role is created or to use an existing IAM role | `bool` | `true` | no |
+| [create\_pod\_identity\_association](#input\_create\_pod\_identity\_association) | Determines whether to create pod identity association | `bool` | `false` | no |
+| [enable\_irsa](#input\_enable\_irsa) | Determines whether to enable support for IAM role for service accounts | `bool` | `false` | no |
+| [enable\_pod\_identity](#input\_enable\_pod\_identity) | Determines whether to enable support for EKS pod identity | `bool` | `true` | no |
| [enable\_spot\_termination](#input\_enable\_spot\_termination) | Determines whether to enable native spot termination handling | `bool` | `true` | no |
-| [iam\_role\_additional\_policies](#input\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `list(string)` | `[]` | no |
-| [iam\_role\_arn](#input\_iam\_role\_arn) | Existing IAM role ARN for the IAM instance profile. Required if `create_iam_role` is set to `false` | `string` | `null` | no |
-| [iam\_role\_attach\_cni\_policy](#input\_iam\_role\_attach\_cni\_policy) | Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster | `bool` | `true` | no |
-| [iam\_role\_description](#input\_iam\_role\_description) | Description of the role | `string` | `null` | no |
+| [iam\_policy\_description](#input\_iam\_policy\_description) | IAM policy description | `string` | `"Karpenter controller IAM policy"` | no |
+| [iam\_policy\_name](#input\_iam\_policy\_name) | Name of the IAM policy | `string` | `"KarpenterController"` | no |
+| [iam\_policy\_path](#input\_iam\_policy\_path) | Path of the IAM policy | `string` | `"/"` | no |
+| [iam\_policy\_use\_name\_prefix](#input\_iam\_policy\_use\_name\_prefix) | Determines whether the name of the IAM policy (`iam_policy_name`) is used as a prefix | `bool` | `true` | no |
+| [iam\_role\_description](#input\_iam\_role\_description) | IAM role description | `string` | `"Karpenter controller IAM role"` | no |
| [iam\_role\_max\_session\_duration](#input\_iam\_role\_max\_session\_duration) | Maximum API session duration in seconds between 3600 and 43200 | `number` | `null` | no |
-| [iam\_role\_name](#input\_iam\_role\_name) | Name to use on IAM role created | `string` | `null` | no |
-| [iam\_role\_path](#input\_iam\_role\_path) | IAM role path | `string` | `"/"` | no |
-| [iam\_role\_permissions\_boundary](#input\_iam\_role\_permissions\_boundary) | ARN of the policy that is used to set the permissions boundary for the IAM role | `string` | `null` | no |
-| [iam\_role\_tags](#input\_iam\_role\_tags) | A map of additional tags to add to the IAM role created | `map(string)` | `{}` | no |
-| [iam\_role\_use\_name\_prefix](#input\_iam\_role\_use\_name\_prefix) | Determines whether the IAM role name (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
+| [iam\_role\_name](#input\_iam\_role\_name) | Name of the IAM role | `string` | `"KarpenterController"` | no |
+| [iam\_role\_path](#input\_iam\_role\_path) | Path of the IAM role | `string` | `"/"` | no |
+| [iam\_role\_permissions\_boundary\_arn](#input\_iam\_role\_permissions\_boundary\_arn) | Permissions boundary ARN to use for the IAM role | `string` | `null` | no |
+| [iam\_role\_policies](#input\_iam\_role\_policies) | Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format | `map(string)` | `{}` | no |
+| [iam\_role\_tags](#input\_iam\_role\_tags) | A map of additional tags to add the the IAM role | `map(any)` | `{}` | no |
+| [iam\_role\_use\_name\_prefix](#input\_iam\_role\_use\_name\_prefix) | Determines whether the name of the IAM role (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| [irsa\_assume\_role\_condition\_test](#input\_irsa\_assume\_role\_condition\_test) | Name of the [IAM condition operator](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) to evaluate when assuming the role | `string` | `"StringEquals"` | no |
-| [irsa\_description](#input\_irsa\_description) | IAM role for service accounts description | `string` | `"Karpenter IAM role for service account"` | no |
-| [irsa\_max\_session\_duration](#input\_irsa\_max\_session\_duration) | Maximum API session duration in seconds between 3600 and 43200 | `number` | `null` | no |
-| [irsa\_name](#input\_irsa\_name) | Name of IAM role for service accounts | `string` | `null` | no |
| [irsa\_namespace\_service\_accounts](#input\_irsa\_namespace\_service\_accounts) | List of `namespace:serviceaccount`pairs to use in trust policy for IAM role for service accounts | `list(string)` | [
"karpenter:karpenter"
]
| no |
| [irsa\_oidc\_provider\_arn](#input\_irsa\_oidc\_provider\_arn) | OIDC provider arn used in trust policy for IAM role for service accounts | `string` | `""` | no |
-| [irsa\_path](#input\_irsa\_path) | Path of IAM role for service accounts | `string` | `"/"` | no |
-| [irsa\_permissions\_boundary\_arn](#input\_irsa\_permissions\_boundary\_arn) | Permissions boundary ARN to use for IAM role for service accounts | `string` | `null` | no |
-| [irsa\_policy\_name](#input\_irsa\_policy\_name) | Name of IAM policy for service accounts | `string` | `null` | no |
-| [irsa\_ssm\_parameter\_arns](#input\_irsa\_ssm\_parameter\_arns) | List of SSM Parameter ARNs that contain AMI IDs launched by Karpenter | `list(string)` | [
"arn:aws:ssm:*:*:parameter/aws/service/*"
]
| no |
-| [irsa\_subnet\_account\_id](#input\_irsa\_subnet\_account\_id) | Account ID of where the subnets Karpenter will utilize resides. Used when subnets are shared from another account | `string` | `""` | no |
-| [irsa\_tag\_key](#input\_irsa\_tag\_key) | Tag key (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner | `string` | `"karpenter.sh/discovery"` | no |
-| [irsa\_tag\_values](#input\_irsa\_tag\_values) | Tag values (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner. Defaults to cluster name when not set. | `list(string)` | `[]` | no |
-| [irsa\_tags](#input\_irsa\_tags) | A map of additional tags to add the the IAM role for service accounts | `map(any)` | `{}` | no |
-| [irsa\_use\_name\_prefix](#input\_irsa\_use\_name\_prefix) | Determines whether the IAM role for service accounts name (`irsa_name`) is used as a prefix | `bool` | `true` | no |
-| [policies](#input\_policies) | Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format | `map(string)` | `{}` | no |
+| [namespace](#input\_namespace) | Namespace to associate with the Karpenter Pod Identity | `string` | `"kube-system"` | no |
+| [node\_iam\_role\_additional\_policies](#input\_node\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `map(string)` | `{}` | no |
+| [node\_iam\_role\_arn](#input\_node\_iam\_role\_arn) | Existing IAM role ARN for the IAM instance profile. Required if `create_iam_role` is set to `false` | `string` | `null` | no |
+| [node\_iam\_role\_attach\_cni\_policy](#input\_node\_iam\_role\_attach\_cni\_policy) | Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster | `bool` | `true` | no |
+| [node\_iam\_role\_description](#input\_node\_iam\_role\_description) | Description of the role | `string` | `null` | no |
+| [node\_iam\_role\_max\_session\_duration](#input\_node\_iam\_role\_max\_session\_duration) | Maximum API session duration in seconds between 3600 and 43200 | `number` | `null` | no |
+| [node\_iam\_role\_name](#input\_node\_iam\_role\_name) | Name to use on IAM role created | `string` | `null` | no |
+| [node\_iam\_role\_path](#input\_node\_iam\_role\_path) | IAM role path | `string` | `"/"` | no |
+| [node\_iam\_role\_permissions\_boundary](#input\_node\_iam\_role\_permissions\_boundary) | ARN of the policy that is used to set the permissions boundary for the IAM role | `string` | `null` | no |
+| [node\_iam\_role\_tags](#input\_node\_iam\_role\_tags) | A map of additional tags to add to the IAM role created | `map(string)` | `{}` | no |
+| [node\_iam\_role\_use\_name\_prefix](#input\_node\_iam\_role\_use\_name\_prefix) | Determines whether the IAM role name (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| [queue\_kms\_data\_key\_reuse\_period\_seconds](#input\_queue\_kms\_data\_key\_reuse\_period\_seconds) | The length of time, in seconds, for which Amazon SQS can reuse a data key to encrypt or decrypt messages before calling AWS KMS again | `number` | `null` | no |
| [queue\_kms\_master\_key\_id](#input\_queue\_kms\_master\_key\_id) | The ID of an AWS-managed customer master key (CMK) for Amazon SQS or a custom CMK | `string` | `null` | no |
| [queue\_managed\_sse\_enabled](#input\_queue\_managed\_sse\_enabled) | Boolean to enable server-side encryption (SSE) of message content with SQS-owned encryption keys | `bool` | `true` | no |
| [queue\_name](#input\_queue\_name) | Name of the SQS queue | `string` | `null` | no |
| [rule\_name\_prefix](#input\_rule\_name\_prefix) | Prefix used for all event bridge rules | `string` | `"Karpenter"` | no |
+| [service\_account](#input\_service\_account) | Service account to associate with the Karpenter Pod Identity | `string` | `"karpenter"` | no |
| [tags](#input\_tags) | A map of tags to add to all resources | `map(string)` | `{}` | no |
## Outputs
@@ -184,17 +180,20 @@ No modules.
| Name | Description |
|------|-------------|
| [event\_rules](#output\_event\_rules) | Map of the event rules created and their attributes |
+| [iam\_role\_arn](#output\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the controller IAM role |
+| [iam\_role\_name](#output\_iam\_role\_name) | The name of the controller IAM role |
+| [iam\_role\_unique\_id](#output\_iam\_role\_unique\_id) | Stable and unique string identifying the controller IAM role |
| [instance\_profile\_arn](#output\_instance\_profile\_arn) | ARN assigned by AWS to the instance profile |
| [instance\_profile\_id](#output\_instance\_profile\_id) | Instance profile's ID |
| [instance\_profile\_name](#output\_instance\_profile\_name) | Name of the instance profile |
| [instance\_profile\_unique](#output\_instance\_profile\_unique) | Stable and unique string identifying the IAM instance profile |
-| [irsa\_arn](#output\_irsa\_arn) | The Amazon Resource Name (ARN) specifying the IAM role for service accounts |
-| [irsa\_name](#output\_irsa\_name) | The name of the IAM role for service accounts |
-| [irsa\_unique\_id](#output\_irsa\_unique\_id) | Stable and unique string identifying the IAM role for service accounts |
+| [namespace](#output\_namespace) | Namespace associated with the Karpenter Pod Identity |
+| [node\_access\_entry\_arn](#output\_node\_access\_entry\_arn) | Amazon Resource Name (ARN) of the node Access Entry |
+| [node\_iam\_role\_arn](#output\_node\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the node IAM role |
+| [node\_iam\_role\_name](#output\_node\_iam\_role\_name) | The name of the node IAM role |
+| [node\_iam\_role\_unique\_id](#output\_node\_iam\_role\_unique\_id) | Stable and unique string identifying the node IAM role |
| [queue\_arn](#output\_queue\_arn) | The ARN of the SQS queue |
| [queue\_name](#output\_queue\_name) | The name of the created Amazon SQS queue |
| [queue\_url](#output\_queue\_url) | The URL for the created Amazon SQS queue |
-| [role\_arn](#output\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
-| [role\_name](#output\_role\_name) | The name of the IAM role |
-| [role\_unique\_id](#output\_role\_unique\_id) | Stable and unique string identifying the IAM role |
+| [service\_account](#output\_service\_account) | Service Account associated with the Karpenter Pod Identity |
diff --git a/modules/karpenter/main.tf b/modules/karpenter/main.tf
index 19399ce99d..4b31eb5da7 100644
--- a/modules/karpenter/main.tf
+++ b/modules/karpenter/main.tf
@@ -1,192 +1,431 @@
+data "aws_region" "current" {}
data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}
locals {
account_id = data.aws_caller_identity.current.account_id
partition = data.aws_partition.current.partition
- dns_suffix = data.aws_partition.current.dns_suffix
+ region = data.aws_region.current.name
}
################################################################################
-# IAM Role for Service Account (IRSA)
-# This is used by the Karpenter controller
+# Karpenter controller IAM Role
################################################################################
locals {
- create_irsa = var.create && var.create_irsa
- irsa_name = coalesce(var.irsa_name, "KarpenterIRSA-${var.cluster_name}")
- irsa_policy_name = coalesce(var.irsa_policy_name, local.irsa_name)
-
+ create_iam_role = var.create && var.create_iam_role
irsa_oidc_provider_url = replace(var.irsa_oidc_provider_arn, "/^(.*provider/)/", "")
}
-data "aws_iam_policy_document" "irsa_assume_role" {
- count = local.create_irsa ? 1 : 0
+data "aws_iam_policy_document" "controller_assume_role" {
+ count = local.create_iam_role ? 1 : 0
- statement {
- effect = "Allow"
- actions = ["sts:AssumeRoleWithWebIdentity"]
+ # Pod Identity
+ dynamic "statement" {
+ for_each = var.enable_pod_identity ? [1] : []
- principals {
- type = "Federated"
- identifiers = [var.irsa_oidc_provider_arn]
- }
+ content {
+ actions = [
+ "sts:AssumeRole",
+ "sts:TagSession",
+ ]
- condition {
- test = var.irsa_assume_role_condition_test
- variable = "${local.irsa_oidc_provider_url}:sub"
- values = [for sa in var.irsa_namespace_service_accounts : "system:serviceaccount:${sa}"]
+ principals {
+ type = "Service"
+ identifiers = ["pods.eks.amazonaws.com"]
+ }
}
+ }
- # https://aws.amazon.com/premiumsupport/knowledge-center/eks-troubleshoot-oidc-and-irsa/?nc1=h_ls
- condition {
- test = var.irsa_assume_role_condition_test
- variable = "${local.irsa_oidc_provider_url}:aud"
- values = ["sts.amazonaws.com"]
+ # IAM Roles for Service Accounts (IRSA)
+ dynamic "statement" {
+ for_each = var.enable_irsa ? [1] : []
+
+ content {
+ actions = ["sts:AssumeRoleWithWebIdentity"]
+
+ principals {
+ type = "Federated"
+ identifiers = [var.irsa_oidc_provider_arn]
+ }
+
+ condition {
+ test = var.irsa_assume_role_condition_test
+ variable = "${local.irsa_oidc_provider_url}:sub"
+ values = [for sa in var.irsa_namespace_service_accounts : "system:serviceaccount:${sa}"]
+ }
+
+ # https://aws.amazon.com/premiumsupport/knowledge-center/eks-troubleshoot-oidc-and-irsa/?nc1=h_ls
+ condition {
+ test = var.irsa_assume_role_condition_test
+ variable = "${local.irsa_oidc_provider_url}:aud"
+ values = ["sts.amazonaws.com"]
+ }
}
}
}
-resource "aws_iam_role" "irsa" {
- count = local.create_irsa ? 1 : 0
+resource "aws_iam_role" "controller" {
+ count = local.create_iam_role ? 1 : 0
- name = var.irsa_use_name_prefix ? null : local.irsa_name
- name_prefix = var.irsa_use_name_prefix ? "${local.irsa_name}-" : null
- path = var.irsa_path
- description = var.irsa_description
+ name = var.iam_role_use_name_prefix ? null : var.iam_role_name
+ name_prefix = var.iam_role_use_name_prefix ? "${var.iam_role_name}-" : null
+ path = var.iam_role_path
+ description = var.iam_role_description
- assume_role_policy = data.aws_iam_policy_document.irsa_assume_role[0].json
- max_session_duration = var.irsa_max_session_duration
- permissions_boundary = var.irsa_permissions_boundary_arn
+ assume_role_policy = data.aws_iam_policy_document.controller_assume_role[0].json
+ max_session_duration = var.iam_role_max_session_duration
+ permissions_boundary = var.iam_role_permissions_boundary_arn
force_detach_policies = true
- tags = merge(var.tags, var.irsa_tags)
+ tags = merge(var.tags, var.iam_role_tags)
}
-locals {
- irsa_tag_values = coalescelist(var.irsa_tag_values, [var.cluster_name])
-}
+data "aws_iam_policy_document" "controller" {
+ count = local.create_iam_role ? 1 : 0
-data "aws_iam_policy_document" "irsa" {
- count = local.create_irsa ? 1 : 0
+ statement {
+ sid = "AllowScopedEC2InstanceActions"
+ resources = [
+ "arn:${local.partition}:ec2:*::image/*",
+ "arn:${local.partition}:ec2:*::snapshot/*",
+ "arn:${local.partition}:ec2:*:*:spot-instances-request/*",
+ "arn:${local.partition}:ec2:*:*:security-group/*",
+ "arn:${local.partition}:ec2:*:*:subnet/*",
+ "arn:${local.partition}:ec2:*:*:launch-template/*",
+ ]
+
+ actions = [
+ "ec2:RunInstances",
+ "ec2:CreateFleet"
+ ]
+ }
statement {
+ sid = "AllowScopedEC2InstanceActionsWithTags"
+ resources = [
+ "arn:${local.partition}:ec2:*:*:fleet/*",
+ "arn:${local.partition}:ec2:*:*:instance/*",
+ "arn:${local.partition}:ec2:*:*:volume/*",
+ "arn:${local.partition}:ec2:*:*:network-interface/*",
+ "arn:${local.partition}:ec2:*:*:launch-template/*",
+ "arn:${local.partition}:ec2:*:*:spot-instances-request/*",
+ ]
actions = [
- "ec2:CreateLaunchTemplate",
+ "ec2:RunInstances",
"ec2:CreateFleet",
- "ec2:CreateTags",
- "ec2:DescribeLaunchTemplates",
- "ec2:DescribeImages",
- "ec2:DescribeInstances",
- "ec2:DescribeSecurityGroups",
- "ec2:DescribeSubnets",
- "ec2:DescribeInstanceTypes",
- "ec2:DescribeInstanceTypeOfferings",
- "ec2:DescribeAvailabilityZones",
- "ec2:DescribeSpotPriceHistory",
- "pricing:GetProducts",
+ "ec2:CreateLaunchTemplate"
]
- resources = ["*"]
+ condition {
+ test = "StringEquals"
+ variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:RequestTag/karpenter.sh/nodepool"
+ values = ["*"]
+ }
}
statement {
- actions = [
- "ec2:TerminateInstances",
- "ec2:DeleteLaunchTemplate",
+ sid = "AllowScopedResourceCreationTagging"
+ resources = [
+ "arn:${local.partition}:ec2:*:*:fleet/*",
+ "arn:${local.partition}:ec2:*:*:instance/*",
+ "arn:${local.partition}:ec2:*:*:volume/*",
+ "arn:${local.partition}:ec2:*:*:network-interface/*",
+ "arn:${local.partition}:ec2:*:*:launch-template/*",
+ "arn:${local.partition}:ec2:*:*:spot-instances-request/*",
]
+ actions = ["ec2:CreateTags"]
- resources = ["*"]
+ condition {
+ test = "StringEquals"
+ variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
condition {
test = "StringEquals"
- variable = "ec2:ResourceTag/${var.irsa_tag_key}"
- values = local.irsa_tag_values
+ variable = "ec2:CreateAction"
+ values = [
+ "RunInstances",
+ "CreateFleet",
+ "CreateLaunchTemplate",
+ ]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:RequestTag/karpenter.sh/nodepool"
+ values = ["*"]
}
}
statement {
- actions = ["ec2:RunInstances"]
- resources = [
- "arn:${local.partition}:ec2:*:${local.account_id}:launch-template/*",
- ]
+ sid = "AllowScopedResourceTagging"
+ resources = ["arn:${local.partition}:ec2:*:*:instance/*"]
+ actions = ["ec2:CreateTags"]
condition {
test = "StringEquals"
- variable = "ec2:ResourceTag/${var.irsa_tag_key}"
- values = local.irsa_tag_values
+ variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:ResourceTag/karpenter.sh/nodepool"
+ values = ["*"]
+ }
+
+ condition {
+ test = "ForAllValues:StringEquals"
+ variable = "aws:TagKeys"
+ values = [
+ "karpenter.sh/nodeclaim",
+ "Name",
+ ]
}
}
statement {
- actions = ["ec2:RunInstances"]
+ sid = "AllowScopedDeletion"
resources = [
- "arn:${local.partition}:ec2:*::image/*",
- "arn:${local.partition}:ec2:*::snapshot/*",
- "arn:${local.partition}:ec2:*:${local.account_id}:instance/*",
- "arn:${local.partition}:ec2:*:${local.account_id}:spot-instances-request/*",
- "arn:${local.partition}:ec2:*:${local.account_id}:security-group/*",
- "arn:${local.partition}:ec2:*:${local.account_id}:volume/*",
- "arn:${local.partition}:ec2:*:${local.account_id}:network-interface/*",
- "arn:${local.partition}:ec2:*:${coalesce(var.irsa_subnet_account_id, local.account_id)}:subnet/*",
+ "arn:${local.partition}:ec2:*:*:instance/*",
+ "arn:${local.partition}:ec2:*:*:launch-template/*"
]
+
+ actions = [
+ "ec2:TerminateInstances",
+ "ec2:DeleteLaunchTemplate"
+ ]
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:ResourceTag/karpenter.sh/nodepool"
+ values = ["*"]
+ }
}
statement {
- actions = ["ssm:GetParameter"]
- resources = var.irsa_ssm_parameter_arns
+ sid = "AllowRegionalReadActions"
+ resources = ["*"]
+ actions = [
+ "ec2:DescribeAvailabilityZones",
+ "ec2:DescribeImages",
+ "ec2:DescribeInstances",
+ "ec2:DescribeInstanceTypeOfferings",
+ "ec2:DescribeInstanceTypes",
+ "ec2:DescribeLaunchTemplates",
+ "ec2:DescribeSecurityGroups",
+ "ec2:DescribeSpotPriceHistory",
+ "ec2:DescribeSubnets"
+ ]
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:RequestedRegion"
+ values = [local.region]
+ }
}
statement {
- actions = ["eks:DescribeCluster"]
- resources = ["arn:${local.partition}:eks:*:${local.account_id}:cluster/${var.cluster_name}"]
+ sid = "AllowSSMReadActions"
+ resources = coalescelist(var.ami_id_ssm_parameter_arns, ["arn:${local.partition}:ssm:${local.region}::parameter/aws/service/*"])
+ actions = ["ssm:GetParameter"]
}
statement {
- actions = ["iam:PassRole"]
- resources = [var.create_iam_role ? aws_iam_role.this[0].arn : var.iam_role_arn]
+ sid = "AllowPricingReadActions"
+ resources = ["*"]
+ actions = ["pricing:GetProducts"]
}
dynamic "statement" {
for_each = local.enable_spot_termination ? [1] : []
content {
+ sid = "AllowInterruptionQueueActions"
+ resources = [try(aws_sqs_queue.this[0].arn, null)]
actions = [
"sqs:DeleteMessage",
- "sqs:GetQueueUrl",
"sqs:GetQueueAttributes",
- "sqs:ReceiveMessage",
+ "sqs:GetQueueUrl",
+ "sqs:ReceiveMessage"
]
- resources = [aws_sqs_queue.this[0].arn]
}
}
+
+ statement {
+ sid = "AllowPassingInstanceRole"
+ resources = var.create_node_iam_role ? [aws_iam_role.node[0].arn] : [var.node_iam_role_arn]
+ actions = ["iam:PassRole"]
+
+ condition {
+ test = "StringEquals"
+ variable = "iam:PassedToService"
+ values = ["ec2.amazonaws.com"]
+ }
+ }
+
+ statement {
+ sid = "AllowScopedInstanceProfileCreationActions"
+ resources = ["*"]
+ actions = ["iam:CreateInstanceProfile"]
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:RequestTag/topology.kubernetes.io/region"
+ values = [local.region]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass"
+ values = ["*"]
+ }
+ }
+
+ statement {
+ sid = "AllowScopedInstanceProfileTagActions"
+ resources = ["*"]
+ actions = ["iam:TagInstanceProfile"]
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:ResourceTag/topology.kubernetes.io/region"
+ values = [local.region]
+ }
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:RequestTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:ResourceTag/topology.kubernetes.io/region"
+ values = [local.region]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass"
+ values = ["*"]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass"
+ values = ["*"]
+ }
+ }
+
+ statement {
+ sid = "AllowScopedInstanceProfileActions"
+ resources = ["*"]
+ actions = [
+ "iam:AddRoleToInstanceProfile",
+ "iam:RemoveRoleFromInstanceProfile",
+ "iam:DeleteInstanceProfile"
+ ]
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:ResourceTag/kubernetes.io/cluster/${var.cluster_name}"
+ values = ["owned"]
+ }
+
+ condition {
+ test = "StringEquals"
+ variable = "aws:ResourceTag/topology.kubernetes.io/region"
+ values = [local.region]
+ }
+
+ condition {
+ test = "StringLike"
+ variable = "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass"
+ values = ["*"]
+ }
+ }
+
+ statement {
+ sid = "AllowInstanceProfileReadActions"
+ resources = ["*"]
+ actions = ["iam:GetInstanceProfile"]
+ }
+
+ statement {
+ sid = "AllowAPIServerEndpointDiscovery"
+ resources = ["arn:${local.partition}:eks:${local.region}:${local.account_id}:cluster/${var.cluster_name}"]
+ actions = ["eks:DescribeCluster"]
+ }
}
-resource "aws_iam_policy" "irsa" {
- count = local.create_irsa ? 1 : 0
+resource "aws_iam_policy" "controller" {
+ count = local.create_iam_role ? 1 : 0
- name_prefix = "${local.irsa_policy_name}-"
- path = var.irsa_path
- description = var.irsa_description
- policy = data.aws_iam_policy_document.irsa[0].json
+ name = var.iam_policy_use_name_prefix ? null : var.iam_policy_name
+ name_prefix = var.iam_policy_use_name_prefix ? "${var.iam_policy_name}-" : null
+ path = var.iam_policy_path
+ description = var.iam_policy_description
+ policy = data.aws_iam_policy_document.controller[0].json
tags = var.tags
}
-resource "aws_iam_role_policy_attachment" "irsa" {
- count = local.create_irsa ? 1 : 0
+resource "aws_iam_role_policy_attachment" "controller" {
+ count = local.create_iam_role ? 1 : 0
- role = aws_iam_role.irsa[0].name
- policy_arn = aws_iam_policy.irsa[0].arn
+ role = aws_iam_role.controller[0].name
+ policy_arn = aws_iam_policy.controller[0].arn
}
-resource "aws_iam_role_policy_attachment" "irsa_additional" {
- for_each = { for k, v in var.policies : k => v if local.create_irsa }
+resource "aws_iam_role_policy_attachment" "controller_additional" {
+ for_each = { for k, v in var.iam_role_policies : k => v if local.create_iam_role }
- role = aws_iam_role.irsa[0].name
+ role = aws_iam_role.controller[0].name
policy_arn = each.value
}
+################################################################################
+# Pod Identity Association
+################################################################################
+
+resource "aws_eks_pod_identity_association" "karpenter" {
+ count = local.create_iam_role && var.enable_pod_identity && var.create_pod_identity_association ? 1 : 0
+
+ cluster_name = var.cluster_name
+ namespace = var.namespace
+ service_account = var.service_account
+ role_arn = aws_iam_role.controller[0].arn
+
+ tags = var.tags
+}
+
################################################################################
# Node Termination Queue
################################################################################
@@ -220,11 +459,10 @@ data "aws_iam_policy_document" "queue" {
principals {
type = "Service"
identifiers = [
- "events.${local.dns_suffix}",
- "sqs.${local.dns_suffix}",
+ "events.amazonaws.com",
+ "sqs.amazonaws.com",
]
}
-
}
}
@@ -249,7 +487,7 @@ locals {
detail-type = ["AWS Health Event"]
}
}
- spot_interupt = {
+ spot_interrupt = {
name = "SpotInterrupt"
description = "Karpenter interrupt - EC2 spot instance interruption warning"
event_pattern = {
@@ -303,15 +541,21 @@ resource "aws_cloudwatch_event_target" "this" {
################################################################################
locals {
- create_iam_role = var.create && var.create_iam_role
+ create_node_iam_role = var.create && var.create_node_iam_role
+
+ node_iam_role_name = coalesce(var.node_iam_role_name, "Karpenter-${var.cluster_name}")
+ node_iam_role_policy_prefix = "arn:${local.partition}:iam::aws:policy"
- iam_role_name = coalesce(var.iam_role_name, "Karpenter-${var.cluster_name}")
- iam_role_policy_prefix = "arn:${local.partition}:iam::aws:policy"
- cni_policy = var.cluster_ip_family == "ipv6" ? "${local.iam_role_policy_prefix}/AmazonEKS_CNI_IPv6_Policy" : "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+ ipv4_cni_policy = { for k, v in {
+ AmazonEKS_CNI_Policy = "${local.node_iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+ } : k => v if var.node_iam_role_attach_cni_policy && var.cluster_ip_family == "ipv4" }
+ ipv6_cni_policy = { for k, v in {
+ AmazonEKS_CNI_IPv6_Policy = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy"
+ } : k => v if var.node_iam_role_attach_cni_policy && var.cluster_ip_family == "ipv6" }
}
-data "aws_iam_policy_document" "assume_role" {
- count = local.create_iam_role ? 1 : 0
+data "aws_iam_policy_document" "node_assume_role" {
+ count = local.create_node_iam_role ? 1 : 0
statement {
sid = "EKSNodeAssumeRole"
@@ -319,62 +563,207 @@ data "aws_iam_policy_document" "assume_role" {
principals {
type = "Service"
- identifiers = ["ec2.${local.dns_suffix}"]
+ identifiers = ["ec2.amazonaws.com"]
}
}
}
-resource "aws_iam_role" "this" {
- count = local.create_iam_role ? 1 : 0
+resource "aws_iam_role" "node" {
+ count = local.create_node_iam_role ? 1 : 0
- name = var.iam_role_use_name_prefix ? null : local.iam_role_name
- name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
- path = var.iam_role_path
- description = var.iam_role_description
+ name = var.node_iam_role_use_name_prefix ? null : local.node_iam_role_name
+ name_prefix = var.node_iam_role_use_name_prefix ? "${local.node_iam_role_name}-" : null
+ path = var.node_iam_role_path
+ description = var.node_iam_role_description
- assume_role_policy = data.aws_iam_policy_document.assume_role[0].json
- max_session_duration = var.iam_role_max_session_duration
- permissions_boundary = var.iam_role_permissions_boundary
+ assume_role_policy = data.aws_iam_policy_document.node_assume_role[0].json
+ max_session_duration = var.node_iam_role_max_session_duration
+ permissions_boundary = var.node_iam_role_permissions_boundary
force_detach_policies = true
- tags = merge(var.tags, var.iam_role_tags)
+ tags = merge(var.tags, var.node_iam_role_tags)
}
# Policies attached ref https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group
-resource "aws_iam_role_policy_attachment" "this" {
- for_each = { for k, v in toset(compact([
- "${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy",
- "${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly",
- var.iam_role_attach_cni_policy ? local.cni_policy : "",
- ])) : k => v if local.create_iam_role }
+resource "aws_iam_role_policy_attachment" "node" {
+ for_each = { for k, v in merge(
+ {
+ AmazonEKSWorkerNodePolicy = "${local.node_iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy"
+ AmazonEC2ContainerRegistryReadOnly = "${local.node_iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly"
+ },
+ local.ipv4_cni_policy,
+ local.ipv6_cni_policy
+ ) : k => v if local.create_node_iam_role }
policy_arn = each.value
- role = aws_iam_role.this[0].name
+ role = aws_iam_role.node[0].name
}
-resource "aws_iam_role_policy_attachment" "additional" {
- for_each = { for k, v in var.iam_role_additional_policies : k => v if local.create_iam_role }
+resource "aws_iam_role_policy_attachment" "node_additional" {
+ for_each = { for k, v in var.node_iam_role_additional_policies : k => v if local.create_node_iam_role }
policy_arn = each.value
- role = aws_iam_role.this[0].name
+ role = aws_iam_role.node[0].name
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+resource "aws_eks_access_entry" "node" {
+ count = var.create && var.create_access_entry ? 1 : 0
+
+ cluster_name = var.cluster_name
+ principal_arn = var.create_node_iam_role ? aws_iam_role.node[0].arn : var.node_iam_role_arn
+ type = var.access_entry_type
+
+ tags = var.tags
+
+ depends_on = [
+ # If we try to add this too quickly, it fails. So .... we wait
+ aws_sqs_queue_policy.this,
+ ]
}
################################################################################
# Node IAM Instance Profile
# This is used by the nodes launched by Karpenter
+# Starting with Karpenter 0.32 this is no longer required as Karpenter will
+# create the Instance Profile
################################################################################
locals {
- external_role_name = try(replace(var.iam_role_arn, "/^(.*role/)/", ""), null)
+ external_role_name = try(replace(var.node_iam_role_arn, "/^(.*role/)/", ""), null)
}
resource "aws_iam_instance_profile" "this" {
count = var.create && var.create_instance_profile ? 1 : 0
- name = var.iam_role_use_name_prefix ? null : local.iam_role_name
- name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
- path = var.iam_role_path
- role = var.create_iam_role ? aws_iam_role.this[0].name : local.external_role_name
+ name = var.node_iam_role_use_name_prefix ? null : local.node_iam_role_name
+ name_prefix = var.node_iam_role_use_name_prefix ? "${local.node_iam_role_name}-" : null
+ path = var.node_iam_role_path
+ role = var.create_node_iam_role ? aws_iam_role.node[0].name : local.external_role_name
+
+ tags = merge(var.tags, var.node_iam_role_tags)
+}
+
+################################################################################
+# create new Iam policy and attach it to role ffor old Karpenter
+#
+################################################################################
+locals {
+
+ irsa_name = coalesce(var.iam_role_name, "KarpenterIRSA-${var.cluster_name}")
+ irsa_policy_name = coalesce(var.iam_policy_name, local.irsa_name)
- tags = merge(var.tags, var.iam_role_tags)
}
+
+data "aws_iam_policy_document" "irsa1" {
+ count = var.enable_irsa ? 1 : 0
+
+ statement {
+ actions = [
+ "ec2:CreateLaunchTemplate",
+ "ec2:CreateFleet",
+ "ec2:CreateTags",
+ "ec2:DescribeLaunchTemplates",
+ "ec2:DescribeImages",
+ "ec2:DescribeInstances",
+ "ec2:DescribeSecurityGroups",
+ "ec2:DescribeSubnets",
+ "ec2:DescribeInstanceTypes",
+ "ec2:DescribeInstanceTypeOfferings",
+ "ec2:DescribeAvailabilityZones",
+ "ec2:DescribeSpotPriceHistory",
+ "pricing:GetProducts",
+ ]
+
+ resources = ["*"]
+ }
+
+ statement {
+ actions = [
+ "ec2:TerminateInstances",
+ "ec2:DeleteLaunchTemplate",
+ ]
+
+ resources = ["*"]
+
+ condition {
+ test = "StringEquals"
+ variable = "ec2:ResourceTag/karpenter.sh/discovery"
+ values = ["${var.cluster_name}"]
+ }
+ }
+
+ statement {
+ actions = ["ec2:RunInstances"]
+ resources = [
+ "arn:${local.partition}:ec2:*:${local.account_id}:launch-template/*",
+ ]
+
+ condition {
+ test = "StringEquals"
+ variable = "ec2:ResourceTag/karpenter.sh/discovery"
+ values = ["${var.cluster_name}"]
+ }
+ }
+
+ statement {
+ actions = ["ec2:RunInstances"]
+ resources = [
+ "arn:${local.partition}:ec2:*::image/*",
+ "arn:${local.partition}:ec2:*::snapshot/*",
+ "arn:${local.partition}:ec2:*:${local.account_id}:instance/*",
+ "arn:${local.partition}:ec2:*:${local.account_id}:spot-instances-request/*",
+ "arn:${local.partition}:ec2:*:${local.account_id}:security-group/*",
+ "arn:${local.partition}:ec2:*:${local.account_id}:volume/*",
+ "arn:${local.partition}:ec2:*:${local.account_id}:network-interface/*",
+ "arn:${local.partition}:ec2:*:${local.account_id}:subnet/*",
+ ]
+ }
+
+ statement {
+ actions = ["ssm:GetParameter"]
+ resources = ["arn:aws:ssm:*:*:parameter/aws/service/*"]
+ }
+
+ statement {
+ actions = ["eks:DescribeCluster"]
+ resources = ["arn:${local.partition}:eks:*:${local.account_id}:cluster/${var.cluster_name}"]
+ }
+
+ statement {
+ actions = ["iam:PassRole"]
+ resources = [var.enable_irsa ? aws_iam_role.node[0].arn : var.node_iam_role_arn]
+ }
+
+ dynamic "statement" {
+ for_each = local.enable_spot_termination ? [1] : []
+
+ content {
+ actions = [
+ "sqs:DeleteMessage",
+ "sqs:GetQueueUrl",
+ "sqs:GetQueueAttributes",
+ "sqs:ReceiveMessage",
+ ]
+ resources = [aws_sqs_queue.this[0].arn]
+ }
+ }
+}
+
+resource "aws_iam_policy" "irsa1" {
+ count = var.enable_irsa ? 1 : 0
+
+ name_prefix = "${local.irsa_policy_name}-"
+ policy = data.aws_iam_policy_document.irsa1[0].json
+
+
+}
+resource "aws_iam_role_policy_attachment" "irsa1" {
+ count = var.enable_irsa ? 1 : 0
+
+ role = aws_iam_role.controller[0].name
+ policy_arn = aws_iam_policy.irsa1[0].arn
+}
\ No newline at end of file
diff --git a/modules/karpenter/migrations.tf b/modules/karpenter/migrations.tf
new file mode 100644
index 0000000000..b40040f330
--- /dev/null
+++ b/modules/karpenter/migrations.tf
@@ -0,0 +1,77 @@
+################################################################################
+# Migrations: v19.21 -> v20.0
+################################################################################
+
+# Node IAM role
+moved {
+ from = aws_iam_role.this
+ to = aws_iam_role.node
+}
+
+moved {
+ from = aws_iam_policy.this
+ to = aws_iam_policy.node
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.this
+ to = aws_iam_role_policy_attachment.node
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.additional
+ to = aws_iam_role_policy_attachment.node_additional
+}
+
+# Controller IAM role
+moved {
+ from = aws_iam_role.irsa
+ to = aws_iam_role.controller
+}
+
+moved {
+ from = aws_iam_policy.irsa
+ to = aws_iam_policy.controller
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.irsa
+ to = aws_iam_role_policy_attachment.controller
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.irsa_additional
+ to = aws_iam_role_policy_attachment.controller_additional
+}
+
+# Spelling correction
+moved {
+ from = aws_cloudwatch_event_target.this["spot_interupt"]
+ to = aws_cloudwatch_event_target.this["spot_interrupt"]
+}
+
+moved {
+ from = aws_cloudwatch_event_rule.this["spot_interupt"]
+ to = aws_cloudwatch_event_rule.this["spot_interrupt"]
+}
+
+################################################################################
+# Migrations: v20.7 -> v20.8
+################################################################################
+
+# Node IAM role policy attachment
+# Commercial partition only - `moved` does now allow multiple moves to same target
+moved {
+ from = aws_iam_role_policy_attachment.node["arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"]
+ to = aws_iam_role_policy_attachment.node["AmazonEKSWorkerNodePolicy"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.node["arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"]
+ to = aws_iam_role_policy_attachment.node["AmazonEC2ContainerRegistryReadOnly"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.node["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"]
+ to = aws_iam_role_policy_attachment.node["AmazonEKS_CNI_Policy"]
+}
diff --git a/modules/karpenter/outputs.tf b/modules/karpenter/outputs.tf
index 947de39bfd..a71d47242d 100644
--- a/modules/karpenter/outputs.tf
+++ b/modules/karpenter/outputs.tf
@@ -1,20 +1,20 @@
################################################################################
-# IAM Role for Service Account (IRSA)
+# Karpenter controller IAM Role
################################################################################
-output "irsa_name" {
- description = "The name of the IAM role for service accounts"
- value = try(aws_iam_role.irsa[0].name, null)
+output "iam_role_name" {
+ description = "The name of the controller IAM role"
+ value = try(aws_iam_role.controller[0].name, null)
}
-output "irsa_arn" {
- description = "The Amazon Resource Name (ARN) specifying the IAM role for service accounts"
- value = try(aws_iam_role.irsa[0].arn, null)
+output "iam_role_arn" {
+ description = "The Amazon Resource Name (ARN) specifying the controller IAM role"
+ value = try(aws_iam_role.controller[0].arn, null)
}
-output "irsa_unique_id" {
- description = "Stable and unique string identifying the IAM role for service accounts"
- value = try(aws_iam_role.irsa[0].unique_id, null)
+output "iam_role_unique_id" {
+ description = "Stable and unique string identifying the controller IAM role"
+ value = try(aws_iam_role.controller[0].unique_id, null)
}
################################################################################
@@ -49,19 +49,28 @@ output "event_rules" {
# Node IAM Role
################################################################################
-output "role_name" {
- description = "The name of the IAM role"
- value = try(aws_iam_role.this[0].name, null)
+output "node_iam_role_name" {
+ description = "The name of the node IAM role"
+ value = try(aws_iam_role.node[0].name, null)
}
-output "role_arn" {
- description = "The Amazon Resource Name (ARN) specifying the IAM role"
- value = try(aws_iam_role.this[0].arn, var.iam_role_arn)
+output "node_iam_role_arn" {
+ description = "The Amazon Resource Name (ARN) specifying the node IAM role"
+ value = try(aws_iam_role.node[0].arn, var.node_iam_role_arn)
}
-output "role_unique_id" {
- description = "Stable and unique string identifying the IAM role"
- value = try(aws_iam_role.this[0].unique_id, null)
+output "node_iam_role_unique_id" {
+ description = "Stable and unique string identifying the node IAM role"
+ value = try(aws_iam_role.node[0].unique_id, null)
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+output "node_access_entry_arn" {
+ description = "Amazon Resource Name (ARN) of the node Access Entry"
+ value = try(aws_eks_access_entry.node[0].access_entry_arn, null)
}
################################################################################
@@ -87,3 +96,17 @@ output "instance_profile_unique" {
description = "Stable and unique string identifying the IAM instance profile"
value = try(aws_iam_instance_profile.this[0].unique_id, null)
}
+
+################################################################################
+# Pod Identity
+################################################################################
+
+output "namespace" {
+ description = "Namespace associated with the Karpenter Pod Identity"
+ value = var.namespace
+}
+
+output "service_account" {
+ description = "Service Account associated with the Karpenter Pod Identity"
+ value = var.service_account
+}
diff --git a/modules/karpenter/variables.tf b/modules/karpenter/variables.tf
index 95a5a1df93..8d4977a587 100644
--- a/modules/karpenter/variables.tf
+++ b/modules/karpenter/variables.tf
@@ -1,5 +1,5 @@
variable "create" {
- description = "Determines whether to create EKS managed node group or not"
+ description = "Controls if resources should be created (affects nearly all resources)"
type = bool
default = true
}
@@ -17,92 +17,107 @@ variable "cluster_name" {
}
################################################################################
-# IAM Role for Service Account (IRSA)
+# Karpenter controller IAM Role
################################################################################
-variable "create_irsa" {
- description = "Determines whether an IAM role for service accounts is created"
+variable "create_iam_role" {
+ description = "Determines whether an IAM role is created"
type = bool
default = true
}
-variable "irsa_name" {
- description = "Name of IAM role for service accounts"
- type = string
- default = null
-}
-
-variable "irsa_policy_name" {
- description = "Name of IAM policy for service accounts"
+variable "iam_role_name" {
+ description = "Name of the IAM role"
type = string
- default = null
+ default = "KarpenterController"
}
-variable "irsa_use_name_prefix" {
- description = "Determines whether the IAM role for service accounts name (`irsa_name`) is used as a prefix"
+variable "iam_role_use_name_prefix" {
+ description = "Determines whether the name of the IAM role (`iam_role_name`) is used as a prefix"
type = bool
default = true
}
-variable "irsa_path" {
- description = "Path of IAM role for service accounts"
+variable "iam_role_path" {
+ description = "Path of the IAM role"
type = string
default = "/"
}
-variable "irsa_description" {
- description = "IAM role for service accounts description"
+variable "iam_role_description" {
+ description = "IAM role description"
type = string
- default = "Karpenter IAM role for service account"
+ default = "Karpenter controller IAM role"
}
-variable "irsa_max_session_duration" {
+variable "iam_role_max_session_duration" {
description = "Maximum API session duration in seconds between 3600 and 43200"
type = number
default = null
}
-variable "irsa_permissions_boundary_arn" {
- description = "Permissions boundary ARN to use for IAM role for service accounts"
+variable "iam_role_permissions_boundary_arn" {
+ description = "Permissions boundary ARN to use for the IAM role"
type = string
default = null
}
-variable "irsa_tags" {
- description = "A map of additional tags to add the the IAM role for service accounts"
+variable "iam_role_tags" {
+ description = "A map of additional tags to add the the IAM role"
type = map(any)
default = {}
}
-variable "policies" {
- description = "Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format"
- type = map(string)
- default = {}
+variable "iam_policy_name" {
+ description = "Name of the IAM policy"
+ type = string
+ default = "KarpenterController"
+}
+
+variable "iam_policy_use_name_prefix" {
+ description = "Determines whether the name of the IAM policy (`iam_policy_name`) is used as a prefix"
+ type = bool
+ default = true
+}
+
+variable "iam_policy_path" {
+ description = "Path of the IAM policy"
+ type = string
+ default = "/"
}
-variable "irsa_tag_key" {
- description = "Tag key (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner"
+variable "iam_policy_description" {
+ description = "IAM policy description"
type = string
- default = "karpenter.sh/discovery"
+ default = "Karpenter controller IAM policy"
}
-variable "irsa_tag_values" {
- description = "Tag values (`{key = value}`) applied to resources launched by Karpenter through the Karpenter provisioner. Defaults to cluster name when not set."
+variable "iam_role_policies" {
+ description = "Policies to attach to the IAM role in `{'static_name' = 'policy_arn'}` format"
+ type = map(string)
+ default = {}
+}
+
+variable "ami_id_ssm_parameter_arns" {
+ description = "List of SSM Parameter ARNs that Karpenter controller is allowed read access (for retrieving AMI IDs)"
type = list(string)
default = []
}
-variable "irsa_ssm_parameter_arns" {
- description = "List of SSM Parameter ARNs that contain AMI IDs launched by Karpenter"
- type = list(string)
- # https://github.com/aws/karpenter/blob/ed9473a9863ca949b61b9846c8b9f33f35b86dbd/pkg/cloudprovider/aws/ami.go#L105-L123
- default = ["arn:aws:ssm:*:*:parameter/aws/service/*"]
+variable "enable_pod_identity" {
+ description = "Determines whether to enable support for EKS pod identity"
+ type = bool
+ default = true
}
-variable "irsa_subnet_account_id" {
- description = "Account ID of where the subnets Karpenter will utilize resides. Used when subnets are shared from another account"
- type = string
- default = ""
+################################################################################
+# IAM Role for Service Account (IRSA)
+################################################################################
+
+variable "enable_irsa" {
+ description = "Determines whether to enable support for IAM role for service accounts"
+ type = bool
+ default = false
}
variable "irsa_oidc_provider_arn" {
@@ -123,6 +138,28 @@ variable "irsa_assume_role_condition_test" {
default = "StringEquals"
}
+################################################################################
+# Pod Identity Association
+################################################################################
+# TODO - Change default to `true` at next breaking change
+variable "create_pod_identity_association" {
+ description = "Determines whether to create pod identity association"
+ type = bool
+ default = false
+}
+
+variable "namespace" {
+ description = "Namespace to associate with the Karpenter Pod Identity"
+ type = string
+ default = "kube-system"
+}
+
+variable "service_account" {
+ description = "Service account to associate with the Karpenter Pod Identity"
+ type = string
+ default = "karpenter"
+}
+
################################################################################
# Node Termination Queue
################################################################################
@@ -158,81 +195,97 @@ variable "queue_kms_data_key_reuse_period_seconds" {
}
################################################################################
-# Node IAM Role & Instance Profile
+# Node IAM Role
################################################################################
-variable "create_iam_role" {
+variable "create_node_iam_role" {
description = "Determines whether an IAM role is created or to use an existing IAM role"
type = bool
default = true
}
variable "cluster_ip_family" {
- description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
+ description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`. Note: If `ipv6` is specified, the `AmazonEKS_CNI_IPv6_Policy` must exist in the account. This policy is created by the EKS module with `create_cni_ipv6_iam_policy = true`"
type = string
- default = null
+ default = "ipv4"
}
-variable "iam_role_arn" {
+variable "node_iam_role_arn" {
description = "Existing IAM role ARN for the IAM instance profile. Required if `create_iam_role` is set to `false`"
type = string
default = null
}
-variable "iam_role_name" {
+variable "node_iam_role_name" {
description = "Name to use on IAM role created"
type = string
default = null
}
-variable "iam_role_use_name_prefix" {
+variable "node_iam_role_use_name_prefix" {
description = "Determines whether the IAM role name (`iam_role_name`) is used as a prefix"
type = bool
default = true
}
-variable "iam_role_path" {
+variable "node_iam_role_path" {
description = "IAM role path"
type = string
default = "/"
}
-variable "iam_role_description" {
+variable "node_iam_role_description" {
description = "Description of the role"
type = string
default = null
}
-variable "iam_role_max_session_duration" {
+variable "node_iam_role_max_session_duration" {
description = "Maximum API session duration in seconds between 3600 and 43200"
type = number
default = null
}
-variable "iam_role_permissions_boundary" {
+variable "node_iam_role_permissions_boundary" {
description = "ARN of the policy that is used to set the permissions boundary for the IAM role"
type = string
default = null
}
-variable "iam_role_attach_cni_policy" {
+variable "node_iam_role_attach_cni_policy" {
description = "Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster"
type = bool
default = true
}
-variable "iam_role_additional_policies" {
+variable "node_iam_role_additional_policies" {
description = "Additional policies to be added to the IAM role"
- type = list(string)
- default = []
+ type = map(string)
+ default = {}
}
-variable "iam_role_tags" {
+variable "node_iam_role_tags" {
description = "A map of additional tags to add to the IAM role created"
type = map(string)
default = {}
}
+################################################################################
+# Access Entry
+################################################################################
+
+variable "create_access_entry" {
+ description = "Determines whether an access entry is created for the IAM role used by the node IAM role"
+ type = bool
+ default = true
+}
+
+variable "access_entry_type" {
+ description = "Type of the access entry. `EC2_LINUX`, `FARGATE_LINUX`, or `EC2_WINDOWS`; defaults to `EC2_LINUX`"
+ type = string
+ default = "EC2_LINUX"
+}
+
################################################################################
# Node IAM Instance Profile
################################################################################
@@ -240,7 +293,7 @@ variable "iam_role_tags" {
variable "create_instance_profile" {
description = "Whether to create an IAM instance profile"
type = bool
- default = true
+ default = false
}
################################################################################
@@ -251,4 +304,4 @@ variable "rule_name_prefix" {
description = "Prefix used for all event bridge rules"
type = string
default = "Karpenter"
-}
+}
\ No newline at end of file
diff --git a/modules/karpenter/versions.tf b/modules/karpenter/versions.tf
index 55eff62b09..6f83215f50 100644
--- a/modules/karpenter/versions.tf
+++ b/modules/karpenter/versions.tf
@@ -1,10 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
+ version = ">= 5.40"
}
}
}
diff --git a/modules/self-managed-node-group/README.md b/modules/self-managed-node-group/README.md
index 8964144994..5fb9cd3212 100644
--- a/modules/self-managed-node-group/README.md
+++ b/modules/self-managed-node-group/README.md
@@ -42,14 +42,14 @@ module "self_managed_node_group" {
| Name | Version |
|------|---------|
-| [terraform](#requirement\_terraform) | >= 1.0 |
-| [aws](#requirement\_aws) | >= 4.57 |
+| [terraform](#requirement\_terraform) | >= 1.3.2 |
+| [aws](#requirement\_aws) | >= 5.40 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 4.57 |
+| [aws](#provider\_aws) | >= 5.40 |
## Modules
@@ -63,36 +63,47 @@ module "self_managed_node_group" {
|------|------|
| [aws_autoscaling_group.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group) | resource |
| [aws_autoscaling_schedule.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_schedule) | resource |
+| [aws_eks_access_entry.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry) | resource |
| [aws_iam_instance_profile.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile) | resource |
| [aws_iam_role.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
| [aws_iam_role_policy_attachment.additional](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_iam_role_policy_attachment.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_launch_template.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template) | resource |
-| [aws_ami.eks_default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
+| [aws_placement_group.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/placement_group) | resource |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
+| [aws_ec2_instance_type.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type) | data source |
+| [aws_ec2_instance_type_offerings.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type_offerings) | data source |
| [aws_iam_policy_document.assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [aws_ssm_parameter.ami](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ssm_parameter) | data source |
+| [aws_subnets.efa](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
+| [additional\_cluster\_dns\_ips](#input\_additional\_cluster\_dns\_ips) | Additional DNS IP addresses to use for the cluster. Only used when `ami_type` = `BOTTLEROCKET_*` | `list(string)` | `[]` | no |
| [ami\_id](#input\_ami\_id) | The AMI from which to launch the instance | `string` | `""` | no |
+| [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the node group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values | `string` | `"AL2_x86_64"` | no |
| [autoscaling\_group\_tags](#input\_autoscaling\_group\_tags) | A map of additional tags to add to the autoscaling group created. Tags are applied to the autoscaling group only and are NOT propagated to instances | `map(string)` | `{}` | no |
| [availability\_zones](#input\_availability\_zones) | A list of one or more availability zones for the group. Used for EC2-Classic and default subnets when not specified with `subnet_ids` argument. Conflicts with `subnet_ids` | `list(string)` | `null` | no |
| [block\_device\_mappings](#input\_block\_device\_mappings) | Specify volumes to attach to the instance besides the volumes specified by the AMI | `any` | `{}` | no |
-| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `platform` = `bottlerocket`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
+| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
| [capacity\_rebalance](#input\_capacity\_rebalance) | Indicates whether capacity rebalance is enabled | `bool` | `null` | no |
| [capacity\_reservation\_specification](#input\_capacity\_reservation\_specification) | Targeting for EC2 capacity reservations | `any` | `{}` | no |
+| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
| [cluster\_auth\_base64](#input\_cluster\_auth\_base64) | Base64 encoded CA of associated EKS cluster | `string` | `""` | no |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Endpoint of associated EKS cluster | `string` | `""` | no |
-| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `null` | no |
+| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
| [cluster\_name](#input\_cluster\_name) | Name of associated EKS cluster | `string` | `""` | no |
| [cluster\_primary\_security\_group\_id](#input\_cluster\_primary\_security\_group\_id) | The ID of the EKS cluster primary security group to associate with the instance(s). This is the security group that is automatically created by the EKS service | `string` | `null` | no |
+| [cluster\_service\_cidr](#input\_cluster\_service\_cidr) | The CIDR block (IPv4 or IPv6) used by the cluster to assign Kubernetes service IP addresses. This is derived from the cluster itself | `string` | `""` | no |
| [cluster\_version](#input\_cluster\_version) | Kubernetes cluster version - used to lookup default AMI ID if one is not provided | `string` | `null` | no |
| [context](#input\_context) | Reserved | `string` | `null` | no |
| [cpu\_options](#input\_cpu\_options) | The CPU options for the instance | `map(string)` | `{}` | no |
| [create](#input\_create) | Determines whether to create self managed node group or not | `bool` | `true` | no |
+| [create\_access\_entry](#input\_create\_access\_entry) | Determines whether an access entry is created for the IAM role used by the nodegroup | `bool` | `true` | no |
| [create\_autoscaling\_group](#input\_create\_autoscaling\_group) | Determines whether to create autoscaling group or not | `bool` | `true` | no |
| [create\_iam\_instance\_profile](#input\_create\_iam\_instance\_profile) | Determines whether an IAM instance profile is created or to use an existing IAM instance profile | `bool` | `true` | no |
| [create\_launch\_template](#input\_create\_launch\_template) | Determines whether to create launch template or not | `bool` | `true` | no |
@@ -106,6 +117,7 @@ module "self_managed_node_group" {
| [ebs\_optimized](#input\_ebs\_optimized) | If true, the launched EC2 instance will be EBS-optimized | `bool` | `null` | no |
| [elastic\_gpu\_specifications](#input\_elastic\_gpu\_specifications) | The elastic GPU to attach to the instance | `any` | `{}` | no |
| [elastic\_inference\_accelerator](#input\_elastic\_inference\_accelerator) | Configuration block containing an Elastic Inference Accelerator to attach to the instance | `map(string)` | `{}` | no |
+| [enable\_efa\_support](#input\_enable\_efa\_support) | Determines whether to enable Elastic Fabric Adapter (EFA) support | `bool` | `false` | no |
| [enable\_monitoring](#input\_enable\_monitoring) | Enables/disables detailed monitoring | `bool` | `true` | no |
| [enabled\_metrics](#input\_enabled\_metrics) | A list of metrics to collect. The allowed values are `GroupDesiredCapacity`, `GroupInServiceCapacity`, `GroupPendingCapacity`, `GroupMinSize`, `GroupMaxSize`, `GroupInServiceInstances`, `GroupPendingInstances`, `GroupStandbyInstances`, `GroupStandbyCapacity`, `GroupTerminatingCapacity`, `GroupTerminatingInstances`, `GroupTotalCapacity`, `GroupTotalInstances` | `list(string)` | `[]` | no |
| [enclave\_options](#input\_enclave\_options) | Enable Nitro Enclaves on launched instances | `map(string)` | `{}` | no |
@@ -116,6 +128,7 @@ module "self_managed_node_group" {
| [hibernation\_options](#input\_hibernation\_options) | The hibernation options for the instance | `map(string)` | `{}` | no |
| [iam\_instance\_profile\_arn](#input\_iam\_instance\_profile\_arn) | Amazon Resource Name (ARN) of an existing IAM instance profile that provides permissions for the node group. Required if `create_iam_instance_profile` = `false` | `string` | `null` | no |
| [iam\_role\_additional\_policies](#input\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `map(string)` | `{}` | no |
+| [iam\_role\_arn](#input\_iam\_role\_arn) | ARN of the IAM role used by the instance profile. Required when `create_access_entry = true` and `create_iam_instance_profile = false` | `string` | `null` | no |
| [iam\_role\_attach\_cni\_policy](#input\_iam\_role\_attach\_cni\_policy) | Whether to attach the `AmazonEKS_CNI_Policy`/`AmazonEKS_CNI_IPv6_Policy` IAM policy to the IAM IAM role. WARNING: If set `false` the permissions must be assigned to the `aws-node` DaemonSet pods via another method or nodes will not be able to join the cluster | `bool` | `true` | no |
| [iam\_role\_description](#input\_iam\_role\_description) | Description of the role | `string` | `null` | no |
| [iam\_role\_name](#input\_iam\_role\_name) | Name to use on IAM role created | `string` | `null` | no |
@@ -125,6 +138,7 @@ module "self_managed_node_group" {
| [iam\_role\_use\_name\_prefix](#input\_iam\_role\_use\_name\_prefix) | Determines whether cluster IAM role name (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| [initial\_lifecycle\_hooks](#input\_initial\_lifecycle\_hooks) | One or more Lifecycle Hooks to attach to the Auto Scaling Group before instances are launched. The syntax is exactly the same as the separate `aws_autoscaling_lifecycle_hook` resource, without the `autoscaling_group_name` attribute. Please note that this will only work when creating a new Auto Scaling Group. For all other use-cases, please use `aws_autoscaling_lifecycle_hook` resource | `list(map(string))` | `[]` | no |
| [instance\_initiated\_shutdown\_behavior](#input\_instance\_initiated\_shutdown\_behavior) | Shutdown behavior for the instance. Can be `stop` or `terminate`. (Default: `stop`) | `string` | `null` | no |
+| [instance\_maintenance\_policy](#input\_instance\_maintenance\_policy) | If this block is configured, add a instance maintenance policy to the specified Auto Scaling group | `any` | `{}` | no |
| [instance\_market\_options](#input\_instance\_market\_options) | The market (purchasing) option for the instance | `any` | `{}` | no |
| [instance\_refresh](#input\_instance\_refresh) | If this block is configured, start an Instance Refresh when this Auto Scaling Group is updated | `any` | {
"preferences": {
"min_healthy_percentage": 66
},
"strategy": "Rolling"
} | no |
| [instance\_requirements](#input\_instance\_requirements) | The attribute requirements for the type of instance. If present then `instance_type` cannot be present | `any` | `{}` | no |
@@ -151,9 +165,9 @@ module "self_managed_node_group" {
| [network\_interfaces](#input\_network\_interfaces) | Customize network interfaces to be attached at instance boot time | `list(any)` | `[]` | no |
| [placement](#input\_placement) | The placement of the instance | `map(string)` | `{}` | no |
| [placement\_group](#input\_placement\_group) | The name of the placement group into which you'll launch your instances, if any | `string` | `null` | no |
-| [platform](#input\_platform) | Identifies if the OS platform is `bottlerocket`, `linux`, or `windows` based | `string` | `"linux"` | no |
-| [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no |
-| [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket` | `string` | `""` | no |
+| [platform](#input\_platform) | [DEPRECATED - must use `ami_type` instead. Will be removed in `v21.0`] | `string` | `null` | no |
+| [post\_bootstrap\_user\_data](#input\_post\_bootstrap\_user\_data) | User data that is appended to the user data script after of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*` | `string` | `""` | no |
+| [pre\_bootstrap\_user\_data](#input\_pre\_bootstrap\_user\_data) | User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*` | `string` | `""` | no |
| [private\_dns\_name\_options](#input\_private\_dns\_name\_options) | The options for the instance hostname. The default values are inherited from the subnet | `map(string)` | `{}` | no |
| [protect\_from\_scale\_in](#input\_protect\_from\_scale\_in) | Allows setting instance protection. The autoscaling group will not select instances with this setting for termination during scale in events. | `bool` | `false` | no |
| [ram\_disk\_id](#input\_ram\_disk\_id) | The ID of the ram disk | `string` | `null` | no |
@@ -178,6 +192,7 @@ module "self_managed_node_group" {
| Name | Description |
|------|-------------|
+| [access\_entry\_arn](#output\_access\_entry\_arn) | Amazon Resource Name (ARN) of the Access Entry |
| [autoscaling\_group\_arn](#output\_autoscaling\_group\_arn) | The ARN for this autoscaling group |
| [autoscaling\_group\_availability\_zones](#output\_autoscaling\_group\_availability\_zones) | The availability zones of the autoscaling group |
| [autoscaling\_group\_default\_cooldown](#output\_autoscaling\_group\_default\_cooldown) | Time between a scaling activity and the succeeding scaling activity |
@@ -201,6 +216,6 @@ module "self_managed_node_group" {
| [launch\_template\_id](#output\_launch\_template\_id) | The ID of the launch template |
| [launch\_template\_latest\_version](#output\_launch\_template\_latest\_version) | The latest version of the launch template |
| [launch\_template\_name](#output\_launch\_template\_name) | The name of the launch template |
-| [platform](#output\_platform) | Identifies if the OS platform is `bottlerocket`, `linux`, or `windows` based |
+| [platform](#output\_platform) | [DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows` |
| [user\_data](#output\_user\_data) | Base64 encoded user data |
diff --git a/modules/self-managed-node-group/main.tf b/modules/self-managed-node-group/main.tf
index f46096470b..25b202dc39 100644
--- a/modules/self-managed-node-group/main.tf
+++ b/modules/self-managed-node-group/main.tf
@@ -1,16 +1,55 @@
data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}
-data "aws_ami" "eks_default" {
- count = var.create && var.create_launch_template ? 1 : 0
+################################################################################
+# AMI SSM Parameter
+################################################################################
- filter {
- name = "name"
- values = ["amazon-eks-node-${var.cluster_version}-v*"]
+locals {
+ # Just to ensure templating doesn't fail when values are not provided
+ ssm_cluster_version = var.cluster_version != null ? var.cluster_version : ""
+
+ # TODO - Temporary stopgap for backwards compatibility until v21.0
+ ami_type_to_user_data_type = {
+ AL2_x86_64 = "linux"
+ AL2_x86_64_GPU = "linux"
+ AL2_ARM_64 = "linux"
+ BOTTLEROCKET_ARM_64 = "bottlerocket"
+ BOTTLEROCKET_x86_64 = "bottlerocket"
+ BOTTLEROCKET_ARM_64_NVIDIA = "bottlerocket"
+ BOTTLEROCKET_x86_64_NVIDIA = "bottlerocket"
+ WINDOWS_CORE_2019_x86_64 = "windows"
+ WINDOWS_FULL_2019_x86_64 = "windows"
+ WINDOWS_CORE_2022_x86_64 = "windows"
+ WINDOWS_FULL_2022_x86_64 = "windows"
+ AL2023_x86_64_STANDARD = "al2023"
+ AL2023_ARM_64_STANDARD = "al2023"
+ }
+
+ user_data_type = local.ami_type_to_user_data_type[var.ami_type]
+
+ # Map the AMI type to the respective SSM param path
+ ami_type_to_ssm_param = {
+ AL2_x86_64 = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2/recommended/image_id"
+ AL2_x86_64_GPU = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2-gpu/recommended/image_id"
+ AL2_ARM_64 = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2-arm64/recommended/image_id"
+ BOTTLEROCKET_ARM_64 = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}/arm64/latest/image_id"
+ BOTTLEROCKET_x86_64 = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}/x86_64/latest/image_id"
+ BOTTLEROCKET_ARM_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}-nvidia/arm64/latest/image_id"
+ BOTTLEROCKET_x86_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-${local.ssm_cluster_version}-nvidia/x86_64/latest/image_id"
+ WINDOWS_CORE_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Full-EKS_Optimized-${local.ssm_cluster_version}/image_id"
+ WINDOWS_FULL_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-${local.ssm_cluster_version}/image_id"
+ WINDOWS_CORE_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-EKS_Optimized-${local.ssm_cluster_version}/image_id"
+ WINDOWS_FULL_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-${local.ssm_cluster_version}/image_id"
+ AL2023_x86_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/standard/recommended/image_id"
+ AL2023_ARM_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/arm64/standard/recommended/image_id"
}
+}
+
+data "aws_ssm_parameter" "ami" {
+ count = var.create ? 1 : 0
- most_recent = true
- owners = ["amazon"]
+ name = local.ami_type_to_ssm_param[var.ami_type]
}
################################################################################
@@ -21,18 +60,54 @@ module "user_data" {
source = "../_user_data"
create = var.create
- platform = var.platform
+ platform = local.user_data_type
+ ami_type = var.ami_type
is_eks_managed_node_group = false
- cluster_name = var.cluster_name
- cluster_endpoint = var.cluster_endpoint
- cluster_auth_base64 = var.cluster_auth_base64
+ cluster_name = var.cluster_name
+ cluster_endpoint = var.cluster_endpoint
+ cluster_auth_base64 = var.cluster_auth_base64
+ cluster_ip_family = var.cluster_ip_family
+ cluster_service_cidr = var.cluster_service_cidr
+ additional_cluster_dns_ips = var.additional_cluster_dns_ips
enable_bootstrap_user_data = true
pre_bootstrap_user_data = var.pre_bootstrap_user_data
post_bootstrap_user_data = var.post_bootstrap_user_data
bootstrap_extra_args = var.bootstrap_extra_args
user_data_template_path = var.user_data_template_path
+
+ cloudinit_pre_nodeadm = var.cloudinit_pre_nodeadm
+ cloudinit_post_nodeadm = var.cloudinit_post_nodeadm
+}
+
+################################################################################
+# EFA Support
+################################################################################
+
+data "aws_ec2_instance_type" "this" {
+ count = local.enable_efa_support ? 1 : 0
+
+ instance_type = var.instance_type
+}
+
+locals {
+ enable_efa_support = var.create && var.enable_efa_support && local.instance_type_provided
+
+ instance_type_provided = var.instance_type != ""
+ num_network_cards = try(data.aws_ec2_instance_type.this[0].maximum_network_cards, 0)
+
+ efa_network_interfaces = [
+ for i in range(local.num_network_cards) : {
+ associate_public_ip_address = false
+ delete_on_termination = true
+ device_index = i == 0 ? 0 : 1
+ network_card_index = i
+ interface_type = "efa"
+ }
+ ]
+
+ network_interfaces = local.enable_efa_support ? local.efa_network_interfaces : var.network_interfaces
}
################################################################################
@@ -42,6 +117,8 @@ module "user_data" {
locals {
launch_template_name = coalesce(var.launch_template_name, var.name)
security_group_ids = compact(concat([var.cluster_primary_security_group_id], var.vpc_security_group_ids))
+
+ placement = local.enable_efa_support ? { group_name = aws_placement_group.this[0].name } : var.placement
}
resource "aws_launch_template" "this" {
@@ -148,7 +225,7 @@ resource "aws_launch_template" "this" {
arn = var.create_iam_instance_profile ? aws_iam_instance_profile.this[0].arn : var.iam_instance_profile_arn
}
- image_id = coalesce(var.ami_id, data.aws_ami.eks_default[0].image_id)
+ image_id = coalesce(var.ami_id, nonsensitive(data.aws_ssm_parameter.ami[0].value))
instance_initiated_shutdown_behavior = var.instance_initiated_shutdown_behavior
dynamic "instance_market_options" {
@@ -285,7 +362,7 @@ resource "aws_launch_template" "this" {
for_each = length(var.license_specifications) > 0 ? var.license_specifications : {}
content {
- license_configuration_arn = license_specifications.value.license_configuration_arn
+ license_configuration_arn = license_specification.value.license_configuration_arn
}
}
@@ -321,7 +398,8 @@ resource "aws_launch_template" "this" {
name_prefix = var.launch_template_use_name_prefix ? "${local.launch_template_name}-" : null
dynamic "network_interfaces" {
- for_each = var.network_interfaces
+ for_each = local.network_interfaces
+
content {
associate_carrier_ip_address = try(network_interfaces.value.associate_carrier_ip_address, null)
associate_public_ip_address = try(network_interfaces.value.associate_public_ip_address, null)
@@ -347,14 +425,14 @@ resource "aws_launch_template" "this" {
}
dynamic "placement" {
- for_each = length(var.placement) > 0 ? [var.placement] : []
+ for_each = length(local.placement) > 0 ? [local.placement] : []
content {
affinity = try(placement.value.affinity, null)
- availability_zone = try(placement.value.availability_zone, null)
- group_name = try(placement.value.group_name, null)
- host_id = try(placement.value.host_id, null)
- host_resource_group_arn = try(placement.value.host_resource_group_arn, null)
+ availability_zone = lookup(placement.value, "availability_zone", null)
+ group_name = lookup(placement.value, "group_name", null)
+ host_id = lookup(placement.value, "host_id", null)
+ host_resource_group_arn = lookup(placement.value, "host_resource_group_arn", null)
partition_number = try(placement.value.partition_number, null)
spread_domain = try(placement.value.spread_domain, null)
tenancy = try(placement.value.tenancy, null)
@@ -384,7 +462,7 @@ resource "aws_launch_template" "this" {
update_default_version = var.update_launch_template_default_version
user_data = module.user_data.user_data
- vpc_security_group_ids = length(var.network_interfaces) > 0 ? [] : local.security_group_ids
+ vpc_security_group_ids = length(local.network_interfaces) > 0 ? [] : local.security_group_ids
tags = var.tags
@@ -438,6 +516,15 @@ resource "aws_autoscaling_group" "this" {
}
}
+ dynamic "instance_maintenance_policy" {
+ for_each = length(var.instance_maintenance_policy) > 0 ? [var.instance_maintenance_policy] : []
+
+ content {
+ min_healthy_percentage = instance_maintenance_policy.value.min_healthy_percentage
+ max_healthy_percentage = instance_maintenance_policy.value.max_healthy_percentage
+ }
+ }
+
dynamic "instance_refresh" {
for_each = length(var.instance_refresh) > 0 ? [var.instance_refresh] : []
@@ -446,11 +533,14 @@ resource "aws_autoscaling_group" "this" {
for_each = try([instance_refresh.value.preferences], [])
content {
- checkpoint_delay = try(preferences.value.checkpoint_delay, null)
- checkpoint_percentages = try(preferences.value.checkpoint_percentages, null)
- instance_warmup = try(preferences.value.instance_warmup, null)
- min_healthy_percentage = try(preferences.value.min_healthy_percentage, null)
- skip_matching = try(preferences.value.skip_matching, null)
+ checkpoint_delay = try(preferences.value.checkpoint_delay, null)
+ checkpoint_percentages = try(preferences.value.checkpoint_percentages, null)
+ instance_warmup = try(preferences.value.instance_warmup, null)
+ max_healthy_percentage = try(preferences.value.max_healthy_percentage, null)
+ min_healthy_percentage = try(preferences.value.min_healthy_percentage, null)
+ scale_in_protected_instances = try(preferences.value.scale_in_protected_instances, null)
+ skip_matching = try(preferences.value.skip_matching, null)
+ standby_instances = try(preferences.value.standby_instances, null)
}
}
@@ -527,8 +617,9 @@ resource "aws_autoscaling_group" "this" {
}
}
- accelerator_types = try(instance_requirements.value.accelerator_types, [])
- bare_metal = try(instance_requirements.value.bare_metal, null)
+ accelerator_types = try(instance_requirements.value.accelerator_types, [])
+ allowed_instance_types = try(instance_requirements.value.allowed_instance_types, null)
+ bare_metal = try(instance_requirements.value.bare_metal, null)
dynamic "baseline_ebs_bandwidth_mbps" {
for_each = try([instance_requirements.value.baseline_ebs_bandwidth_mbps], [])
@@ -708,7 +799,7 @@ resource "aws_autoscaling_group" "this" {
target_group_arns = var.target_group_arns
termination_policies = var.termination_policies
- vpc_zone_identifier = var.subnet_ids
+ vpc_zone_identifier = local.enable_efa_support ? data.aws_subnets.efa[0].ids : var.subnet_ids
wait_for_capacity_timeout = var.wait_for_capacity_timeout
wait_for_elb_capacity = var.wait_for_elb_capacity
@@ -742,40 +833,26 @@ resource "aws_autoscaling_group" "this" {
}
}
-################################################################################
-# Autoscaling group schedule
-################################################################################
-
-resource "aws_autoscaling_schedule" "this" {
- for_each = { for k, v in var.schedules : k => v if var.create && var.create_schedule }
-
- scheduled_action_name = each.key
- autoscaling_group_name = aws_autoscaling_group.this[0].name
-
- min_size = try(each.value.min_size, null)
- max_size = try(each.value.max_size, null)
- desired_capacity = try(each.value.desired_size, null)
- start_time = try(each.value.start_time, null)
- end_time = try(each.value.end_time, null)
- time_zone = try(each.value.time_zone, null)
-
- # [Minute] [Hour] [Day_of_Month] [Month_of_Year] [Day_of_Week]
- # Cron examples: https://crontab.guru/examples.html
- recurrence = try(each.value.recurrence, null)
-}
-
################################################################################
# IAM Role
################################################################################
locals {
- iam_role_name = coalesce(var.iam_role_name, var.name)
+ create_iam_instance_profile = var.create && var.create_iam_instance_profile
+
+ iam_role_name = coalesce(var.iam_role_name, "${var.name}")
iam_role_policy_prefix = "arn:${data.aws_partition.current.partition}:iam::aws:policy"
- cni_policy = var.cluster_ip_family == "ipv6" ? "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy" : "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+
+ ipv4_cni_policy = { for k, v in {
+ AmazonEKS_CNI_Policy = "${local.iam_role_policy_prefix}/AmazonEKS_CNI_Policy"
+ } : k => v if var.iam_role_attach_cni_policy && var.cluster_ip_family == "ipv4" }
+ ipv6_cni_policy = { for k, v in {
+ AmazonEKS_CNI_IPv6_Policy = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:policy/AmazonEKS_CNI_IPv6_Policy"
+ } : k => v if var.iam_role_attach_cni_policy && var.cluster_ip_family == "ipv6" }
}
data "aws_iam_policy_document" "assume_role_policy" {
- count = var.create && var.create_iam_instance_profile ? 1 : 0
+ count = local.create_iam_instance_profile ? 1 : 0
statement {
sid = "EKSNodeAssumeRole"
@@ -783,13 +860,13 @@ data "aws_iam_policy_document" "assume_role_policy" {
principals {
type = "Service"
- identifiers = ["ec2.${data.aws_partition.current.dns_suffix}"]
+ identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "this" {
- count = var.create && var.create_iam_instance_profile ? 1 : 0
+ count = local.create_iam_instance_profile ? 1 : 0
name = var.iam_role_use_name_prefix ? null : local.iam_role_name
name_prefix = var.iam_role_use_name_prefix ? "${local.iam_role_name}-" : null
@@ -803,26 +880,30 @@ resource "aws_iam_role" "this" {
tags = merge(var.tags, var.iam_role_tags)
}
+# Policies attached ref https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group
resource "aws_iam_role_policy_attachment" "this" {
- for_each = { for k, v in toset(compact([
- "${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy",
- "${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly",
- var.iam_role_attach_cni_policy ? local.cni_policy : "",
- ])) : k => v if var.create && var.create_iam_instance_profile }
+ for_each = { for k, v in merge(
+ {
+ AmazonEKSWorkerNodePolicy = "${local.iam_role_policy_prefix}/AmazonEKSWorkerNodePolicy"
+ AmazonEC2ContainerRegistryReadOnly = "${local.iam_role_policy_prefix}/AmazonEC2ContainerRegistryReadOnly"
+ },
+ local.ipv4_cni_policy,
+ local.ipv6_cni_policy
+ ) : k => v if local.create_iam_instance_profile }
policy_arn = each.value
role = aws_iam_role.this[0].name
}
resource "aws_iam_role_policy_attachment" "additional" {
- for_each = { for k, v in var.iam_role_additional_policies : k => v if var.create && var.create_iam_instance_profile }
+ for_each = { for k, v in var.iam_role_additional_policies : k => v if local.create_iam_instance_profile }
policy_arn = each.value
role = aws_iam_role.this[0].name
}
resource "aws_iam_instance_profile" "this" {
- count = var.create && var.create_iam_instance_profile ? 1 : 0
+ count = local.create_iam_instance_profile ? 1 : 0
role = aws_iam_role.this[0].name
@@ -836,3 +917,89 @@ resource "aws_iam_instance_profile" "this" {
create_before_destroy = true
}
}
+
+################################################################################
+# Placement Group
+################################################################################
+
+resource "aws_placement_group" "this" {
+ count = local.enable_efa_support ? 1 : 0
+
+ name = "${var.cluster_name}-${var.name}"
+ strategy = "cluster"
+
+ tags = var.tags
+}
+
+################################################################################
+# Instance AZ Lookup
+
+# Instances usually used in placement groups w/ EFA are only available in
+# select availability zones. These data sources will cross reference the availability
+# zones supported by the instance type with the subnets provided to ensure only
+# AZs/subnets that are supported are used.
+################################################################################
+
+# Find the availability zones supported by the instance type
+data "aws_ec2_instance_type_offerings" "this" {
+ count = local.enable_efa_support ? 1 : 0
+
+ filter {
+ name = "instance-type"
+ values = [var.instance_type]
+ }
+
+ location_type = "availability-zone-id"
+}
+
+# Reverse the lookup to find one of the subnets provided based on the availability
+# availability zone ID of the queried instance type (supported)
+data "aws_subnets" "efa" {
+ count = local.enable_efa_support ? 1 : 0
+
+ filter {
+ name = "subnet-id"
+ values = var.subnet_ids
+ }
+
+ filter {
+ name = "availability-zone-id"
+ values = data.aws_ec2_instance_type_offerings.this[0].locations
+ }
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+resource "aws_eks_access_entry" "this" {
+ count = var.create && var.create_access_entry ? 1 : 0
+
+ cluster_name = var.cluster_name
+ principal_arn = var.create_iam_instance_profile ? aws_iam_role.this[0].arn : var.iam_role_arn
+ type = local.user_data_type == "windows" ? "EC2_WINDOWS" : "EC2_LINUX"
+
+ tags = var.tags
+}
+
+################################################################################
+# Autoscaling group schedule
+################################################################################
+
+resource "aws_autoscaling_schedule" "this" {
+ for_each = { for k, v in var.schedules : k => v if var.create && var.create_schedule }
+
+ scheduled_action_name = each.key
+ autoscaling_group_name = aws_autoscaling_group.this[0].name
+
+ min_size = try(each.value.min_size, null)
+ max_size = try(each.value.max_size, null)
+ desired_capacity = try(each.value.desired_size, null)
+ start_time = try(each.value.start_time, null)
+ end_time = try(each.value.end_time, null)
+ time_zone = try(each.value.time_zone, null)
+
+ # [Minute] [Hour] [Day_of_Month] [Month_of_Year] [Day_of_Week]
+ # Cron examples: https://crontab.guru/examples.html
+ recurrence = try(each.value.recurrence, null)
+}
diff --git a/modules/self-managed-node-group/migrations.tf b/modules/self-managed-node-group/migrations.tf
new file mode 100644
index 0000000000..5d51a7208a
--- /dev/null
+++ b/modules/self-managed-node-group/migrations.tf
@@ -0,0 +1,20 @@
+################################################################################
+# Migrations: v20.7 -> v20.8
+################################################################################
+
+# Node IAM role policy attachment
+# Commercial partition only - `moved` does now allow multiple moves to same target
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"]
+ to = aws_iam_role_policy_attachment.this["AmazonEKSWorkerNodePolicy"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"]
+ to = aws_iam_role_policy_attachment.this["AmazonEC2ContainerRegistryReadOnly"]
+}
+
+moved {
+ from = aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"]
+ to = aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"]
+}
diff --git a/modules/self-managed-node-group/outputs.tf b/modules/self-managed-node-group/outputs.tf
index 5c83497218..9607810ac3 100644
--- a/modules/self-managed-node-group/outputs.tf
+++ b/modules/self-managed-node-group/outputs.tf
@@ -81,15 +81,6 @@ output "autoscaling_group_vpc_zone_identifier" {
value = try(aws_autoscaling_group.this[0].vpc_zone_identifier, null)
}
-################################################################################
-# Autoscaling Group Schedule
-################################################################################
-
-output "autoscaling_group_schedule_arns" {
- description = "ARNs of autoscaling group schedules"
- value = { for k, v in aws_autoscaling_schedule.this : k => v.arn }
-}
-
################################################################################
# IAM Role
################################################################################
@@ -128,13 +119,31 @@ output "iam_instance_profile_unique" {
value = try(aws_iam_instance_profile.this[0].unique_id, null)
}
+################################################################################
+# Access Entry
+################################################################################
+
+output "access_entry_arn" {
+ description = "Amazon Resource Name (ARN) of the Access Entry"
+ value = try(aws_eks_access_entry.this[0].access_entry_arn, null)
+}
+
+################################################################################
+# Autoscaling Group Schedule
+################################################################################
+
+output "autoscaling_group_schedule_arns" {
+ description = "ARNs of autoscaling group schedules"
+ value = { for k, v in aws_autoscaling_schedule.this : k => v.arn }
+}
+
################################################################################
# Additional
################################################################################
output "platform" {
- description = "Identifies if the OS platform is `bottlerocket`, `linux`, or `windows` based"
- value = var.platform
+ description = "[DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows`"
+ value = module.user_data.platform
}
output "image_id" {
diff --git a/modules/self-managed-node-group/variables.tf b/modules/self-managed-node-group/variables.tf
index 7e5d0cecb6..aea1269080 100644
--- a/modules/self-managed-node-group/variables.tf
+++ b/modules/self-managed-node-group/variables.tf
@@ -10,10 +10,16 @@ variable "tags" {
default = {}
}
+# tflint-ignore: terraform_unused_declarations
variable "platform" {
- description = "Identifies if the OS platform is `bottlerocket`, `linux`, or `windows` based"
+ description = "[DEPRECATED - must use `ami_type` instead. Will be removed in `v21.0`]"
type = string
- default = "linux"
+ default = null
+
+ validation {
+ condition = var.platform == null
+ error_message = "`platform` is no longer valid due to the number of OS choices. Please provide an [`ami_type`](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-nodegroup.html#cfn-eks-nodegroup-amitype) instead."
+ }
}
################################################################################
@@ -38,20 +44,38 @@ variable "cluster_auth_base64" {
default = ""
}
+variable "cluster_service_cidr" {
+ description = "The CIDR block (IPv4 or IPv6) used by the cluster to assign Kubernetes service IP addresses. This is derived from the cluster itself"
+ type = string
+ default = ""
+}
+
+variable "cluster_ip_family" {
+ description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
+ type = string
+ default = "ipv4"
+}
+
+variable "additional_cluster_dns_ips" {
+ description = "Additional DNS IP addresses to use for the cluster. Only used when `ami_type` = `BOTTLEROCKET_*`"
+ type = list(string)
+ default = []
+}
+
variable "pre_bootstrap_user_data" {
- description = "User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `platform` = `bottlerocket`"
+ description = "User data that is injected into the user data script ahead of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*`"
type = string
default = ""
}
variable "post_bootstrap_user_data" {
- description = "User data that is appended to the user data script after of the EKS bootstrap script. Not used when `platform` = `bottlerocket`"
+ description = "User data that is appended to the user data script after of the EKS bootstrap script. Not used when `ami_type` = `BOTTLEROCKET_*`"
type = string
default = ""
}
variable "bootstrap_extra_args" {
- description = "Additional arguments passed to the bootstrap script. When `platform` = `bottlerocket`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data"
+ description = "Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data"
type = string
default = ""
}
@@ -62,6 +86,28 @@ variable "user_data_template_path" {
default = ""
}
+variable "cloudinit_pre_nodeadm" {
+ description = "Array of cloud-init document parts that are created before the nodeadm document part"
+ type = list(object({
+ content = string
+ content_type = optional(string)
+ filename = optional(string)
+ merge_type = optional(string)
+ }))
+ default = []
+}
+
+variable "cloudinit_post_nodeadm" {
+ description = "Array of cloud-init document parts that are created after the nodeadm document part"
+ type = list(object({
+ content = string
+ content_type = optional(string)
+ filename = optional(string)
+ merge_type = optional(string)
+ }))
+ default = []
+}
+
################################################################################
# Launch template
################################################################################
@@ -228,6 +274,12 @@ variable "ami_id" {
default = ""
}
+variable "ami_type" {
+ description = "Type of Amazon Machine Image (AMI) associated with the node group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values"
+ type = string
+ default = "AL2_x86_64"
+}
+
variable "cluster_version" {
description = "Kubernetes cluster version - used to lookup default AMI ID if one is not provided"
type = string
@@ -270,6 +322,12 @@ variable "enable_monitoring" {
default = true
}
+variable "enable_efa_support" {
+ description = "Determines whether to enable Elastic Fabric Adapter (EFA) support"
+ type = bool
+ default = false
+}
+
variable "metadata_options" {
description = "Customize the metadata options for the instance"
type = map(string)
@@ -476,6 +534,12 @@ variable "initial_lifecycle_hooks" {
default = []
}
+variable "instance_maintenance_policy" {
+ description = "If this block is configured, add a instance maintenance policy to the specified Auto Scaling group"
+ type = any
+ default = {}
+}
+
variable "instance_refresh" {
description = "If this block is configured, start an Instance Refresh when this Auto Scaling Group is updated"
type = any
@@ -517,22 +581,6 @@ variable "autoscaling_group_tags" {
default = {}
}
-################################################################################
-# Autoscaling group schedule
-################################################################################
-
-variable "create_schedule" {
- description = "Determines whether to create autoscaling group schedule or not"
- type = bool
- default = true
-}
-
-variable "schedules" {
- description = "Map of autoscaling group schedule to create"
- type = map(any)
- default = {}
-}
-
################################################################################
# IAM Role
################################################################################
@@ -543,12 +591,6 @@ variable "create_iam_instance_profile" {
default = true
}
-variable "cluster_ip_family" {
- description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`"
- type = string
- default = null
-}
-
variable "iam_instance_profile_arn" {
description = "Amazon Resource Name (ARN) of an existing IAM instance profile that provides permissions for the node group. Required if `create_iam_instance_profile` = `false`"
type = string
@@ -602,3 +644,35 @@ variable "iam_role_tags" {
type = map(string)
default = {}
}
+
+################################################################################
+# Access Entry
+################################################################################
+
+variable "create_access_entry" {
+ description = "Determines whether an access entry is created for the IAM role used by the nodegroup"
+ type = bool
+ default = true
+}
+
+variable "iam_role_arn" {
+ description = "ARN of the IAM role used by the instance profile. Required when `create_access_entry = true` and `create_iam_instance_profile = false`"
+ type = string
+ default = null
+}
+
+################################################################################
+# Autoscaling group schedule
+################################################################################
+
+variable "create_schedule" {
+ description = "Determines whether to create autoscaling group schedule or not"
+ type = bool
+ default = true
+}
+
+variable "schedules" {
+ description = "Map of autoscaling group schedule to create"
+ type = map(any)
+ default = {}
+}
diff --git a/modules/self-managed-node-group/versions.tf b/modules/self-managed-node-group/versions.tf
index 01d187af62..6f83215f50 100644
--- a/modules/self-managed-node-group/versions.tf
+++ b/modules/self-managed-node-group/versions.tf
@@ -1,10 +1,10 @@
terraform {
- required_version = ">= 1.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.57"
+ version = ">= 5.40"
}
}
}
diff --git a/node_groups.tf b/node_groups.tf
index db78861a81..7228931071 100644
--- a/node_groups.tf
+++ b/node_groups.tf
@@ -17,6 +17,8 @@ locals {
min_healthy_percentage = 66
}
}
+
+ kubernetes_network_config = try(aws_eks_cluster.this[0].kubernetes_network_config[0], {})
}
# This sleep resource is used to provide a timed gap between the cluster creation and the downstream dependencies
@@ -30,9 +32,10 @@ resource "time_sleep" "this" {
create_duration = var.dataplane_wait_duration
triggers = {
- cluster_name = aws_eks_cluster.this[0].name
- cluster_endpoint = aws_eks_cluster.this[0].endpoint
- cluster_version = aws_eks_cluster.this[0].version
+ cluster_name = aws_eks_cluster.this[0].name
+ cluster_endpoint = aws_eks_cluster.this[0].endpoint
+ cluster_version = aws_eks_cluster.this[0].version
+ cluster_service_cidr = var.cluster_ip_family == "ipv6" ? try(local.kubernetes_network_config.service_ipv6_cidr, "") : try(local.kubernetes_network_config.service_ipv4_cidr, "")
cluster_certificate_authority_data = aws_eks_cluster.this[0].certificate_authority[0].data
}
@@ -40,7 +43,6 @@ resource "time_sleep" "this" {
################################################################################
# EKS IPV6 CNI Policy
-# TODO - hopefully AWS releases a managed policy which can replace this
# https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html#cni-iam-role-create-ipv6-policy
################################################################################
@@ -62,7 +64,7 @@ data "aws_iam_policy_document" "cni_ipv6_policy" {
statement {
sid = "CreateTags"
actions = ["ec2:CreateTags"]
- resources = ["arn:${data.aws_partition.current.partition}:ec2:*:*:network-interface/*"]
+ resources = ["arn:${local.partition}:ec2:*:*:network-interface/*"]
}
}
@@ -180,6 +182,27 @@ locals {
ipv6_cidr_blocks = var.cluster_ip_family == "ipv6" ? ["::/0"] : null
}
} : k => v if var.node_security_group_enable_recommended_rules }
+
+ efa_security_group_rules = { for k, v in
+ {
+ ingress_all_self_efa = {
+ description = "Node to node EFA"
+ protocol = "-1"
+ from_port = 0
+ to_port = 0
+ type = "ingress"
+ self = true
+ }
+ egress_all_self_efa = {
+ description = "Node to node EFA"
+ protocol = "-1"
+ from_port = 0
+ to_port = 0
+ type = "egress"
+ self = true
+ }
+ } : k => v if var.enable_efa_support
+ }
}
resource "aws_security_group" "node" {
@@ -206,6 +229,7 @@ resource "aws_security_group" "node" {
resource "aws_security_group_rule" "node" {
for_each = { for k, v in merge(
+ local.efa_security_group_rules,
local.node_security_group_rules,
local.node_security_group_recommended_rules,
var.node_security_group_additional_rules,
@@ -274,9 +298,8 @@ module "eks_managed_node_group" {
create = try(each.value.create, true)
- cluster_name = time_sleep.this[0].triggers["cluster_name"]
- cluster_version = try(each.value.cluster_version, var.eks_managed_node_group_defaults.cluster_version, time_sleep.this[0].triggers["cluster_version"])
- cluster_ip_family = var.cluster_ip_family
+ cluster_name = time_sleep.this[0].triggers["cluster_name"]
+ cluster_version = try(each.value.cluster_version, var.eks_managed_node_group_defaults.cluster_version, time_sleep.this[0].triggers["cluster_version"])
# EKS Managed Node Group
name = try(each.value.name, each.key)
@@ -288,9 +311,10 @@ module "eks_managed_node_group" {
max_size = try(each.value.max_size, var.eks_managed_node_group_defaults.max_size, 3)
desired_size = try(each.value.desired_size, var.eks_managed_node_group_defaults.desired_size, 1)
- ami_id = try(each.value.ami_id, var.eks_managed_node_group_defaults.ami_id, "")
- ami_type = try(each.value.ami_type, var.eks_managed_node_group_defaults.ami_type, null)
- ami_release_version = try(each.value.ami_release_version, var.eks_managed_node_group_defaults.ami_release_version, null)
+ ami_id = try(each.value.ami_id, var.eks_managed_node_group_defaults.ami_id, "")
+ ami_type = try(each.value.ami_type, var.eks_managed_node_group_defaults.ami_type, null)
+ ami_release_version = try(each.value.ami_release_version, var.eks_managed_node_group_defaults.ami_release_version, null)
+ use_latest_ami_release_version = try(each.value.use_latest_ami_release_version, var.eks_managed_node_group_defaults.use_latest_ami_release_version, false)
capacity_type = try(each.value.capacity_type, var.eks_managed_node_group_defaults.capacity_type, null)
disk_size = try(each.value.disk_size, var.eks_managed_node_group_defaults.disk_size, null)
@@ -308,11 +332,15 @@ module "eks_managed_node_group" {
cluster_endpoint = try(time_sleep.this[0].triggers["cluster_endpoint"], "")
cluster_auth_base64 = try(time_sleep.this[0].triggers["cluster_certificate_authority_data"], "")
cluster_service_ipv4_cidr = var.cluster_service_ipv4_cidr
+ cluster_ip_family = var.cluster_ip_family
+ cluster_service_cidr = try(time_sleep.this[0].triggers["cluster_service_cidr"], "")
enable_bootstrap_user_data = try(each.value.enable_bootstrap_user_data, var.eks_managed_node_group_defaults.enable_bootstrap_user_data, false)
pre_bootstrap_user_data = try(each.value.pre_bootstrap_user_data, var.eks_managed_node_group_defaults.pre_bootstrap_user_data, "")
post_bootstrap_user_data = try(each.value.post_bootstrap_user_data, var.eks_managed_node_group_defaults.post_bootstrap_user_data, "")
bootstrap_extra_args = try(each.value.bootstrap_extra_args, var.eks_managed_node_group_defaults.bootstrap_extra_args, "")
user_data_template_path = try(each.value.user_data_template_path, var.eks_managed_node_group_defaults.user_data_template_path, "")
+ cloudinit_pre_nodeadm = try(each.value.cloudinit_pre_nodeadm, var.eks_managed_node_group_defaults.cloudinit_pre_nodeadm, [])
+ cloudinit_post_nodeadm = try(each.value.cloudinit_post_nodeadm, var.eks_managed_node_group_defaults.cloudinit_post_nodeadm, [])
# Launch Template
create_launch_template = try(each.value.create_launch_template, var.eks_managed_node_group_defaults.create_launch_template, true)
@@ -344,6 +372,9 @@ module "eks_managed_node_group" {
license_specifications = try(each.value.license_specifications, var.eks_managed_node_group_defaults.license_specifications, {})
metadata_options = try(each.value.metadata_options, var.eks_managed_node_group_defaults.metadata_options, local.metadata_options)
enable_monitoring = try(each.value.enable_monitoring, var.eks_managed_node_group_defaults.enable_monitoring, true)
+ enable_efa_support = try(each.value.enable_efa_support, var.eks_managed_node_group_defaults.enable_efa_support, false)
+ create_placement_group = try(each.value.create_placement_group, var.eks_managed_node_group_defaults.create_placement_group, false)
+ placement_group_strategy = try(each.value.placement_group_strategy, var.eks_managed_node_group_defaults.placement_group_strategy, "cluster")
network_interfaces = try(each.value.network_interfaces, var.eks_managed_node_group_defaults.network_interfaces, [])
placement = try(each.value.placement, var.eks_managed_node_group_defaults.placement, {})
maintenance_options = try(each.value.maintenance_options, var.eks_managed_node_group_defaults.maintenance_options, {})
@@ -363,6 +394,7 @@ module "eks_managed_node_group" {
# https://github.com/hashicorp/terraform/issues/31646#issuecomment-1217279031
iam_role_additional_policies = lookup(each.value, "iam_role_additional_policies", lookup(var.eks_managed_node_group_defaults, "iam_role_additional_policies", {}))
+ # Autoscaling group schedule
create_schedule = try(each.value.create_schedule, var.eks_managed_node_group_defaults.create_schedule, true)
schedules = try(each.value.schedules, var.eks_managed_node_group_defaults.schedules, {})
@@ -384,8 +416,7 @@ module "self_managed_node_group" {
create = try(each.value.create, true)
- cluster_name = time_sleep.this[0].triggers["cluster_name"]
- cluster_ip_family = var.cluster_ip_family
+ cluster_name = time_sleep.this[0].triggers["cluster_name"]
# Autoscaling Group
create_autoscaling_group = try(each.value.create_autoscaling_group, var.self_managed_node_group_defaults.create_autoscaling_group, true)
@@ -423,26 +454,30 @@ module "self_managed_node_group" {
metrics_granularity = try(each.value.metrics_granularity, var.self_managed_node_group_defaults.metrics_granularity, null)
service_linked_role_arn = try(each.value.service_linked_role_arn, var.self_managed_node_group_defaults.service_linked_role_arn, null)
- initial_lifecycle_hooks = try(each.value.initial_lifecycle_hooks, var.self_managed_node_group_defaults.initial_lifecycle_hooks, [])
- instance_refresh = try(each.value.instance_refresh, var.self_managed_node_group_defaults.instance_refresh, local.default_instance_refresh)
- use_mixed_instances_policy = try(each.value.use_mixed_instances_policy, var.self_managed_node_group_defaults.use_mixed_instances_policy, false)
- mixed_instances_policy = try(each.value.mixed_instances_policy, var.self_managed_node_group_defaults.mixed_instances_policy, null)
- warm_pool = try(each.value.warm_pool, var.self_managed_node_group_defaults.warm_pool, {})
-
- create_schedule = try(each.value.create_schedule, var.self_managed_node_group_defaults.create_schedule, true)
- schedules = try(each.value.schedules, var.self_managed_node_group_defaults.schedules, {})
+ initial_lifecycle_hooks = try(each.value.initial_lifecycle_hooks, var.self_managed_node_group_defaults.initial_lifecycle_hooks, [])
+ instance_maintenance_policy = try(each.value.instance_maintenance_policy, var.self_managed_node_group_defaults.instance_maintenance_policy, {})
+ instance_refresh = try(each.value.instance_refresh, var.self_managed_node_group_defaults.instance_refresh, local.default_instance_refresh)
+ use_mixed_instances_policy = try(each.value.use_mixed_instances_policy, var.self_managed_node_group_defaults.use_mixed_instances_policy, false)
+ mixed_instances_policy = try(each.value.mixed_instances_policy, var.self_managed_node_group_defaults.mixed_instances_policy, null)
+ warm_pool = try(each.value.warm_pool, var.self_managed_node_group_defaults.warm_pool, {})
delete_timeout = try(each.value.delete_timeout, var.self_managed_node_group_defaults.delete_timeout, null)
autoscaling_group_tags = try(each.value.autoscaling_group_tags, var.self_managed_node_group_defaults.autoscaling_group_tags, {})
# User data
- platform = try(each.value.platform, var.self_managed_node_group_defaults.platform, "linux")
+ platform = try(each.value.platform, var.self_managed_node_group_defaults.platform, null)
+ # TODO - update this when `var.platform` is removed in v21.0
+ ami_type = try(each.value.ami_type, var.self_managed_node_group_defaults.ami_type, "AL2_x86_64")
cluster_endpoint = try(time_sleep.this[0].triggers["cluster_endpoint"], "")
cluster_auth_base64 = try(time_sleep.this[0].triggers["cluster_certificate_authority_data"], "")
+ cluster_service_cidr = try(time_sleep.this[0].triggers["cluster_service_cidr"], "")
+ cluster_ip_family = var.cluster_ip_family
pre_bootstrap_user_data = try(each.value.pre_bootstrap_user_data, var.self_managed_node_group_defaults.pre_bootstrap_user_data, "")
post_bootstrap_user_data = try(each.value.post_bootstrap_user_data, var.self_managed_node_group_defaults.post_bootstrap_user_data, "")
bootstrap_extra_args = try(each.value.bootstrap_extra_args, var.self_managed_node_group_defaults.bootstrap_extra_args, "")
user_data_template_path = try(each.value.user_data_template_path, var.self_managed_node_group_defaults.user_data_template_path, "")
+ cloudinit_pre_nodeadm = try(each.value.cloudinit_pre_nodeadm, var.self_managed_node_group_defaults.cloudinit_pre_nodeadm, [])
+ cloudinit_post_nodeadm = try(each.value.cloudinit_post_nodeadm, var.self_managed_node_group_defaults.cloudinit_post_nodeadm, [])
# Launch Template
create_launch_template = try(each.value.create_launch_template, var.self_managed_node_group_defaults.create_launch_template, true)
@@ -480,6 +515,7 @@ module "self_managed_node_group" {
license_specifications = try(each.value.license_specifications, var.self_managed_node_group_defaults.license_specifications, {})
metadata_options = try(each.value.metadata_options, var.self_managed_node_group_defaults.metadata_options, local.metadata_options)
enable_monitoring = try(each.value.enable_monitoring, var.self_managed_node_group_defaults.enable_monitoring, true)
+ enable_efa_support = try(each.value.enable_efa_support, var.self_managed_node_group_defaults.enable_efa_support, false)
network_interfaces = try(each.value.network_interfaces, var.self_managed_node_group_defaults.network_interfaces, [])
placement = try(each.value.placement, var.self_managed_node_group_defaults.placement, {})
maintenance_options = try(each.value.maintenance_options, var.self_managed_node_group_defaults.maintenance_options, {})
@@ -499,6 +535,14 @@ module "self_managed_node_group" {
# https://github.com/hashicorp/terraform/issues/31646#issuecomment-1217279031
iam_role_additional_policies = lookup(each.value, "iam_role_additional_policies", lookup(var.self_managed_node_group_defaults, "iam_role_additional_policies", {}))
+ # Access entry
+ create_access_entry = try(each.value.create_access_entry, var.self_managed_node_group_defaults.create_access_entry, true)
+ iam_role_arn = try(each.value.iam_role_arn, var.self_managed_node_group_defaults.iam_role_arn, null)
+
+ # Autoscaling group schedule
+ create_schedule = try(each.value.create_schedule, var.self_managed_node_group_defaults.create_schedule, true)
+ schedules = try(each.value.schedules, var.self_managed_node_group_defaults.schedules, {})
+
# Security group
vpc_security_group_ids = compact(concat([local.node_security_group_id], try(each.value.vpc_security_group_ids, var.self_managed_node_group_defaults.vpc_security_group_ids, [])))
cluster_primary_security_group_id = try(each.value.attach_cluster_primary_security_group, var.self_managed_node_group_defaults.attach_cluster_primary_security_group, false) ? aws_eks_cluster.this[0].vpc_config[0].cluster_security_group_id : null
diff --git a/outputs.tf b/outputs.tf
index ea02a3a8cc..45b68a4a23 100644
--- a/outputs.tf
+++ b/outputs.tf
@@ -5,16 +5,31 @@
output "cluster_arn" {
description = "The Amazon Resource Name (ARN) of the cluster"
value = try(aws_eks_cluster.this[0].arn, null)
+
+ depends_on = [
+ aws_eks_access_entry.this,
+ aws_eks_access_policy_association.this,
+ ]
}
output "cluster_certificate_authority_data" {
description = "Base64 encoded certificate data required to communicate with the cluster"
value = try(aws_eks_cluster.this[0].certificate_authority[0].data, null)
+
+ depends_on = [
+ aws_eks_access_entry.this,
+ aws_eks_access_policy_association.this,
+ ]
}
output "cluster_endpoint" {
description = "Endpoint for your Kubernetes API server"
value = try(aws_eks_cluster.this[0].endpoint, null)
+
+ depends_on = [
+ aws_eks_access_entry.this,
+ aws_eks_access_policy_association.this,
+ ]
}
output "cluster_id" {
@@ -25,6 +40,11 @@ output "cluster_id" {
output "cluster_name" {
description = "The name of the EKS cluster"
value = try(aws_eks_cluster.this[0].name, "")
+
+ depends_on = [
+ aws_eks_access_entry.this,
+ aws_eks_access_policy_association.this,
+ ]
}
output "cluster_oidc_issuer_url" {
@@ -52,6 +72,30 @@ output "cluster_primary_security_group_id" {
value = try(aws_eks_cluster.this[0].vpc_config[0].cluster_security_group_id, null)
}
+output "cluster_service_cidr" {
+ description = "The CIDR block where Kubernetes pod and service IP addresses are assigned from"
+ value = var.cluster_ip_family == "ipv6" ? try(aws_eks_cluster.this[0].kubernetes_network_config[0].service_ipv6_cidr, null) : try(aws_eks_cluster.this[0].kubernetes_network_config[0].service_ipv4_cidr, null)
+}
+
+output "cluster_ip_family" {
+ description = "The IP family used by the cluster (e.g. `ipv4` or `ipv6`)"
+ value = try(aws_eks_cluster.this[0].kubernetes_network_config[0].ip_family, null)
+}
+
+################################################################################
+# Access Entry
+################################################################################
+
+output "access_entries" {
+ description = "Map of access entries created and their attributes"
+ value = aws_eks_access_entry.this
+}
+
+output "access_policy_associations" {
+ description = "Map of eks cluster access policy associations created and their attributes"
+ value = aws_eks_access_policy_association.this
+}
+
################################################################################
# KMS Key
################################################################################
@@ -205,19 +249,3 @@ output "self_managed_node_groups_autoscaling_group_names" {
description = "List of the autoscaling group names created by self-managed node groups"
value = compact([for group in module.self_managed_node_group : group.autoscaling_group_name])
}
-
-################################################################################
-# Additional
-################################################################################
-
-output "aws_auth_configmap_yaml" {
- description = "[DEPRECATED - use `var.manage_aws_auth_configmap`] Formatted yaml output for base aws-auth configmap containing roles used in cluster node groups/fargate profiles"
- value = templatefile("${path.module}/templates/aws_auth_cm.tpl",
- {
- eks_managed_role_arns = distinct(compact([for group in module.eks_managed_node_group : group.iam_role_arn]))
- self_managed_role_arns = distinct(compact([for group in module.self_managed_node_group : group.iam_role_arn if group.platform != "windows"]))
- win32_self_managed_role_arns = distinct(compact([for group in module.self_managed_node_group : group.iam_role_arn if group.platform == "windows"]))
- fargate_profile_pod_execution_role_arns = distinct(compact([for group in module.fargate_profile : group.fargate_profile_pod_execution_role_arn]))
- }
- )
-}
diff --git a/templates/al2023_user_data.tpl b/templates/al2023_user_data.tpl
new file mode 100644
index 0000000000..cc360e6d65
--- /dev/null
+++ b/templates/al2023_user_data.tpl
@@ -0,0 +1,11 @@
+%{ if enable_bootstrap_user_data ~}
+---
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+ cluster:
+ name: ${cluster_name}
+ apiServerEndpoint: ${cluster_endpoint}
+ certificateAuthority: ${cluster_auth_base64}
+ cidr: ${cluster_service_cidr}
+%{ endif ~}
diff --git a/templates/aws_auth_cm.tpl b/templates/aws_auth_cm.tpl
deleted file mode 100644
index 73a898e966..0000000000
--- a/templates/aws_auth_cm.tpl
+++ /dev/null
@@ -1,37 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: aws-auth
- namespace: kube-system
-data:
- mapRoles: |
-%{ for role in eks_managed_role_arns ~}
- - rolearn: ${role}
- username: system:node:{{EC2PrivateDNSName}}
- groups:
- - system:bootstrappers
- - system:nodes
-%{ endfor ~}
-%{ for role in self_managed_role_arns ~}
- - rolearn: ${role}
- username: system:node:{{EC2PrivateDNSName}}
- groups:
- - system:bootstrappers
- - system:nodes
-%{ endfor ~}
-%{ for role in win32_self_managed_role_arns ~}
- - rolearn: ${role}
- username: system:node:{{EC2PrivateDNSName}}
- groups:
- - eks:kube-proxy-windows
- - system:bootstrappers
- - system:nodes
-%{ endfor ~}
-%{ for role in fargate_profile_pod_execution_role_arns ~}
- - rolearn: ${role}
- username: system:node:{{SessionName}}
- groups:
- - system:bootstrappers
- - system:nodes
- - system:node-proxier
-%{ endfor ~}
diff --git a/templates/bottlerocket_user_data.tpl b/templates/bottlerocket_user_data.tpl
index 640c801438..666d666069 100644
--- a/templates/bottlerocket_user_data.tpl
+++ b/templates/bottlerocket_user_data.tpl
@@ -3,5 +3,6 @@
"cluster-name" = "${cluster_name}"
"api-server" = "${cluster_endpoint}"
"cluster-certificate" = "${cluster_auth_base64}"
+"cluster-dns-ip" = ${cluster_dns_ips}
%{ endif ~}
${bootstrap_extra_args ~}
diff --git a/templates/linux_user_data.tpl b/templates/linux_user_data.tpl
index 14acbd2aff..d75d549ccc 100644
--- a/templates/linux_user_data.tpl
+++ b/templates/linux_user_data.tpl
@@ -3,12 +3,10 @@
set -e
%{ endif ~}
${pre_bootstrap_user_data ~}
-%{ if length(cluster_service_ipv4_cidr) > 0 ~}
-export SERVICE_IPV4_CIDR=${cluster_service_ipv4_cidr}
-%{ endif ~}
%{ if enable_bootstrap_user_data ~}
B64_CLUSTER_CA=${cluster_auth_base64}
API_SERVER_URL=${cluster_endpoint}
-/etc/eks/bootstrap.sh ${cluster_name} ${bootstrap_extra_args} --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
+/etc/eks/bootstrap.sh ${cluster_name} ${bootstrap_extra_args} --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL \
+ --ip-family ${cluster_ip_family} --service-${cluster_ip_family}-cidr ${cluster_service_cidr}
${post_bootstrap_user_data ~}
%{ endif ~}
diff --git a/templates/windows_user_data.tpl b/templates/windows_user_data.tpl
index 5000850604..9721d3cc33 100644
--- a/templates/windows_user_data.tpl
+++ b/templates/windows_user_data.tpl
@@ -1,5 +1,8 @@
+%{ if enable_bootstrap_user_data ~}
+%{ endif ~}
${pre_bootstrap_user_data ~}
+%{ if enable_bootstrap_user_data ~}
[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
@@ -7,3 +10,4 @@ ${pre_bootstrap_user_data ~}
$LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
${post_bootstrap_user_data ~}
+%{ endif ~}
diff --git a/variables.tf b/variables.tf
index 988b97970c..639110a9fc 100644
--- a/variables.tf
+++ b/variables.tf
@@ -1,5 +1,5 @@
variable "create" {
- description = "Controls if EKS resources should be created (affects nearly all resources)"
+ description = "Controls if resources should be created (affects nearly all resources)"
type = bool
default = true
}
@@ -38,6 +38,12 @@ variable "cluster_enabled_log_types" {
default = ["audit", "api", "authenticator"]
}
+variable "authentication_mode" {
+ description = "The authentication mode for the cluster. Valid values are `CONFIG_MAP`, `API` or `API_AND_CONFIG_MAP`"
+ type = string
+ default = "API_AND_CONFIG_MAP"
+}
+
variable "cluster_additional_security_group_ids" {
description = "List of additional, externally created security group IDs to attach to the cluster control plane"
type = list(string)
@@ -77,7 +83,7 @@ variable "cluster_endpoint_public_access_cidrs" {
variable "cluster_ip_family" {
description = "The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6`. You can only specify an IP family when you create a cluster, changing this value will force a new cluster to be created"
type = string
- default = null
+ default = "ipv4"
}
variable "cluster_service_ipv4_cidr" {
@@ -130,6 +136,22 @@ variable "cluster_timeouts" {
default = {}
}
+################################################################################
+# Access Entry
+################################################################################
+
+variable "access_entries" {
+ description = "Map of access entries to add to the cluster"
+ type = any
+ default = {}
+}
+
+variable "enable_cluster_creator_admin_permissions" {
+ description = "Indicates whether or not to add the cluster creator (the identity used by Terraform) as an administrator via access entry"
+ type = bool
+ default = false
+}
+
################################################################################
# KMS Key
################################################################################
@@ -153,15 +175,15 @@ variable "kms_key_deletion_window_in_days" {
}
variable "enable_kms_key_rotation" {
- description = "Specifies whether key rotation is enabled. Defaults to `true`"
+ description = "Specifies whether key rotation is enabled"
type = bool
default = true
}
variable "kms_key_enable_default_policy" {
- description = "Specifies whether to enable the default key policy. Defaults to `false`"
+ description = "Specifies whether to enable the default key policy"
type = bool
- default = false
+ default = true
}
variable "kms_key_owners" {
@@ -228,6 +250,18 @@ variable "cloudwatch_log_group_kms_key_id" {
default = null
}
+variable "cloudwatch_log_group_class" {
+ description = "Specified the log class of the log group. Possible values are: `STANDARD` or `INFREQUENT_ACCESS`"
+ type = string
+ default = null
+}
+
+variable "cloudwatch_log_group_tags" {
+ description = "A map of additional tags to add to the cloudwatch log group created"
+ type = map(string)
+ default = {}
+}
+
################################################################################
# Cluster Security Group
################################################################################
@@ -342,6 +376,12 @@ variable "node_security_group_tags" {
default = {}
}
+variable "enable_efa_support" {
+ description = "Determines whether to enable Elastic Fabric Adapter (EFA) support"
+ type = bool
+ default = false
+}
+
################################################################################
# IRSA
################################################################################
@@ -358,6 +398,12 @@ variable "openid_connect_audiences" {
default = []
}
+variable "include_oidc_root_ca_thumbprint" {
+ description = "Determines whether to include the root CA thumbprint in the OpenID Connect (OIDC) identity provider's server certificate(s)"
+ type = bool
+ default = true
+}
+
variable "custom_oidc_thumbprints" {
description = "Additional list of server certificate thumbprints for the OpenID Connect (OIDC) identity provider's server certificate(s)"
type = list(string)
@@ -416,14 +462,6 @@ variable "iam_role_additional_policies" {
default = {}
}
-# TODO - hopefully this can be removed once the AWS endpoint is named properly in China
-# https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1904
-variable "cluster_iam_role_dns_suffix" {
- description = "Base DNS domain name for the current partition (e.g., amazonaws.com in AWS Commercial, amazonaws.com.cn in AWS China)"
- type = string
- default = null
-}
-
variable "iam_role_tags" {
description = "A map of additional tags to add to the IAM role created"
type = map(string)
@@ -545,55 +583,3 @@ variable "putin_khuylo" {
type = bool
default = true
}
-
-################################################################################
-# aws-auth configmap
-################################################################################
-
-variable "manage_aws_auth_configmap" {
- description = "Determines whether to manage the aws-auth configmap"
- type = bool
- default = false
-}
-
-variable "create_aws_auth_configmap" {
- description = "Determines whether to create the aws-auth configmap. NOTE - this is only intended for scenarios where the configmap does not exist (i.e. - when using only self-managed node groups). Most users should use `manage_aws_auth_configmap`"
- type = bool
- default = false
-}
-
-variable "aws_auth_node_iam_role_arns_non_windows" {
- description = "List of non-Windows based node IAM role ARNs to add to the aws-auth configmap"
- type = list(string)
- default = []
-}
-
-variable "aws_auth_node_iam_role_arns_windows" {
- description = "List of Windows based node IAM role ARNs to add to the aws-auth configmap"
- type = list(string)
- default = []
-}
-
-variable "aws_auth_fargate_profile_pod_execution_role_arns" {
- description = "List of Fargate profile pod execution role ARNs to add to the aws-auth configmap"
- type = list(string)
- default = []
-}
-
-variable "aws_auth_roles" {
- description = "List of role maps to add to the aws-auth configmap"
- type = list(any)
- default = []
-}
-
-variable "aws_auth_users" {
- description = "List of user maps to add to the aws-auth configmap"
- type = list(any)
- default = []
-}
-
-variable "aws_auth_accounts" {
- description = "List of account maps to add to the aws-auth configmap"
- type = list(any)
- default = []
-}
diff --git a/versions.tf b/versions.tf
index 1dcaa257ea..d0f347a88a 100644
--- a/versions.tf
+++ b/versions.tf
@@ -1,19 +1,15 @@
terraform {
- required_version = ">= 1.2.0"
+ required_version = ">= 1.3.2"
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 4.47"
+ version = ">= 5.40"
}
tls = {
source = "hashicorp/tls"
version = ">= 3.0"
}
- kubernetes = {
- source = "hashicorp/kubernetes"
- version = ">= 2.10"
- }
time = {
source = "hashicorp/time"
version = ">= 0.9"