` version to use for the EKS cluster (i.e.: `1.27`) | `string` | `null` | no |
+| [cluster\_zonal\_shift\_config](#input\_cluster\_zonal\_shift\_config) | Configuration block for the cluster zonal shift | `any` | `{}` | no |
| [control\_plane\_subnet\_ids](#input\_control\_plane\_subnet\_ids) | A list of subnet IDs where the EKS cluster control plane (ENIs) will be provisioned. Used for expanding the pool of subnets used by nodes/node groups without replacing the EKS control plane | `list(string)` | `[]` | no |
| [create](#input\_create) | Controls if resources should be created (affects nearly all resources) | `bool` | `true` | no |
| [create\_cloudwatch\_log\_group](#input\_create\_cloudwatch\_log\_group) | Determines whether a log group is created by this module for the cluster logs. If not, AWS will automatically create one if logging is enabled | `bool` | `true` | no |
@@ -365,7 +364,7 @@ We are grateful to the community for contributing bugfixes and improvements! Ple
| [oidc\_provider\_arn](#output\_oidc\_provider\_arn) | The ARN of the OIDC Provider if `enable_irsa = true` |
| [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
| [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
-
+
## License
diff --git a/docs/UPGRADE-18.0.md b/docs/UPGRADE-18.0.md
index 3f25ca8e43..8b3d0accad 100644
--- a/docs/UPGRADE-18.0.md
+++ b/docs/UPGRADE-18.0.md
@@ -48,7 +48,7 @@ See more information [here](https://github.com/terraform-aws-modules/terraform-a
- Additional changes for the `self-managed-node-group` sub-module over the previous `node_groups` variable include:
- The underlying autoscaling group and launch template have been updated to more closely match that of the [`terraform-aws-autoscaling`](https://github.com/terraform-aws-modules/terraform-aws-autoscaling) module and the features it offers
- The previous iteration used a count over a list of node group definitions which was prone to disruptive updates; this is now replaced with a map/for_each to align with that of the EKS managed node group and Fargate profile behaviors/style
-- The user data configuration supported across the module has been completely revamped. A new `_user_data` internal sub-module has been created to consolidate all user data configuration in one location which provides better support for testability (via the [`examples/user_data`](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/user_data) example). The new sub-module supports nearly all possible combinations including the ability to allow users to provide their own user data template which will be rendered by the module. See the `examples/user_data` example project for the full plethora of example configuration possibilities and more details on the logic of the design can be found in the [`modules/_user_data`](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/_user_data_) directory.
+- The user data configuration supported across the module has been completely revamped. A new `_user_data` internal sub-module has been created to consolidate all user data configuration in one location which provides better support for testability (via the [`tests/user-data`](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/tests/user-data) example). The new sub-module supports nearly all possible combinations including the ability to allow users to provide their own user data template which will be rendered by the module. See the `tests/user-data` example project for the full plethora of example configuration possibilities and more details on the logic of the design can be found in the [`modules/_user_data`](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/_user_data_) directory.
- Resource name changes may cause issues with existing resources. For example, security groups and IAM roles cannot be renamed, they must be recreated. Recreation of these resources may also trigger a recreation of the cluster. To use the legacy (< 18.x) resource naming convention, set `prefix_separator` to "".
- Security group usage has been overhauled to provide only the bare minimum network connectivity required to launch a bare bones cluster. See the [security group documentation section](https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups) for more details. Users upgrading to v18.x will want to review the rules they have in place today versus the rules provisioned by the v18.x module and ensure to make any necessary adjustments for their specific workload.
diff --git a/docs/UPGRADE-20.0.md b/docs/UPGRADE-20.0.md
index 6f53e326c2..c93a031ee9 100644
--- a/docs/UPGRADE-20.0.md
+++ b/docs/UPGRADE-20.0.md
@@ -170,7 +170,7 @@ To give users advanced notice and provide some future direction for this module,
- ]
}
-+ module "eks" {
++ module "eks_aws_auth" {
+ source = "terraform-aws-modules/eks/aws//modules/aws-auth"
+ version = "~> 20.0"
@@ -197,7 +197,7 @@ To give users advanced notice and provide some future direction for this module,
### Karpenter Diff of Before (v19.21) vs After (v20.0)
```diff
- module "eks" {
+ module "eks_karpenter" {
source = "terraform-aws-modules/eks/aws//modules/karpenter"
- version = "~> 19.21"
+ version = "~> 20.0"
diff --git a/docs/compute_resources.md b/docs/compute_resources.md
index 721c29c1ee..6b74c4a507 100644
--- a/docs/compute_resources.md
+++ b/docs/compute_resources.md
@@ -104,7 +104,7 @@ Refer to the [EKS Managed Node Group documentation](https://docs.aws.amazon.com/
}
```
-See the [`examples/eks_managed_node_group/` example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/eks_managed_node_group) for a working example of various configurations.
+See the [`examples/eks-managed-node-group/` example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/eks-managed-node-group) for a working example of various configurations.
### Self Managed Node Groups
@@ -113,7 +113,7 @@ Refer to the [Self Managed Node Group documentation](https://docs.aws.amazon.com
1. The `self-managed-node-group` uses the latest AWS EKS Optimized AMI (Linux) for the given Kubernetes version by default:
```hcl
- cluster_version = "1.27"
+ cluster_version = "1.31"
# This self managed node group will use the latest AWS EKS Optimized AMI for Kubernetes 1.27
self_managed_node_groups = {
@@ -124,7 +124,7 @@ Refer to the [Self Managed Node Group documentation](https://docs.aws.amazon.com
2. To use Bottlerocket, specify the `ami_type` as one of the respective `"BOTTLEROCKET_*" types` and supply a Bottlerocket OS AMI:
```hcl
- cluster_version = "1.27"
+ cluster_version = "1.31"
self_managed_node_groups = {
bottlerocket = {
@@ -134,11 +134,11 @@ Refer to the [Self Managed Node Group documentation](https://docs.aws.amazon.com
}
```
-See the [`examples/self_managed_node_group/` example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/self_managed_node_group) for a working example of various configurations.
+See the [`examples/self-managed-node-group/` example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/self-managed-node-group) for a working example of various configurations.
### Fargate Profiles
-Fargate profiles are straightforward to use and therefore no further details are provided here. See the [`examples/fargate_profile/` example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/fargate_profile) for a working example of various configurations.
+Fargate profiles are straightforward to use and therefore no further details are provided here. See the [`tests/fargate-profile/` tests](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/tests/fargate-profile) for a working example of various configurations.
### Default Configurations
diff --git a/docs/user_data.md b/docs/user_data.md
index 1ddc5d1aa7..3ca2263deb 100644
--- a/docs/user_data.md
+++ b/docs/user_data.md
@@ -1,6 +1,6 @@
# User Data & Bootstrapping
-Users can see the various methods of using and providing user data through the [user data examples](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/user_data) as well more detailed information on the design and possible configurations via the [user data module itself](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/_user_data)
+Users can see the various methods of using and providing user data through the [user data tests](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/tests/user-data) as well more detailed information on the design and possible configurations via the [user data module itself](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/_user_data)
## Summary
diff --git a/examples/eks-managed-node-group/eks-al2.tf b/examples/eks-managed-node-group/eks-al2.tf
index 2dfb5b019f..910fa216b2 100644
--- a/examples/eks-managed-node-group/eks-al2.tf
+++ b/examples/eks-managed-node-group/eks-al2.tf
@@ -3,7 +3,7 @@ module "eks_al2" {
version = "~> 20.0"
cluster_name = "${local.name}-al2"
- cluster_version = "1.30"
+ cluster_version = "1.31"
# EKS Addons
cluster_addons = {
diff --git a/examples/eks-managed-node-group/eks-al2023.tf b/examples/eks-managed-node-group/eks-al2023.tf
index 1b112d23a2..349da821f9 100644
--- a/examples/eks-managed-node-group/eks-al2023.tf
+++ b/examples/eks-managed-node-group/eks-al2023.tf
@@ -3,7 +3,7 @@ module "eks_al2023" {
version = "~> 20.0"
cluster_name = "${local.name}-al2023"
- cluster_version = "1.30"
+ cluster_version = "1.31"
# EKS Addons
cluster_addons = {
diff --git a/examples/eks-managed-node-group/eks-bottlerocket.tf b/examples/eks-managed-node-group/eks-bottlerocket.tf
index 44efa593c0..01a6878814 100644
--- a/examples/eks-managed-node-group/eks-bottlerocket.tf
+++ b/examples/eks-managed-node-group/eks-bottlerocket.tf
@@ -3,7 +3,7 @@ module "eks_bottlerocket" {
version = "~> 20.0"
cluster_name = "${local.name}-bottlerocket"
- cluster_version = "1.30"
+ cluster_version = "1.31"
# EKS Addons
cluster_addons = {
diff --git a/examples/eks-managed-node-group/versions.tf b/examples/eks-managed-node-group/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/examples/eks-managed-node-group/versions.tf
+++ b/examples/eks-managed-node-group/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/examples/karpenter/README.md b/examples/karpenter/README.md
index 62291cdc07..15d51bcdb9 100644
--- a/examples/karpenter/README.md
+++ b/examples/karpenter/README.md
@@ -83,13 +83,13 @@ terraform destroy --auto-approve
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
| [helm](#requirement\_helm) | >= 2.7 |
| [kubectl](#requirement\_kubectl) | >= 2.0 |
@@ -97,8 +97,8 @@ Note that this example may create resources which cost money. Run `terraform des
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
-| [aws.virginia](#provider\_aws.virginia) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
+| [aws.virginia](#provider\_aws.virginia) | >= 5.75 |
| [helm](#provider\_helm) | >= 2.7 |
| [kubectl](#provider\_kubectl) | >= 2.0 |
@@ -176,4 +176,4 @@ No inputs.
| [oidc\_provider\_arn](#output\_oidc\_provider\_arn) | The ARN of the OIDC Provider if `enable_irsa = true` |
| [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
| [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
-
+
diff --git a/examples/karpenter/main.tf b/examples/karpenter/main.tf
index 72ceff275a..49321c977e 100644
--- a/examples/karpenter/main.tf
+++ b/examples/karpenter/main.tf
@@ -62,7 +62,7 @@ module "eks" {
source = "../.."
cluster_name = local.name
- cluster_version = "1.30"
+ cluster_version = "1.31"
# Gives Terraform identity admin access to cluster which will
# allow deploying resources (Karpenter) into the cluster
@@ -157,7 +157,7 @@ resource "helm_release" "karpenter" {
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
chart = "karpenter"
- version = "1.0.0"
+ version = "1.0.6"
wait = false
values = [
diff --git a/examples/karpenter/versions.tf b/examples/karpenter/versions.tf
index 8a0624ba0d..5caab8394a 100644
--- a/examples/karpenter/versions.tf
+++ b/examples/karpenter/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
helm = {
source = "hashicorp/helm"
diff --git a/examples/outposts/main.tf b/examples/outposts/main.tf
deleted file mode 100644
index 3d04ea3223..0000000000
--- a/examples/outposts/main.tf
+++ /dev/null
@@ -1,141 +0,0 @@
-provider "aws" {
- region = var.region
-}
-
-locals {
- name = "ex-${basename(path.cwd)}"
- cluster_version = "1.30"
-
- outpost_arn = element(tolist(data.aws_outposts_outposts.this.arns), 0)
- instance_type = element(tolist(data.aws_outposts_outpost_instance_types.this.instance_types), 0)
-
- tags = {
- Example = local.name
- GithubRepo = "terraform-aws-eks"
- GithubOrg = "terraform-aws-modules"
- }
-}
-
-################################################################################
-# EKS Module
-################################################################################
-
-module "eks" {
- source = "../.."
-
- cluster_name = local.name
- cluster_version = local.cluster_version
-
- cluster_endpoint_public_access = false # Not available on Outpost
- cluster_endpoint_private_access = true
-
- # Gives Terraform identity admin access to cluster which will
- # allow deploying resources (EBS storage class) into the cluster
- enable_cluster_creator_admin_permissions = true
-
- vpc_id = data.aws_vpc.this.id
- subnet_ids = data.aws_subnets.this.ids
-
- outpost_config = {
- control_plane_instance_type = local.instance_type
- outpost_arns = [local.outpost_arn]
- }
-
- # Extend cluster security group rules
- cluster_security_group_additional_rules = {
- ingress_vpc_https = {
- description = "Remote host to control plane"
- protocol = "tcp"
- from_port = 443
- to_port = 443
- type = "ingress"
- cidr_blocks = [data.aws_vpc.this.cidr_block]
- }
- }
-
- self_managed_node_group_defaults = {
- attach_cluster_primary_security_group = true
-
- iam_role_additional_policies = {
- AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
- }
- }
-
- self_managed_node_groups = {
- outpost = {
- name = local.name
-
- min_size = 2
- max_size = 5
- desired_size = 3
- instance_type = local.instance_type
-
- # Additional information is required to join local clusters to EKS
- bootstrap_extra_args = <<-EOT
- --enable-local-outpost true --cluster-id ${module.eks.cluster_id} --container-runtime containerd
- EOT
- }
- }
-
- tags = local.tags
-}
-
-resource "kubernetes_storage_class_v1" "this" {
- metadata {
- name = "ebs-sc"
- annotations = {
- "storageclass.kubernetes.io/is-default-class" = "true"
- }
- }
-
- storage_provisioner = "ebs.csi.aws.com"
- volume_binding_mode = "WaitForFirstConsumer"
- allow_volume_expansion = true
-
- parameters = {
- type = "gp2"
- encrypted = "true"
- }
-}
-
-################################################################################
-# Supporting Resources
-################################################################################
-
-data "aws_outposts_outposts" "this" {}
-
-data "aws_outposts_outpost_instance_types" "this" {
- arn = local.outpost_arn
-}
-
-# This just grabs the first Outpost and returns its subnets
-data "aws_subnets" "lookup" {
- filter {
- name = "outpost-arn"
- values = [local.outpost_arn]
- }
-}
-
-# This grabs a single subnet to reverse lookup those that belong to same VPC
-# This is whats used for the cluster
-data "aws_subnet" "this" {
- id = element(tolist(data.aws_subnets.lookup.ids), 0)
-}
-
-# These are subnets for the Outpost and restricted to the same VPC
-# This is whats used for the cluster
-data "aws_subnets" "this" {
- filter {
- name = "outpost-arn"
- values = [local.outpost_arn]
- }
-
- filter {
- name = "vpc-id"
- values = [data.aws_subnet.this.vpc_id]
- }
-}
-
-data "aws_vpc" "this" {
- id = data.aws_subnet.this.vpc_id
-}
diff --git a/examples/outposts/prerequisites/main.tf b/examples/outposts/prerequisites/main.tf
deleted file mode 100644
index 1a1dd18a4b..0000000000
--- a/examples/outposts/prerequisites/main.tf
+++ /dev/null
@@ -1,149 +0,0 @@
-provider "aws" {
- region = var.region
-}
-
-locals {
- name = "ex-${basename(path.cwd)}"
-
- terraform_version = "1.3.6"
-
- outpost_arn = element(tolist(data.aws_outposts_outposts.this.arns), 0)
- instance_type = element(tolist(data.aws_outposts_outpost_instance_types.this.instance_types), 0)
-
- tags = {
- Example = local.name
- GithubRepo = "terraform-aws-eks"
- GithubOrg = "terraform-aws-modules"
- }
-}
-
-################################################################################
-# Pre-Requisites
-################################################################################
-
-module "ssm_bastion_ec2" {
- source = "terraform-aws-modules/ec2-instance/aws"
- version = "~> 5.5"
-
- name = "${local.name}-bastion"
-
- create_iam_instance_profile = true
- iam_role_policies = {
- AdministratorAccess = "arn:aws:iam::aws:policy/AdministratorAccess"
- }
-
- instance_type = local.instance_type
-
- user_data = <<-EOT
- #!/bin/bash
-
- # Add ssm-user since it won't exist until first login
- adduser -m ssm-user
- tee /etc/sudoers.d/ssm-agent-users <<'EOF'
- # User rules for ssm-user
- ssm-user ALL=(ALL) NOPASSWD:ALL
- EOF
- chmod 440 /etc/sudoers.d/ssm-agent-users
-
- cd /home/ssm-user
-
- # Install git to clone repo
- yum install git -y
-
- # Install Terraform
- curl -sSO https://releases.hashicorp.com/terraform/${local.terraform_version}/terraform_${local.terraform_version}_linux_amd64.zip
- sudo unzip -qq terraform_${local.terraform_version}_linux_amd64.zip terraform -d /usr/bin/
- rm terraform_${local.terraform_version}_linux_amd64.zip 2> /dev/null
-
- # Install kubectl
- curl -LO https://dl.k8s.io/release/v1.30.0/bin/linux/amd64/kubectl
- install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
-
- # Remove default awscli which is v1 - we want latest v2
- yum remove awscli -y
- curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
- unzip -qq awscliv2.zip
- ./aws/install
-
- # Clone repo
- git clone https://github.com/terraform-aws-modules/terraform-aws-eks.git \
- && cd /home/ssm-user/terraform-aws-eks
-
- chown -R ssm-user:ssm-user /home/ssm-user/
- EOT
-
- vpc_security_group_ids = [module.bastion_security_group.security_group_id]
- subnet_id = element(data.aws_subnets.this.ids, 0)
-
- tags = local.tags
-}
-
-module "bastion_security_group" {
- source = "terraform-aws-modules/security-group/aws"
- version = "~> 5.0"
-
- name = "${local.name}-bastion"
- description = "Security group to allow provisioning ${local.name} EKS local cluster on Outposts"
- vpc_id = data.aws_vpc.this.id
-
- ingress_with_cidr_blocks = [
- {
- from_port = 443
- to_port = 443
- protocol = "tcp"
- cidr_blocks = data.aws_vpc.this.cidr_block
- },
- ]
- egress_with_cidr_blocks = [
- {
- from_port = 0
- to_port = 0
- protocol = "-1"
- cidr_blocks = "0.0.0.0/0"
- },
- ]
-
- tags = local.tags
-}
-
-################################################################################
-# Supporting Resources
-################################################################################
-
-data "aws_outposts_outposts" "this" {}
-
-data "aws_outposts_outpost_instance_types" "this" {
- arn = local.outpost_arn
-}
-
-# This just grabs the first Outpost and returns its subnets
-data "aws_subnets" "lookup" {
- filter {
- name = "outpost-arn"
- values = [local.outpost_arn]
- }
-}
-
-# This grabs a single subnet to reverse lookup those that belong to same VPC
-# This is whats used for the cluster
-data "aws_subnet" "this" {
- id = element(tolist(data.aws_subnets.lookup.ids), 0)
-}
-
-# These are subnets for the Outpost and restricted to the same VPC
-# This is whats used for the cluster
-data "aws_subnets" "this" {
- filter {
- name = "outpost-arn"
- values = [local.outpost_arn]
- }
-
- filter {
- name = "vpc-id"
- values = [data.aws_subnet.this.vpc_id]
- }
-}
-
-data "aws_vpc" "this" {
- id = data.aws_subnet.this.vpc_id
-}
diff --git a/examples/outposts/prerequisites/outputs.tf b/examples/outposts/prerequisites/outputs.tf
deleted file mode 100644
index f2ff81ab70..0000000000
--- a/examples/outposts/prerequisites/outputs.tf
+++ /dev/null
@@ -1,4 +0,0 @@
-output "ssm_start_session" {
- description = "SSM start session command to connect to remote host created"
- value = "aws ssm start-session --region ${var.region} --target ${module.ssm_bastion_ec2.id}"
-}
diff --git a/examples/outposts/prerequisites/variables.tf b/examples/outposts/prerequisites/variables.tf
deleted file mode 100644
index 47945c8501..0000000000
--- a/examples/outposts/prerequisites/variables.tf
+++ /dev/null
@@ -1,5 +0,0 @@
-variable "region" {
- description = "The AWS region to deploy into (e.g. us-east-1)"
- type = string
- default = "us-west-2"
-}
diff --git a/examples/outposts/variables.tf b/examples/outposts/variables.tf
deleted file mode 100644
index 47945c8501..0000000000
--- a/examples/outposts/variables.tf
+++ /dev/null
@@ -1,5 +0,0 @@
-variable "region" {
- description = "The AWS region to deploy into (e.g. us-east-1)"
- type = string
- default = "us-west-2"
-}
diff --git a/examples/outposts/versions.tf b/examples/outposts/versions.tf
deleted file mode 100644
index 9836b3a468..0000000000
--- a/examples/outposts/versions.tf
+++ /dev/null
@@ -1,14 +0,0 @@
-terraform {
- required_version = ">= 1.3.2"
-
- required_providers {
- aws = {
- source = "hashicorp/aws"
- version = ">= 5.61"
- }
- kubernetes = {
- source = "hashicorp/kubernetes"
- version = ">= 2.20"
- }
- }
-}
diff --git a/examples/self-managed-node-group/eks-al2.tf b/examples/self-managed-node-group/eks-al2.tf
index be5c65ade5..e015982733 100644
--- a/examples/self-managed-node-group/eks-al2.tf
+++ b/examples/self-managed-node-group/eks-al2.tf
@@ -3,7 +3,7 @@ module "eks_al2" {
version = "~> 20.0"
cluster_name = "${local.name}-al2"
- cluster_version = "1.30"
+ cluster_version = "1.31"
# EKS Addons
cluster_addons = {
diff --git a/examples/self-managed-node-group/eks-al2023.tf b/examples/self-managed-node-group/eks-al2023.tf
index 701560527f..4d7c5dc7e1 100644
--- a/examples/self-managed-node-group/eks-al2023.tf
+++ b/examples/self-managed-node-group/eks-al2023.tf
@@ -3,7 +3,7 @@ module "eks_al2023" {
version = "~> 20.0"
cluster_name = "${local.name}-al2023"
- cluster_version = "1.30"
+ cluster_version = "1.31"
# EKS Addons
cluster_addons = {
diff --git a/examples/self-managed-node-group/eks-bottlerocket.tf b/examples/self-managed-node-group/eks-bottlerocket.tf
index 2afb079472..ff2fb2cf3d 100644
--- a/examples/self-managed-node-group/eks-bottlerocket.tf
+++ b/examples/self-managed-node-group/eks-bottlerocket.tf
@@ -3,7 +3,7 @@ module "eks_bottlerocket" {
version = "~> 20.0"
cluster_name = "${local.name}-bottlerocket"
- cluster_version = "1.30"
+ cluster_version = "1.31"
# EKS Addons
cluster_addons = {
diff --git a/examples/self-managed-node-group/versions.tf b/examples/self-managed-node-group/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/examples/self-managed-node-group/versions.tf
+++ b/examples/self-managed-node-group/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/main.tf b/main.tf
index 1846a1da05..50f4323c24 100644
--- a/main.tf
+++ b/main.tf
@@ -92,6 +92,14 @@ resource "aws_eks_cluster" "this" {
}
}
+ dynamic "zonal_shift_config" {
+ for_each = length(var.cluster_zonal_shift_config) > 0 ? [var.cluster_zonal_shift_config] : []
+
+ content {
+ enabled = try(zonal_shift_config.value.enabled, null)
+ }
+ }
+
tags = merge(
{ terraform-aws-modules = "eks" },
var.tags,
@@ -200,7 +208,7 @@ locals {
resource "aws_eks_access_entry" "this" {
for_each = { for k, v in local.merged_access_entries : k => v if local.create }
- cluster_name = aws_eks_cluster.this[0].name
+ cluster_name = aws_eks_cluster.this[0].id
kubernetes_groups = try(each.value.kubernetes_groups, null)
principal_arn = each.value.principal_arn
type = try(each.value.type, "STANDARD")
@@ -217,7 +225,7 @@ resource "aws_eks_access_policy_association" "this" {
type = each.value.association_access_scope_type
}
- cluster_name = aws_eks_cluster.this[0].name
+ cluster_name = aws_eks_cluster.this[0].id
policy_arn = each.value.association_policy_arn
principal_arn = each.value.principal_arn
@@ -411,29 +419,6 @@ resource "aws_iam_role" "this" {
permissions_boundary = var.iam_role_permissions_boundary
force_detach_policies = true
- # https://github.com/terraform-aws-modules/terraform-aws-eks/issues/920
- # Resources running on the cluster are still generating logs when destroying the module resources
- # which results in the log group being re-created even after Terraform destroys it. Removing the
- # ability for the cluster role to create the log group prevents this log group from being re-created
- # outside of Terraform due to services still generating logs during destroy process
- dynamic "inline_policy" {
- for_each = var.create_cloudwatch_log_group ? [1] : []
- content {
- name = local.iam_role_name
-
- policy = jsonencode({
- Version = "2012-10-17"
- Statement = [
- {
- Action = ["logs:CreateLogGroup"]
- Effect = "Deny"
- Resource = "*"
- },
- ]
- })
- }
- }
-
tags = merge(var.tags, var.iam_role_tags)
}
@@ -496,25 +481,42 @@ resource "aws_iam_policy" "cluster_encryption" {
# EKS Addons
################################################################################
+locals {
+ # TODO - Set to `NONE` on next breaking change when default addons are disabled
+ resolve_conflicts_on_create_default = coalesce(var.bootstrap_self_managed_addons, true) ? "OVERWRITE" : "NONE"
+}
+
data "aws_eks_addon_version" "this" {
for_each = { for k, v in var.cluster_addons : k => v if local.create && !local.create_outposts_local_cluster }
addon_name = try(each.value.name, each.key)
kubernetes_version = coalesce(var.cluster_version, aws_eks_cluster.this[0].version)
- most_recent = try(each.value.most_recent, null)
+ # TODO - Set default fallback to `true` on next breaking change
+ most_recent = try(each.value.most_recent, null)
}
resource "aws_eks_addon" "this" {
# Not supported on outposts
for_each = { for k, v in var.cluster_addons : k => v if !try(v.before_compute, false) && local.create && !local.create_outposts_local_cluster }
- cluster_name = aws_eks_cluster.this[0].name
+ cluster_name = aws_eks_cluster.this[0].id
addon_name = try(each.value.name, each.key)
- addon_version = coalesce(try(each.value.addon_version, null), data.aws_eks_addon_version.this[each.key].version)
- configuration_values = try(each.value.configuration_values, null)
- preserve = try(each.value.preserve, true)
- resolve_conflicts_on_create = try(each.value.resolve_conflicts_on_create, "OVERWRITE")
+ addon_version = coalesce(try(each.value.addon_version, null), data.aws_eks_addon_version.this[each.key].version)
+ configuration_values = try(each.value.configuration_values, null)
+
+ dynamic "pod_identity_association" {
+ for_each = try(each.value.pod_identity_association, [])
+
+ content {
+ role_arn = pod_identity_association.value.role_arn
+ service_account = pod_identity_association.value.service_account
+ }
+ }
+
+ preserve = try(each.value.preserve, true)
+ # TODO - Set to `NONE` on next breaking change when default addons are disabled
+ resolve_conflicts_on_create = try(each.value.resolve_conflicts_on_create, local.resolve_conflicts_on_create_default)
resolve_conflicts_on_update = try(each.value.resolve_conflicts_on_update, "OVERWRITE")
service_account_role_arn = try(each.value.service_account_role_arn, null)
@@ -537,13 +539,24 @@ resource "aws_eks_addon" "before_compute" {
# Not supported on outposts
for_each = { for k, v in var.cluster_addons : k => v if try(v.before_compute, false) && local.create && !local.create_outposts_local_cluster }
- cluster_name = aws_eks_cluster.this[0].name
+ cluster_name = aws_eks_cluster.this[0].id
addon_name = try(each.value.name, each.key)
- addon_version = coalesce(try(each.value.addon_version, null), data.aws_eks_addon_version.this[each.key].version)
- configuration_values = try(each.value.configuration_values, null)
- preserve = try(each.value.preserve, true)
- resolve_conflicts_on_create = try(each.value.resolve_conflicts_on_create, "OVERWRITE")
+ addon_version = coalesce(try(each.value.addon_version, null), data.aws_eks_addon_version.this[each.key].version)
+ configuration_values = try(each.value.configuration_values, null)
+
+ dynamic "pod_identity_association" {
+ for_each = try(each.value.pod_identity_association, [])
+
+ content {
+ role_arn = pod_identity_association.value.role_arn
+ service_account = pod_identity_association.value.service_account
+ }
+ }
+
+ preserve = try(each.value.preserve, true)
+ # TODO - Set to `NONE` on next breaking change when default addons are disabled
+ resolve_conflicts_on_create = try(each.value.resolve_conflicts_on_create, local.resolve_conflicts_on_create_default)
resolve_conflicts_on_update = try(each.value.resolve_conflicts_on_update, "OVERWRITE")
service_account_role_arn = try(each.value.service_account_role_arn, null)
@@ -565,6 +578,7 @@ locals {
# Maintain current behavior for <= 1.29, remove default for >= 1.30
# `null` will return the latest Kubernetes version from the EKS API, which at time of writing is 1.30
# https://github.com/kubernetes/kubernetes/pull/123561
+ # TODO - remove on next breaking change in conjunction with issuer URL change below
idpc_backwards_compat_version = contains(["1.21", "1.22", "1.23", "1.24", "1.25", "1.26", "1.27", "1.28", "1.29"], coalesce(var.cluster_version, "1.30"))
idpc_issuer_url = local.idpc_backwards_compat_version ? try(aws_eks_cluster.this[0].identity[0].oidc[0].issuer, null) : null
}
@@ -572,7 +586,7 @@ locals {
resource "aws_eks_identity_provider_config" "this" {
for_each = { for k, v in var.cluster_identity_providers : k => v if local.create && !local.create_outposts_local_cluster }
- cluster_name = aws_eks_cluster.this[0].name
+ cluster_name = aws_eks_cluster.this[0].id
oidc {
client_id = each.value.client_id
diff --git a/modules/_user_data/README.md b/modules/_user_data/README.md
index e5207d9443..185335b16a 100644
--- a/modules/_user_data/README.md
+++ b/modules/_user_data/README.md
@@ -2,9 +2,9 @@
Configuration in this directory renders the appropriate user data for the given inputs. See [`docs/user_data.md`](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/user_data.md) for more info.
-See [`examples/user_data/`](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/user_data) for various examples using this module.
+See [`tests/user-data/`](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/tests/user-data) for various tests cases using this module.
-
+
## Requirements
| Name | Version |
@@ -39,8 +39,8 @@ No modules.
| [additional\_cluster\_dns\_ips](#input\_additional\_cluster\_dns\_ips) | Additional DNS IP addresses to use for the cluster. Only used when `ami_type` = `BOTTLEROCKET_*` | `list(string)` | `[]` | no |
| [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html#AmazonEKS-Type-Nodegroup-amiType) for valid values | `string` | `null` | no |
| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
-| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
-| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
| [cluster\_auth\_base64](#input\_cluster\_auth\_base64) | Base64 encoded CA of associated EKS cluster | `string` | `""` | no |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Endpoint of associated EKS cluster | `string` | `""` | no |
| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
@@ -61,4 +61,4 @@ No modules.
|------|-------------|
| [platform](#output\_platform) | [DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023, or `windows |
| [user\_data](#output\_user\_data) | Base64 encoded user data rendered for the provided inputs |
-
+
diff --git a/modules/_user_data/main.tf b/modules/_user_data/main.tf
index 79b8cbae5d..e66cd3d466 100644
--- a/modules/_user_data/main.tf
+++ b/modules/_user_data/main.tf
@@ -30,6 +30,8 @@ locals {
WINDOWS_FULL_2022_x86_64 = "windows"
AL2023_x86_64_STANDARD = "al2023"
AL2023_ARM_64_STANDARD = "al2023"
+ AL2023_x86_64_NEURON = "al2023"
+ AL2023_x86_64_NVIDIA = "al2023"
}
# Try to use `ami_type` first, but fall back to current, default behavior
# TODO - will be removed in v21.0
diff --git a/modules/aws-auth/README.md b/modules/aws-auth/README.md
index 5ba490b7f1..d4b97f14ca 100644
--- a/modules/aws-auth/README.md
+++ b/modules/aws-auth/README.md
@@ -39,7 +39,7 @@ module "eks" {
## Usage
-
+
## Requirements
| Name | Version |
@@ -78,4 +78,4 @@ No modules.
## Outputs
No outputs.
-
+
diff --git a/modules/eks-managed-node-group/README.md b/modules/eks-managed-node-group/README.md
index 96b6c4f290..23df973444 100644
--- a/modules/eks-managed-node-group/README.md
+++ b/modules/eks-managed-node-group/README.md
@@ -10,7 +10,7 @@ module "eks_managed_node_group" {
name = "separate-eks-mng"
cluster_name = "my-cluster"
- cluster_version = "1.27"
+ cluster_version = "1.31"
subnet_ids = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]
@@ -58,19 +58,19 @@ module "eks_managed_node_group" {
}
```
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
@@ -110,8 +110,8 @@ module "eks_managed_node_group" {
| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
| [capacity\_reservation\_specification](#input\_capacity\_reservation\_specification) | Targeting for EC2 capacity reservations | `any` | `{}` | no |
| [capacity\_type](#input\_capacity\_type) | Type of capacity associated with the EKS Node Group. Valid values: `ON_DEMAND`, `SPOT` | `string` | `"ON_DEMAND"` | no |
-| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
-| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
| [cluster\_auth\_base64](#input\_cluster\_auth\_base64) | Base64 encoded CA of associated EKS cluster | `string` | `""` | no |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Endpoint of associated EKS cluster | `string` | `""` | no |
| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
@@ -132,9 +132,11 @@ module "eks_managed_node_group" {
| [disable\_api\_termination](#input\_disable\_api\_termination) | If true, enables EC2 instance termination protection | `bool` | `null` | no |
| [disk\_size](#input\_disk\_size) | Disk size in GiB for nodes. Defaults to `20`. Only valid when `use_custom_launch_template` = `false` | `number` | `null` | no |
| [ebs\_optimized](#input\_ebs\_optimized) | If true, the launched EC2 instance(s) will be EBS-optimized | `bool` | `null` | no |
+| [efa\_indices](#input\_efa\_indices) | The indices of the network interfaces that should be EFA-enabled. Only valid when `enable_efa_support` = `true` | `list(number)` | [
0
]
| no |
| [elastic\_gpu\_specifications](#input\_elastic\_gpu\_specifications) | The elastic GPU to attach to the instance | `any` | `{}` | no |
| [elastic\_inference\_accelerator](#input\_elastic\_inference\_accelerator) | Configuration block containing an Elastic Inference Accelerator to attach to the instance | `map(string)` | `{}` | no |
| [enable\_bootstrap\_user\_data](#input\_enable\_bootstrap\_user\_data) | Determines whether the bootstrap configurations are populated within the user data template. Only valid when using a custom AMI via `ami_id` | `bool` | `false` | no |
+| [enable\_efa\_only](#input\_enable\_efa\_only) | Determines whether to enable EFA (`false`, default) or EFA and EFA-only (`true`) network interfaces. Note: requires vpc-cni version `v1.18.4` or later | `bool` | `false` | no |
| [enable\_efa\_support](#input\_enable\_efa\_support) | Determines whether to enable Elastic Fabric Adapter (EFA) support | `bool` | `false` | no |
| [enable\_monitoring](#input\_enable\_monitoring) | Enables/disables detailed monitoring | `bool` | `true` | no |
| [enclave\_options](#input\_enclave\_options) | Enable Nitro Enclaves on launched instances | `map(string)` | `{}` | no |
@@ -164,7 +166,7 @@ module "eks_managed_node_group" {
| [license\_specifications](#input\_license\_specifications) | A map of license specifications to associate with | `any` | `{}` | no |
| [maintenance\_options](#input\_maintenance\_options) | The maintenance options for the instance | `any` | `{}` | no |
| [max\_size](#input\_max\_size) | Maximum number of instances/nodes | `number` | `3` | no |
-| [metadata\_options](#input\_metadata\_options) | Customize the metadata options for the instance | `map(string)` | {
"http_endpoint": "enabled",
"http_put_response_hop_limit": 2,
"http_tokens": "required"
} | no |
+| [metadata\_options](#input\_metadata\_options) | Customize the metadata options for the instance | `map(string)` | {
"http_endpoint": "enabled",
"http_put_response_hop_limit": 2,
"http_tokens": "required"
} | no |
| [min\_size](#input\_min\_size) | Minimum number of instances/nodes | `number` | `0` | no |
| [name](#input\_name) | Name of the EKS managed node group | `string` | `""` | no |
| [network\_interfaces](#input\_network\_interfaces) | Customize network interfaces to be attached at instance boot time | `list(any)` | `[]` | no |
@@ -179,11 +181,11 @@ module "eks_managed_node_group" {
| [remote\_access](#input\_remote\_access) | Configuration block with remote access settings. Only valid when `use_custom_launch_template` = `false` | `any` | `{}` | no |
| [schedules](#input\_schedules) | Map of autoscaling group schedule to create | `map(any)` | `{}` | no |
| [subnet\_ids](#input\_subnet\_ids) | Identifiers of EC2 Subnets to associate with the EKS Node Group. These subnets must have the following resource tag: `kubernetes.io/cluster/CLUSTER_NAME` | `list(string)` | `null` | no |
-| [tag\_specifications](#input\_tag\_specifications) | The tags to apply to the resources during launch | `list(string)` | [
"instance",
"volume",
"network-interface"
]
| no |
+| [tag\_specifications](#input\_tag\_specifications) | The tags to apply to the resources during launch | `list(string)` | [
"instance",
"volume",
"network-interface"
]
| no |
| [tags](#input\_tags) | A map of tags to add to all resources | `map(string)` | `{}` | no |
| [taints](#input\_taints) | The Kubernetes taints to be applied to the nodes in the node group. Maximum of 50 taints per node group | `any` | `{}` | no |
| [timeouts](#input\_timeouts) | Create, update, and delete timeout configurations for the node group | `map(string)` | `{}` | no |
-| [update\_config](#input\_update\_config) | Configuration block of settings for max unavailable resources during node group updates | `map(string)` | {
"max_unavailable_percentage": 33
} | no |
+| [update\_config](#input\_update\_config) | Configuration block of settings for max unavailable resources during node group updates | `map(string)` | {
"max_unavailable_percentage": 33
} | no |
| [update\_launch\_template\_default\_version](#input\_update\_launch\_template\_default\_version) | Whether to update the launch templates default version on each update. Conflicts with `launch_template_default_version` | `bool` | `true` | no |
| [use\_custom\_launch\_template](#input\_use\_custom\_launch\_template) | Determines whether to use a custom launch template or not. If set to `false`, EKS will use its own default launch template | `bool` | `true` | no |
| [use\_latest\_ami\_release\_version](#input\_use\_latest\_ami\_release\_version) | Determines whether to use the latest AMI release version for the given `ami_type` (except for `CUSTOM`). Note: `ami_type` and `cluster_version` must be supplied in order to enable this feature | `bool` | `false` | no |
@@ -211,4 +213,4 @@ module "eks_managed_node_group" {
| [node\_group\_status](#output\_node\_group\_status) | Status of the EKS Node Group |
| [node\_group\_taints](#output\_node\_group\_taints) | List of objects containing information about taints applied to the node group |
| [platform](#output\_platform) | [DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows` |
-
+
diff --git a/modules/eks-managed-node-group/main.tf b/modules/eks-managed-node-group/main.tf
index 80c012fa55..a1a3d736d2 100644
--- a/modules/eks-managed-node-group/main.tf
+++ b/modules/eks-managed-node-group/main.tf
@@ -44,13 +44,14 @@ locals {
efa_instance_type = try(element(var.instance_types, 0), "")
num_network_cards = try(data.aws_ec2_instance_type.this[0].maximum_network_cards, 0)
+ # Primary network interface must be EFA, remaining can be EFA or EFA-only
efa_network_interfaces = [
for i in range(local.num_network_cards) : {
associate_public_ip_address = false
delete_on_termination = true
device_index = i == 0 ? 0 : 1
network_card_index = i
- interface_type = "efa"
+ interface_type = var.enable_efa_only ? contains(concat([0], var.efa_indices), i) ? "efa" : "efa-only" : "efa"
}
]
@@ -362,6 +363,8 @@ locals {
WINDOWS_FULL_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-${local.ssm_cluster_version}"
AL2023_x86_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/standard/recommended/release_version"
AL2023_ARM_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/arm64/standard/recommended/release_version"
+ AL2023_x86_64_NEURON = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/neuron/recommended/release_version"
+ AL2023_x86_64_NVIDIA = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/nvidia/recommended/release_version"
}
# The Windows SSM params currently do not have a release version, so we have to get the full output JSON blob and parse out the release version
diff --git a/modules/eks-managed-node-group/variables.tf b/modules/eks-managed-node-group/variables.tf
index bb60b85665..e0ee435785 100644
--- a/modules/eks-managed-node-group/variables.tf
+++ b/modules/eks-managed-node-group/variables.tf
@@ -285,6 +285,19 @@ variable "enable_efa_support" {
default = false
}
+# TODO - make this true by default at next breaking change (remove variable, only pass indices)
+variable "enable_efa_only" {
+ description = "Determines whether to enable EFA (`false`, default) or EFA and EFA-only (`true`) network interfaces. Note: requires vpc-cni version `v1.18.4` or later"
+ type = bool
+ default = false
+}
+
+variable "efa_indices" {
+ description = "The indices of the network interfaces that should be EFA-enabled. Only valid when `enable_efa_support` = `true`"
+ type = list(number)
+ default = [0]
+}
+
variable "network_interfaces" {
description = "Customize network interfaces to be attached at instance boot time"
type = list(any)
diff --git a/modules/eks-managed-node-group/versions.tf b/modules/eks-managed-node-group/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/modules/eks-managed-node-group/versions.tf
+++ b/modules/eks-managed-node-group/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/modules/fargate-profile/README.md b/modules/fargate-profile/README.md
index b1ee9e0ade..a7b12553ff 100644
--- a/modules/fargate-profile/README.md
+++ b/modules/fargate-profile/README.md
@@ -23,19 +23,19 @@ module "fargate_profile" {
}
```
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
@@ -92,4 +92,4 @@ No modules.
| [iam\_role\_arn](#output\_iam\_role\_arn) | The Amazon Resource Name (ARN) specifying the IAM role |
| [iam\_role\_name](#output\_iam\_role\_name) | The name of the IAM role |
| [iam\_role\_unique\_id](#output\_iam\_role\_unique\_id) | Stable and unique string identifying the IAM role |
-
+
diff --git a/modules/fargate-profile/versions.tf b/modules/fargate-profile/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/modules/fargate-profile/versions.tf
+++ b/modules/fargate-profile/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/modules/karpenter/README.md b/modules/karpenter/README.md
index 6810d0fa0c..ef2be2099c 100644
--- a/modules/karpenter/README.md
+++ b/modules/karpenter/README.md
@@ -80,19 +80,19 @@ module "karpenter" {
}
```
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
@@ -158,7 +158,7 @@ No modules.
| [iam\_role\_tags](#input\_iam\_role\_tags) | A map of additional tags to add the the IAM role | `map(any)` | `{}` | no |
| [iam\_role\_use\_name\_prefix](#input\_iam\_role\_use\_name\_prefix) | Determines whether the name of the IAM role (`iam_role_name`) is used as a prefix | `bool` | `true` | no |
| [irsa\_assume\_role\_condition\_test](#input\_irsa\_assume\_role\_condition\_test) | Name of the [IAM condition operator](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) to evaluate when assuming the role | `string` | `"StringEquals"` | no |
-| [irsa\_namespace\_service\_accounts](#input\_irsa\_namespace\_service\_accounts) | List of `namespace:serviceaccount`pairs to use in trust policy for IAM role for service accounts | `list(string)` | [
"karpenter:karpenter"
]
| no |
+| [irsa\_namespace\_service\_accounts](#input\_irsa\_namespace\_service\_accounts) | List of `namespace:serviceaccount`pairs to use in trust policy for IAM role for service accounts | `list(string)` | [
"karpenter:karpenter"
]
| no |
| [irsa\_oidc\_provider\_arn](#input\_irsa\_oidc\_provider\_arn) | OIDC provider arn used in trust policy for IAM role for service accounts | `string` | `""` | no |
| [namespace](#input\_namespace) | Namespace to associate with the Karpenter Pod Identity | `string` | `"kube-system"` | no |
| [node\_iam\_role\_additional\_policies](#input\_node\_iam\_role\_additional\_policies) | Additional policies to be added to the IAM role | `map(string)` | `{}` | no |
@@ -200,4 +200,4 @@ No modules.
| [queue\_name](#output\_queue\_name) | The name of the created Amazon SQS queue |
| [queue\_url](#output\_queue\_url) | The URL for the created Amazon SQS queue |
| [service\_account](#output\_service\_account) | Service Account associated with the Karpenter Pod Identity |
-
+
diff --git a/modules/karpenter/main.tf b/modules/karpenter/main.tf
index fc4a5dca57..d03dfa49f8 100644
--- a/modules/karpenter/main.tf
+++ b/modules/karpenter/main.tf
@@ -4,6 +4,7 @@ data "aws_caller_identity" "current" {}
locals {
account_id = data.aws_caller_identity.current.account_id
+ dns_suffix = data.aws_partition.current.dns_suffix
partition = data.aws_partition.current.partition
region = data.aws_region.current.name
}
@@ -286,7 +287,7 @@ data "aws_iam_policy_document" "node_assume_role" {
principals {
type = "Service"
- identifiers = ["ec2.amazonaws.com"]
+ identifiers = ["ec2.${local.dns_suffix}"]
}
}
}
diff --git a/modules/karpenter/policy.tf b/modules/karpenter/policy.tf
index 456a27f417..7fb04e47b2 100644
--- a/modules/karpenter/policy.tf
+++ b/modules/karpenter/policy.tf
@@ -195,7 +195,7 @@ data "aws_iam_policy_document" "v033" {
condition {
test = "StringEquals"
variable = "iam:PassedToService"
- values = ["ec2.amazonaws.com"]
+ values = ["ec2.${local.dns_suffix}"]
}
}
@@ -584,7 +584,7 @@ data "aws_iam_policy_document" "v1" {
condition {
test = "StringEquals"
variable = "iam:PassedToService"
- values = ["ec2.amazonaws.com"]
+ values = ["ec2.${local.dns_suffix}"]
}
}
diff --git a/modules/karpenter/versions.tf b/modules/karpenter/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/modules/karpenter/versions.tf
+++ b/modules/karpenter/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/modules/self-managed-node-group/README.md b/modules/self-managed-node-group/README.md
index 8422b0c7ef..7c76477049 100644
--- a/modules/self-managed-node-group/README.md
+++ b/modules/self-managed-node-group/README.md
@@ -10,7 +10,7 @@ module "self_managed_node_group" {
name = "separate-self-mng"
cluster_name = "my-cluster"
- cluster_version = "1.27"
+ cluster_version = "1.31"
cluster_endpoint = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
cluster_auth_base64 = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="
@@ -37,19 +37,19 @@ module "self_managed_node_group" {
}
```
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
@@ -93,8 +93,8 @@ module "self_managed_node_group" {
| [bootstrap\_extra\_args](#input\_bootstrap\_extra\_args) | Additional arguments passed to the bootstrap script. When `ami_type` = `BOTTLEROCKET_*`; these are additional [settings](https://github.com/bottlerocket-os/bottlerocket#settings) that are provided to the Bottlerocket user data | `string` | `""` | no |
| [capacity\_rebalance](#input\_capacity\_rebalance) | Indicates whether capacity rebalance is enabled | `bool` | `null` | no |
| [capacity\_reservation\_specification](#input\_capacity\_reservation\_specification) | Targeting for EC2 capacity reservations | `any` | `{}` | no |
-| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
-| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_post\_nodeadm](#input\_cloudinit\_post\_nodeadm) | Array of cloud-init document parts that are created after the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
+| [cloudinit\_pre\_nodeadm](#input\_cloudinit\_pre\_nodeadm) | Array of cloud-init document parts that are created before the nodeadm document part | list(object({
content = string
content_type = optional(string)
filename = optional(string)
merge_type = optional(string)
})) | `[]` | no |
| [cluster\_auth\_base64](#input\_cluster\_auth\_base64) | Base64 encoded CA of associated EKS cluster | `string` | `""` | no |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Endpoint of associated EKS cluster | `string` | `""` | no |
| [cluster\_ip\_family](#input\_cluster\_ip\_family) | The IP family used to assign Kubernetes pod and service addresses. Valid values are `ipv4` (default) and `ipv6` | `string` | `"ipv4"` | no |
@@ -117,10 +117,13 @@ module "self_managed_node_group" {
| [default\_instance\_warmup](#input\_default\_instance\_warmup) | Amount of time, in seconds, until a newly launched instance can contribute to the Amazon CloudWatch metrics. This delay lets an instance finish initializing before Amazon EC2 Auto Scaling aggregates instance metrics, resulting in more reliable usage data | `number` | `null` | no |
| [delete\_timeout](#input\_delete\_timeout) | Delete timeout to wait for destroying autoscaling group | `string` | `null` | no |
| [desired\_size](#input\_desired\_size) | The number of Amazon EC2 instances that should be running in the autoscaling group | `number` | `1` | no |
+| [desired\_size\_type](#input\_desired\_size\_type) | The unit of measurement for the value specified for `desired_size`. Supported for attribute-based instance type selection only. Valid values: `units`, `vcpu`, `memory-mib` | `string` | `null` | no |
| [disable\_api\_termination](#input\_disable\_api\_termination) | If true, enables EC2 instance termination protection | `bool` | `null` | no |
| [ebs\_optimized](#input\_ebs\_optimized) | If true, the launched EC2 instance will be EBS-optimized | `bool` | `null` | no |
+| [efa\_indices](#input\_efa\_indices) | The indices of the network interfaces that should be EFA-enabled. Only valid when `enable_efa_support` = `true` | `list(number)` | [
0
]
| no |
| [elastic\_gpu\_specifications](#input\_elastic\_gpu\_specifications) | The elastic GPU to attach to the instance | `any` | `{}` | no |
| [elastic\_inference\_accelerator](#input\_elastic\_inference\_accelerator) | Configuration block containing an Elastic Inference Accelerator to attach to the instance | `map(string)` | `{}` | no |
+| [enable\_efa\_only](#input\_enable\_efa\_only) | Determines whether to enable EFA (`false`, default) or EFA and EFA-only (`true`) network interfaces. Note: requires vpc-cni version `v1.18.4` or later | `bool` | `false` | no |
| [enable\_efa\_support](#input\_enable\_efa\_support) | Determines whether to enable Elastic Fabric Adapter (EFA) support | `bool` | `false` | no |
| [enable\_monitoring](#input\_enable\_monitoring) | Enables/disables detailed monitoring | `bool` | `true` | no |
| [enabled\_metrics](#input\_enabled\_metrics) | A list of metrics to collect. The allowed values are `GroupDesiredCapacity`, `GroupInServiceCapacity`, `GroupPendingCapacity`, `GroupMinSize`, `GroupMaxSize`, `GroupInServiceInstances`, `GroupPendingInstances`, `GroupStandbyInstances`, `GroupStandbyCapacity`, `GroupTerminatingCapacity`, `GroupTerminatingInstances`, `GroupTotalCapacity`, `GroupTotalInstances` | `list(string)` | `[]` | no |
@@ -146,7 +149,7 @@ module "self_managed_node_group" {
| [instance\_initiated\_shutdown\_behavior](#input\_instance\_initiated\_shutdown\_behavior) | Shutdown behavior for the instance. Can be `stop` or `terminate`. (Default: `stop`) | `string` | `null` | no |
| [instance\_maintenance\_policy](#input\_instance\_maintenance\_policy) | If this block is configured, add a instance maintenance policy to the specified Auto Scaling group | `any` | `{}` | no |
| [instance\_market\_options](#input\_instance\_market\_options) | The market (purchasing) option for the instance | `any` | `{}` | no |
-| [instance\_refresh](#input\_instance\_refresh) | If this block is configured, start an Instance Refresh when this Auto Scaling Group is updated | `any` | {
"preferences": {
"min_healthy_percentage": 66
},
"strategy": "Rolling"
} | no |
+| [instance\_refresh](#input\_instance\_refresh) | If this block is configured, start an Instance Refresh when this Auto Scaling Group is updated | `any` | {
"preferences": {
"min_healthy_percentage": 66
},
"strategy": "Rolling"
} | no |
| [instance\_requirements](#input\_instance\_requirements) | The attribute requirements for the type of instance. If present then `instance_type` cannot be present | `any` | `{}` | no |
| [instance\_type](#input\_instance\_type) | The type of the instance to launch | `string` | `""` | no |
| [kernel\_id](#input\_kernel\_id) | The kernel ID | `string` | `null` | no |
@@ -162,7 +165,7 @@ module "self_managed_node_group" {
| [maintenance\_options](#input\_maintenance\_options) | The maintenance options for the instance | `any` | `{}` | no |
| [max\_instance\_lifetime](#input\_max\_instance\_lifetime) | The maximum amount of time, in seconds, that an instance can be in service, values must be either equal to 0 or between 604800 and 31536000 seconds | `number` | `null` | no |
| [max\_size](#input\_max\_size) | The maximum size of the autoscaling group | `number` | `3` | no |
-| [metadata\_options](#input\_metadata\_options) | Customize the metadata options for the instance | `map(string)` | {
"http_endpoint": "enabled",
"http_put_response_hop_limit": 2,
"http_tokens": "required"
} | no |
+| [metadata\_options](#input\_metadata\_options) | Customize the metadata options for the instance | `map(string)` | {
"http_endpoint": "enabled",
"http_put_response_hop_limit": 2,
"http_tokens": "required"
} | no |
| [metrics\_granularity](#input\_metrics\_granularity) | The granularity to associate with the metrics to collect. The only valid value is `1Minute` | `string` | `null` | no |
| [min\_elb\_capacity](#input\_min\_elb\_capacity) | Setting this causes Terraform to wait for this number of instances to show up healthy in the ELB only on creation. Updates will not wait on ELB instance number changes | `number` | `null` | no |
| [min\_size](#input\_min\_size) | The minimum size of the autoscaling group | `number` | `0` | no |
@@ -182,7 +185,7 @@ module "self_managed_node_group" {
| [service\_linked\_role\_arn](#input\_service\_linked\_role\_arn) | The ARN of the service-linked role that the ASG will use to call other AWS services | `string` | `null` | no |
| [subnet\_ids](#input\_subnet\_ids) | A list of subnet IDs to launch resources in. Subnets automatically determine which availability zones the group will reside. Conflicts with `availability_zones` | `list(string)` | `null` | no |
| [suspended\_processes](#input\_suspended\_processes) | A list of processes to suspend for the Auto Scaling Group. The allowed values are `Launch`, `Terminate`, `HealthCheck`, `ReplaceUnhealthy`, `AZRebalance`, `AlarmNotification`, `ScheduledActions`, `AddToLoadBalancer`. Note that if you suspend either the `Launch` or `Terminate` process types, it can prevent your Auto Scaling Group from functioning properly | `list(string)` | `[]` | no |
-| [tag\_specifications](#input\_tag\_specifications) | The tags to apply to the resources during launch | `list(string)` | [
"instance",
"volume",
"network-interface"
]
| no |
+| [tag\_specifications](#input\_tag\_specifications) | The tags to apply to the resources during launch | `list(string)` | [
"instance",
"volume",
"network-interface"
]
| no |
| [tags](#input\_tags) | A map of tags to add to all resources | `map(string)` | `{}` | no |
| [target\_group\_arns](#input\_target\_group\_arns) | A set of `aws_alb_target_group` ARNs, for use with Application or Network Load Balancing | `list(string)` | `[]` | no |
| [termination\_policies](#input\_termination\_policies) | A list of policies to decide how the instances in the Auto Scaling Group should be terminated. The allowed values are `OldestInstance`, `NewestInstance`, `OldestLaunchConfiguration`, `ClosestToNextInstanceHour`, `OldestLaunchTemplate`, `AllocationStrategy`, `Default` | `list(string)` | `[]` | no |
@@ -225,4 +228,4 @@ module "self_managed_node_group" {
| [launch\_template\_name](#output\_launch\_template\_name) | The name of the launch template |
| [platform](#output\_platform) | [DEPRECATED - Will be removed in `v21.0`] Identifies the OS platform as `bottlerocket`, `linux` (AL2), `al2023`, or `windows` |
| [user\_data](#output\_user\_data) | Base64 encoded user data |
-
+
diff --git a/modules/self-managed-node-group/main.tf b/modules/self-managed-node-group/main.tf
index bd79468472..ae9af52f73 100644
--- a/modules/self-managed-node-group/main.tf
+++ b/modules/self-managed-node-group/main.tf
@@ -24,6 +24,8 @@ locals {
WINDOWS_FULL_2022_x86_64 = "windows"
AL2023_x86_64_STANDARD = "al2023"
AL2023_ARM_64_STANDARD = "al2023"
+ AL2023_x86_64_NEURON = "al2023"
+ AL2023_x86_64_NVIDIA = "al2023"
}
user_data_type = local.ami_type_to_user_data_type[var.ami_type]
@@ -43,6 +45,8 @@ locals {
WINDOWS_FULL_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-${local.ssm_cluster_version}/image_id"
AL2023_x86_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/standard/recommended/image_id"
AL2023_ARM_64_STANDARD = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/arm64/standard/recommended/image_id"
+ AL2023_x86_64_NEURON = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/neuron/recommended/image_id"
+ AL2023_x86_64_NVIDIA = "/aws/service/eks/optimized-ami/${local.ssm_cluster_version}/amazon-linux-2023/x86_64/nvidia/recommended/image_id"
}
}
@@ -86,7 +90,7 @@ module "user_data" {
################################################################################
data "aws_ec2_instance_type" "this" {
- count = local.enable_efa_support ? 1 : 0
+ count = var.create && var.enable_efa_support ? 1 : 0
instance_type = var.instance_type
}
@@ -97,13 +101,14 @@ locals {
instance_type_provided = var.instance_type != ""
num_network_cards = try(data.aws_ec2_instance_type.this[0].maximum_network_cards, 0)
+ # Primary network interface must be EFA, remaining can be EFA or EFA-only
efa_network_interfaces = [
for i in range(local.num_network_cards) : {
associate_public_ip_address = false
delete_on_termination = true
device_index = i == 0 ? 0 : 1
network_card_index = i
- interface_type = "efa"
+ interface_type = var.enable_efa_only ? contains(concat([0], var.efa_indices), i) ? "efa" : "efa-only" : "efa"
}
]
@@ -417,6 +422,7 @@ resource "aws_launch_template" "this" {
ipv6_prefixes = try(network_interfaces.value.ipv6_prefixes, [])
network_card_index = try(network_interfaces.value.network_card_index, null)
network_interface_id = try(network_interfaces.value.network_interface_id, null)
+ primary_ipv6 = try(network_interfaces.value.primary_ipv6, null)
private_ip_address = try(network_interfaces.value.private_ip_address, null)
# Ref: https://github.com/hashicorp/terraform-provider-aws/issues/4570
security_groups = compact(concat(try(network_interfaces.value.security_groups, []), local.security_group_ids))
@@ -497,6 +503,7 @@ resource "aws_autoscaling_group" "this" {
default_cooldown = var.default_cooldown
default_instance_warmup = var.default_instance_warmup
desired_capacity = var.desired_size
+ desired_capacity_type = var.desired_size_type
enabled_metrics = var.enabled_metrics
force_delete = var.force_delete
force_delete_warm_pool = var.force_delete_warm_pool
diff --git a/modules/self-managed-node-group/variables.tf b/modules/self-managed-node-group/variables.tf
index 92121ea750..9076dab5af 100644
--- a/modules/self-managed-node-group/variables.tf
+++ b/modules/self-managed-node-group/variables.tf
@@ -334,6 +334,19 @@ variable "enable_efa_support" {
default = false
}
+# TODO - make this true by default at next breaking change (remove variable, only pass indices)
+variable "enable_efa_only" {
+ description = "Determines whether to enable EFA (`false`, default) or EFA and EFA-only (`true`) network interfaces. Note: requires vpc-cni version `v1.18.4` or later"
+ type = bool
+ default = false
+}
+
+variable "efa_indices" {
+ description = "The indices of the network interfaces that should be EFA-enabled. Only valid when `enable_efa_support` = `true`"
+ type = list(number)
+ default = [0]
+}
+
variable "metadata_options" {
description = "Customize the metadata options for the instance"
type = map(string)
@@ -420,6 +433,12 @@ variable "desired_size" {
default = 1
}
+variable "desired_size_type" {
+ description = "The unit of measurement for the value specified for `desired_size`. Supported for attribute-based instance type selection only. Valid values: `units`, `vcpu`, `memory-mib`"
+ type = string
+ default = null
+}
+
variable "ignore_failed_scaling_activities" {
description = "Whether to ignore failed Auto Scaling scaling activities while waiting for capacity."
type = bool
diff --git a/modules/self-managed-node-group/versions.tf b/modules/self-managed-node-group/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/modules/self-managed-node-group/versions.tf
+++ b/modules/self-managed-node-group/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/node_groups.tf b/node_groups.tf
index d971f8c13d..4a02a7b155 100644
--- a/node_groups.tf
+++ b/node_groups.tf
@@ -32,7 +32,7 @@ resource "time_sleep" "this" {
create_duration = var.dataplane_wait_duration
triggers = {
- cluster_name = aws_eks_cluster.this[0].name
+ cluster_name = aws_eks_cluster.this[0].id
cluster_endpoint = aws_eks_cluster.this[0].endpoint
cluster_version = aws_eks_cluster.this[0].version
cluster_service_cidr = var.cluster_ip_family == "ipv6" ? try(local.kubernetes_network_config.service_ipv6_cidr, "") : try(local.kubernetes_network_config.service_ipv4_cidr, "")
@@ -375,6 +375,8 @@ module "eks_managed_node_group" {
metadata_options = try(each.value.metadata_options, var.eks_managed_node_group_defaults.metadata_options, local.metadata_options)
enable_monitoring = try(each.value.enable_monitoring, var.eks_managed_node_group_defaults.enable_monitoring, true)
enable_efa_support = try(each.value.enable_efa_support, var.eks_managed_node_group_defaults.enable_efa_support, false)
+ enable_efa_only = try(each.value.enable_efa_only, var.eks_managed_node_group_defaults.enable_efa_only, false)
+ efa_indices = try(each.value.efa_indices, var.eks_managed_node_group_defaults.efa_indices, [0])
create_placement_group = try(each.value.create_placement_group, var.eks_managed_node_group_defaults.create_placement_group, false)
placement = try(each.value.placement, var.eks_managed_node_group_defaults.placement, {})
placement_group_az = try(each.value.placement_group_az, var.eks_managed_node_group_defaults.placement_group_az, null)
@@ -435,6 +437,7 @@ module "self_managed_node_group" {
min_size = try(each.value.min_size, var.self_managed_node_group_defaults.min_size, 0)
max_size = try(each.value.max_size, var.self_managed_node_group_defaults.max_size, 3)
desired_size = try(each.value.desired_size, var.self_managed_node_group_defaults.desired_size, 1)
+ desired_size_type = try(each.value.desired_size_type, var.self_managed_node_group_defaults.desired_size_type, null)
capacity_rebalance = try(each.value.capacity_rebalance, var.self_managed_node_group_defaults.capacity_rebalance, null)
min_elb_capacity = try(each.value.min_elb_capacity, var.self_managed_node_group_defaults.min_elb_capacity, null)
wait_for_elb_capacity = try(each.value.wait_for_elb_capacity, var.self_managed_node_group_defaults.wait_for_elb_capacity, null)
@@ -525,6 +528,8 @@ module "self_managed_node_group" {
metadata_options = try(each.value.metadata_options, var.self_managed_node_group_defaults.metadata_options, local.metadata_options)
enable_monitoring = try(each.value.enable_monitoring, var.self_managed_node_group_defaults.enable_monitoring, true)
enable_efa_support = try(each.value.enable_efa_support, var.self_managed_node_group_defaults.enable_efa_support, false)
+ enable_efa_only = try(each.value.enable_efa_only, var.self_managed_node_group_defaults.enable_efa_only, false)
+ efa_indices = try(each.value.efa_indices, var.self_managed_node_group_defaults.efa_indices, [0])
network_interfaces = try(each.value.network_interfaces, var.self_managed_node_group_defaults.network_interfaces, [])
placement = try(each.value.placement, var.self_managed_node_group_defaults.placement, {})
maintenance_options = try(each.value.maintenance_options, var.self_managed_node_group_defaults.maintenance_options, {})
diff --git a/tests/eks-managed-node-group/README.md b/tests/eks-managed-node-group/README.md
index 28b5c1d773..75419b638f 100644
--- a/tests/eks-managed-node-group/README.md
+++ b/tests/eks-managed-node-group/README.md
@@ -12,24 +12,25 @@ $ terraform apply --auto-approve
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
| Name | Source | Version |
|------|--------|---------|
+| [aws\_vpc\_cni\_ipv6\_pod\_identity](#module\_aws\_vpc\_cni\_ipv6\_pod\_identity) | terraform-aws-modules/eks-pod-identity/aws | ~> 1.6 |
| [disabled\_eks](#module\_disabled\_eks) | ../.. | n/a |
| [disabled\_eks\_managed\_node\_group](#module\_disabled\_eks\_managed\_node\_group) | ../../modules/eks-managed-node-group | n/a |
| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 2.1 |
@@ -95,4 +96,4 @@ No inputs.
| [oidc\_provider\_arn](#output\_oidc\_provider\_arn) | The ARN of the OIDC Provider if `enable_irsa = true` |
| [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
| [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
-
+
diff --git a/tests/eks-managed-node-group/main.tf b/tests/eks-managed-node-group/main.tf
index 0a66cdfc19..851c7b5cda 100644
--- a/tests/eks-managed-node-group/main.tf
+++ b/tests/eks-managed-node-group/main.tf
@@ -7,7 +7,7 @@ data "aws_availability_zones" "available" {}
locals {
name = "ex-${replace(basename(path.cwd), "_", "-")}"
- cluster_version = "1.30"
+ cluster_version = "1.31"
region = "eu-west-1"
vpc_cidr = "10.0.0.0/16"
@@ -45,6 +45,10 @@ module "eks" {
coredns = {
most_recent = true
}
+ eks-pod-identity-agent = {
+ before_compute = true
+ most_recent = true
+ }
kube-proxy = {
most_recent = true
}
@@ -58,6 +62,10 @@ module "eks" {
WARM_PREFIX_TARGET = "1"
}
})
+ pod_identity_association = [{
+ role_arn = module.aws_vpc_cni_ipv6_pod_identity.iam_role_arn
+ service_account = "aws-node"
+ }]
}
}
@@ -65,12 +73,16 @@ module "eks" {
support_type = "STANDARD"
}
+ cluster_zonal_shift_config = {
+ enabled = true
+ }
+
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
control_plane_subnet_ids = module.vpc.intra_subnets
eks_managed_node_group_defaults = {
- ami_type = "AL2_x86_64"
+ ami_type = "AL2023_x86_64_STANDARD"
instance_types = ["m6i.large", "m5.large", "m5n.large", "m5zn.large"]
}
@@ -180,7 +192,7 @@ module "eks" {
# Use a custom AMI
custom_ami = {
- ami_type = "AL2_ARM_64"
+ ami_type = "AL2023_ARM_64_STANDARD"
# Current default AMI used by managed node groups - pseudo "custom"
ami_id = data.aws_ami.eks_default_arm.image_id
@@ -207,13 +219,28 @@ module "eks" {
ami_id = data.aws_ami.eks_default.image_id
enable_bootstrap_user_data = true
- pre_bootstrap_user_data = <<-EOT
- export FOO=bar
- EOT
-
- post_bootstrap_user_data = <<-EOT
- echo "you are free little kubelet!"
- EOT
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+
+ # This is only possible with a custom AMI or self-managed node group
+ cloudinit_post_nodeadm = [{
+ content = <<-EOT
+ echo "All done"
+ EOT
+ content_type = "text/x-shellscript; charset=\"us-ascii\""
+ }]
capacity_type = "SPOT"
force_update_version = true
@@ -223,14 +250,6 @@ module "eks" {
GithubOrg = "terraform-aws-modules"
}
- taints = [
- {
- key = "dedicated"
- value = "gpuGroup"
- effect = "NO_SCHEDULE"
- }
- ]
-
update_config = {
max_unavailable_percentage = 33 # or set `max_unavailable`
}
@@ -308,27 +327,60 @@ module "eks" {
# Can be enabled when appropriate for testing/validation
create = false
- ami_type = "AL2_x86_64_GPU"
- instance_types = ["trn1n.32xlarge"]
+ # The EKS AL2023 NVIDIA AMI provides all of the necessary components
+ # for accelerated workloads w/ EFA
+ ami_type = "AL2023_x86_64_NVIDIA"
+ instance_types = ["p5e.48xlarge"]
- enable_efa_support = true
- pre_bootstrap_user_data = <<-EOT
- # Mount NVME instance store volumes since they are typically
- # available on instances that support EFA
- setup-local-disks raid0
- EOT
+ # Mount instance store volumes in RAID-0 for kubelet and containerd
+ # https://github.com/awslabs/amazon-eks-ami/blob/master/doc/USER_GUIDE.md#raid-0-for-kubelet-and-containerd-raid0
+ cloudinit_pre_nodeadm = [
+ {
+ content_type = "application/node.eks.aws"
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ instance:
+ localStorage:
+ strategy: RAID0
+ EOT
+ }
+ ]
+
+ # This will:
+ # 1. Create a placement group to place the instances close to one another
+ # 2. Ignore subnets that reside in AZs that do not support the instance type
+ # 3. Expose all of the available EFA interfaces on the launch template
+ enable_efa_support = true
+ enable_efa_only = true
+ efa_indices = [0, 4, 8, 12]
+
+ min_size = 1
+ max_size = 1
+ desired_size = 1
+
+ labels = {
+ "vpc.amazonaws.com/efa.present" = "true"
+ "nvidia.com/gpu.present" = "true"
+ }
- min_size = 2
- max_size = 2
- desired_size = 2
+ taints = {
+ # Ensure only GPU workloads are scheduled on this node group
+ gpu = {
+ key = "nvidia.com/gpu"
+ value = "true"
+ effect = "NO_SCHEDULE"
+ }
+ }
}
}
access_entries = {
# One access entry with a policy associated
ex-single = {
- kubernetes_groups = []
- principal_arn = aws_iam_role.this["single"].arn
+ principal_arn = aws_iam_role.this["single"].arn
policy_associations = {
single = {
@@ -343,8 +395,7 @@ module "eks" {
# Example of adding multiple policies to a single access entry
ex-multiple = {
- kubernetes_groups = []
- principal_arn = aws_iam_role.this["multiple"].arn
+ principal_arn = aws_iam_role.this["multiple"].arn
policy_associations = {
ex-one = {
@@ -450,6 +501,18 @@ module "vpc" {
tags = local.tags
}
+module "aws_vpc_cni_ipv6_pod_identity" {
+ source = "terraform-aws-modules/eks-pod-identity/aws"
+ version = "~> 1.6"
+
+ name = "aws-vpc-cni-ipv6"
+
+ attach_aws_vpc_cni_policy = true
+ aws_vpc_cni_enable_ipv6 = true
+
+ tags = local.tags
+}
+
module "ebs_kms_key" {
source = "terraform-aws-modules/kms/aws"
version = "~> 2.1"
@@ -534,7 +597,7 @@ data "aws_ami" "eks_default" {
filter {
name = "name"
- values = ["amazon-eks-node-${local.cluster_version}-v*"]
+ values = ["amazon-eks-node-al2023-x86_64-standard-${local.cluster_version}-v*"]
}
}
@@ -544,7 +607,7 @@ data "aws_ami" "eks_default_arm" {
filter {
name = "name"
- values = ["amazon-eks-arm64-node-${local.cluster_version}-v*"]
+ values = ["amazon-eks-node-al2023-arm64-standard-${local.cluster_version}-v*"]
}
}
diff --git a/tests/eks-managed-node-group/versions.tf b/tests/eks-managed-node-group/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/tests/eks-managed-node-group/versions.tf
+++ b/tests/eks-managed-node-group/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/tests/fargate-profile/README.md b/tests/fargate-profile/README.md
index f0ac3d5390..a50029c722 100644
--- a/tests/fargate-profile/README.md
+++ b/tests/fargate-profile/README.md
@@ -12,19 +12,19 @@ $ terraform apply --auto-approve
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
@@ -85,4 +85,4 @@ No inputs.
| [oidc\_provider\_arn](#output\_oidc\_provider\_arn) | The ARN of the OIDC Provider if `enable_irsa = true` |
| [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
| [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
-
+
diff --git a/tests/fargate-profile/main.tf b/tests/fargate-profile/main.tf
index 6106022373..fc9afc76ed 100644
--- a/tests/fargate-profile/main.tf
+++ b/tests/fargate-profile/main.tf
@@ -6,7 +6,7 @@ data "aws_availability_zones" "available" {}
locals {
name = "ex-${basename(path.cwd)}"
- cluster_version = "1.30"
+ cluster_version = "1.31"
region = "eu-west-1"
vpc_cidr = "10.0.0.0/16"
diff --git a/tests/fargate-profile/versions.tf b/tests/fargate-profile/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/tests/fargate-profile/versions.tf
+++ b/tests/fargate-profile/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/examples/outposts/README.md b/tests/fast-addons/README.md
similarity index 70%
rename from examples/outposts/README.md
rename to tests/fast-addons/README.md
index 5057fc4a5e..693784e79d 100644
--- a/examples/outposts/README.md
+++ b/tests/fast-addons/README.md
@@ -1,34 +1,14 @@
-# EKS on Outposts Example
+# Fast Addons
-Configuration in this directory creates an AWS EKS local cluster on AWS Outposts
+Refer to https://github.com/terraform-aws-modules/terraform-aws-eks/pull/3214 for additional information.
-See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/eks-outposts.html) for further details.
-
-Note: This example requires an an AWS Outpost to provision.
+
## Usage
-To run this example you need to:
-
-1. Deploy the remote host where the cluster will be provisioned from. The remote host is required since only private access is permitted to clusters created on Outposts. If you have access to the network where Outposts are provisioned (VPN, etc.), you can skip this step:
-
-```bash
-$ cd prerequisites
-$ terraform init
-$ terraform plan
-$ terraform apply --auto-approve
-```
-
-2. If provisioning using the remote host deployed in step 1, connect to the remote host using SSM. Note, you will need to have the [SSM plugin for the AWS CLI installed](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html). You can use the output generated by step 1 to connect:
-
-```bash
-$ aws ssm start-session --region --target
-```
-
-3. Once connected to the remote host, navigate to the cloned project example directory and deploy the example:
+To provision the provided configurations you need to execute:
```bash
-$ cd $HOME/terraform-aws-eks/examples/outposts
$ terraform init
$ terraform plan
$ terraform apply --auto-approve
@@ -36,49 +16,39 @@ $ terraform apply --auto-approve
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
-```bash
-terraform destroy --auto-approve
-```
-
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
-| [kubernetes](#requirement\_kubernetes) | >= 2.20 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
-| [kubernetes](#provider\_kubernetes) | >= 2.20 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
| Name | Source | Version |
|------|--------|---------|
| [eks](#module\_eks) | ../.. | n/a |
+| [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |
## Resources
| Name | Type |
|------|------|
-| [kubernetes_storage_class_v1.this](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/storage_class_v1) | resource |
-| [aws_outposts_outpost_instance_types.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/outposts_outpost_instance_types) | data source |
-| [aws_outposts_outposts.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/outposts_outposts) | data source |
-| [aws_subnet.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnet) | data source |
-| [aws_subnets.lookup](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
-| [aws_subnets.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
-| [aws_vpc.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc) | data source |
+| [aws_route_table_association.custom_network](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table_association) | resource |
+| [aws_subnet.custom_network](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/subnet) | resource |
+| [aws_vpc_ipv4_cidr_block_association.custom_network](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_ipv4_cidr_block_association) | resource |
+| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
## Inputs
-| Name | Description | Type | Default | Required |
-|------|-------------|------|---------|:--------:|
-| [region](#input\_region) | The AWS region to deploy into (e.g. us-east-1) | `string` | `"us-west-2"` | no |
+No inputs.
## Outputs
@@ -119,4 +89,4 @@ terraform destroy --auto-approve
| [oidc\_provider\_arn](#output\_oidc\_provider\_arn) | The ARN of the OIDC Provider if `enable_irsa = true` |
| [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
| [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
-
+
diff --git a/tests/fast-addons/main.tf b/tests/fast-addons/main.tf
new file mode 100644
index 0000000000..12d2a41f8c
--- /dev/null
+++ b/tests/fast-addons/main.tf
@@ -0,0 +1,159 @@
+provider "aws" {
+ region = local.region
+}
+
+locals {
+ name = "ex-${basename(path.cwd)}"
+ cluster_version = "1.31"
+ region = "eu-west-1"
+
+ tags = {
+ Test = local.name
+ GithubRepo = "terraform-aws-eks"
+ GithubOrg = "terraform-aws-modules"
+ }
+}
+
+################################################################################
+# EKS Module
+################################################################################
+
+module "eks" {
+ source = "../.."
+
+ cluster_name = local.name
+ cluster_version = local.cluster_version
+ cluster_endpoint_public_access = true
+
+ enable_cluster_creator_admin_permissions = true
+
+ # Disable the default self-managed addons to avoid the penalty of adopting them later
+ bootstrap_self_managed_addons = false
+
+ # Addons will be provisioned net new via the EKS addon API
+ cluster_addons = {
+ coredns = {
+ most_recent = true
+ }
+ eks-pod-identity-agent = {
+ before_compute = true
+ most_recent = true
+ }
+ kube-proxy = {
+ most_recent = true
+ }
+ vpc-cni = {
+ most_recent = true
+ before_compute = true
+ configuration_values = jsonencode({
+ env = {
+ # Use subnet tags to avoid the need to inject the ENIConfig
+ # which requires a live API server endpoint which leads to a dependency of:
+ # Control plane -> API request to create ENIConfig -> VPC CNI addon -> nodes/compute
+ # With the subnet discovery feature, we can avoid this dependency:
+ # Control plane -> VPC CNI addon -> nodes/compute
+ ENABLE_SUBNET_DISCOVERY = "true"
+ }
+ })
+ }
+ }
+
+ vpc_id = module.vpc.vpc_id
+ subnet_ids = module.vpc.private_subnets
+
+ eks_managed_node_groups = {
+ example = {
+ instance_types = ["m6i.large"]
+
+ min_size = 2
+ max_size = 5
+ desired_size = 2
+ }
+ }
+
+ tags = local.tags
+}
+
+################################################################################
+# VPC
+################################################################################
+
+data "aws_availability_zones" "available" {
+ # Exclude local zones
+ filter {
+ name = "opt-in-status"
+ values = ["opt-in-not-required"]
+ }
+}
+
+locals {
+ vpc_cidr = "10.0.0.0/16"
+ azs = slice(data.aws_availability_zones.available.names, 0, 3)
+}
+
+module "vpc" {
+ source = "terraform-aws-modules/vpc/aws"
+ version = "~> 5.0"
+
+ name = local.name
+ cidr = local.vpc_cidr
+
+ azs = local.azs
+ private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
+ public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
+
+ enable_nat_gateway = true
+ single_nat_gateway = true
+
+ public_subnet_tags = {
+ "kubernetes.io/role/elb" = 1
+ }
+
+ tags = local.tags
+}
+
+################################################################################
+# Custom Networking
+################################################################################
+
+locals {
+ custom_network_vpc_cidr = "10.99.0.0/16"
+
+ custom_network_subnets = [for k, v in local.azs : cidrsubnet(local.custom_network_vpc_cidr, 4, k)]
+}
+
+resource "aws_vpc_ipv4_cidr_block_association" "custom_network" {
+ vpc_id = module.vpc.vpc_id
+ cidr_block = local.custom_network_vpc_cidr
+}
+
+resource "aws_subnet" "custom_network" {
+ count = length(local.custom_network_subnets)
+
+ vpc_id = module.vpc.vpc_id
+ cidr_block = element(local.custom_network_subnets, count.index)
+
+ tags = merge(
+ local.tags,
+ {
+ # Tag for subnet discovery
+ "kubernetes.io/role/cni" = 1
+ "kubernetes.io/role/internal-elb" = 1
+ }
+ )
+
+ depends_on = [
+ aws_vpc_ipv4_cidr_block_association.custom_network
+ ]
+}
+
+resource "aws_route_table_association" "custom_network" {
+ count = length(local.custom_network_subnets)
+
+ subnet_id = element(aws_subnet.custom_network[*].id, count.index)
+ route_table_id = element(module.vpc.private_route_table_ids, 0)
+
+ depends_on = [
+ aws_vpc_ipv4_cidr_block_association.custom_network
+ ]
+}
diff --git a/examples/outposts/outputs.tf b/tests/fast-addons/outputs.tf
similarity index 100%
rename from examples/outposts/outputs.tf
rename to tests/fast-addons/outputs.tf
diff --git a/tests/fast-addons/variables.tf b/tests/fast-addons/variables.tf
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/examples/outposts/prerequisites/versions.tf b/tests/fast-addons/versions.tf
similarity index 82%
rename from examples/outposts/prerequisites/versions.tf
rename to tests/fast-addons/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/examples/outposts/prerequisites/versions.tf
+++ b/tests/fast-addons/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/tests/self-managed-node-group/README.md b/tests/self-managed-node-group/README.md
index 6f5ddce892..1587f7c177 100644
--- a/tests/self-managed-node-group/README.md
+++ b/tests/self-managed-node-group/README.md
@@ -12,24 +12,25 @@ $ terraform apply --auto-approve
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
-
+
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.3.2 |
-| [aws](#requirement\_aws) | >= 5.61 |
+| [aws](#requirement\_aws) | >= 5.75 |
## Providers
| Name | Version |
|------|---------|
-| [aws](#provider\_aws) | >= 5.61 |
+| [aws](#provider\_aws) | >= 5.75 |
## Modules
| Name | Source | Version |
|------|--------|---------|
+| [aws\_vpc\_cni\_ipv4\_pod\_identity](#module\_aws\_vpc\_cni\_ipv4\_pod\_identity) | terraform-aws-modules/eks-pod-identity/aws | ~> 1.6 |
| [disabled\_self\_managed\_node\_group](#module\_disabled\_self\_managed\_node\_group) | ../../modules/self-managed-node-group | n/a |
| [ebs\_kms\_key](#module\_ebs\_kms\_key) | terraform-aws-modules/kms/aws | ~> 2.0 |
| [eks](#module\_eks) | ../.. | n/a |
@@ -90,4 +91,4 @@ No inputs.
| [oidc\_provider\_arn](#output\_oidc\_provider\_arn) | The ARN of the OIDC Provider if `enable_irsa = true` |
| [self\_managed\_node\_groups](#output\_self\_managed\_node\_groups) | Map of attribute maps for all self managed node groups created |
| [self\_managed\_node\_groups\_autoscaling\_group\_names](#output\_self\_managed\_node\_groups\_autoscaling\_group\_names) | List of the autoscaling group names created by self-managed node groups |
-
+
diff --git a/tests/self-managed-node-group/main.tf b/tests/self-managed-node-group/main.tf
index 7fc6171e95..afe7aac9a1 100644
--- a/tests/self-managed-node-group/main.tf
+++ b/tests/self-managed-node-group/main.tf
@@ -7,7 +7,7 @@ data "aws_availability_zones" "available" {}
locals {
name = "ex-${replace(basename(path.cwd), "_", "-")}"
- cluster_version = "1.29"
+ cluster_version = "1.31"
region = "eu-west-1"
vpc_cidr = "10.0.0.0/16"
@@ -41,11 +41,18 @@ module "eks" {
coredns = {
most_recent = true
}
+ eks-pod-identity-agent = {
+ most_recent = true
+ }
kube-proxy = {
most_recent = true
}
vpc-cni = {
most_recent = true
+ pod_identity_association = [{
+ role_arn = module.aws_vpc_cni_ipv4_pod_identity.iam_role_arn
+ service_account = "aws-node"
+ }]
}
}
@@ -61,6 +68,9 @@ module "eks" {
}
self_managed_node_group_defaults = {
+ ami_type = "AL2023_x86_64_STANDARD"
+ ami_id = data.aws_ami.eks_default.image_id
+
# enable discovery of autoscaling groups by cluster-autoscaler
autoscaling_group_tags = {
"k8s.io/cluster-autoscaler/enabled" : true,
@@ -72,29 +82,6 @@ module "eks" {
# Default node group - as provisioned by the module defaults
default_node_group = {}
- # AL2023 node group utilizing new user data format which utilizes nodeadm
- # to join nodes to the cluster (instead of /etc/eks/bootstrap.sh)
- al2023_nodeadm = {
- ami_type = "AL2023_x86_64_STANDARD"
-
- cloudinit_pre_nodeadm = [
- {
- content_type = "application/node.eks.aws"
- content = <<-EOT
- ---
- apiVersion: node.eks.aws/v1alpha1
- kind: NodeConfig
- spec:
- kubelet:
- config:
- shutdownGracePeriod: 30s
- featureGates:
- DisableKubeletCloudCredentialProviders: true
- EOT
- }
- ]
- }
-
# Bottlerocket node group
bottlerocket = {
name = "bottlerocket-self-mng"
@@ -138,7 +125,18 @@ module "eks" {
max_size = 5
desired_size = 2
- bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ flags:
+ - --node-labels=node.kubernetes.io/lifecycle=spot
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
use_mixed_instances_policy = true
mixed_instances_policy = {
@@ -172,15 +170,18 @@ module "eks" {
max_size = 7
desired_size = 1
- ami_id = data.aws_ami.eks_default.id
-
- pre_bootstrap_user_data = <<-EOT
- export FOO=bar
- EOT
-
- post_bootstrap_user_data = <<-EOT
- echo "you are free little kubelet!"
- EOT
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ flags:
+ - --node-labels=node.kubernetes.io/lifecycle=spot
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
instance_type = "m6i.large"
@@ -215,6 +216,21 @@ module "eks" {
bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
+ cloudinit_pre_nodeadm = [{
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ kubelet:
+ config:
+ shutdownGracePeriod: 30s
+ featureGates:
+ DisableKubeletCloudCredentialProviders: true
+ EOT
+ content_type = "application/node.eks.aws"
+ }]
+
instance_type = null
# launch template configuration
@@ -287,19 +303,53 @@ module "eks" {
# Can be enabled when appropriate for testing/validation
create = false
- ami_type = "AL2_x86_64_GPU"
- instance_type = "trn1n.32xlarge"
+ # The EKS AL2023 NVIDIA AMI provides all of the necessary components
+ # for accelerated workloads w/ EFA
+ ami_type = "AL2023_x86_64_NVIDIA"
+ instance_types = ["p5e.48xlarge"]
- enable_efa_support = true
- pre_bootstrap_user_data = <<-EOT
- # Mount NVME instance store volumes since they are typically
- # available on instances that support EFA
- setup-local-disks raid0
- EOT
+ # Mount instance store volumes in RAID-0 for kubelet and containerd
+ # https://github.com/awslabs/amazon-eks-ami/blob/master/doc/USER_GUIDE.md#raid-0-for-kubelet-and-containerd-raid0
+ cloudinit_pre_nodeadm = [
+ {
+ content_type = "application/node.eks.aws"
+ content = <<-EOT
+ ---
+ apiVersion: node.eks.aws/v1alpha1
+ kind: NodeConfig
+ spec:
+ instance:
+ localStorage:
+ strategy: RAID0
+ EOT
+ }
+ ]
+
+ # This will:
+ # 1. Create a placement group to place the instances close to one another
+ # 2. Ignore subnets that reside in AZs that do not support the instance type
+ # 3. Expose all of the available EFA interfaces on the launch template
+ enable_efa_support = true
+ enable_efa_only = true
+ efa_indices = [0, 4, 8, 12]
min_size = 2
max_size = 2
desired_size = 2
+
+ labels = {
+ "vpc.amazonaws.com/efa.present" = "true"
+ "nvidia.com/gpu.present" = "true"
+ }
+
+ taints = {
+ # Ensure only GPU workloads are scheduled on this node group
+ gpu = {
+ key = "nvidia.com/gpu"
+ value = "true"
+ effect = "NO_SCHEDULE"
+ }
+ }
}
}
@@ -345,13 +395,25 @@ module "vpc" {
tags = local.tags
}
+module "aws_vpc_cni_ipv4_pod_identity" {
+ source = "terraform-aws-modules/eks-pod-identity/aws"
+ version = "~> 1.6"
+
+ name = "aws-vpc-cni-ipv4"
+
+ attach_aws_vpc_cni_policy = true
+ aws_vpc_cni_enable_ipv4 = true
+
+ tags = local.tags
+}
+
data "aws_ami" "eks_default" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
- values = ["amazon-eks-node-${local.cluster_version}-v*"]
+ values = ["amazon-eks-node-al2023-x86_64-standard-${local.cluster_version}-v*"]
}
}
diff --git a/tests/self-managed-node-group/versions.tf b/tests/self-managed-node-group/versions.tf
index 5bfe6da389..0099e6baaf 100644
--- a/tests/self-managed-node-group/versions.tf
+++ b/tests/self-managed-node-group/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
}
}
diff --git a/tests/user-data/README.md b/tests/user-data/README.md
index 3278c7f0f9..88988ab2f4 100644
--- a/tests/user-data/README.md
+++ b/tests/user-data/README.md
@@ -12,7 +12,7 @@ $ terraform plan
$ terraform apply --auto-approve
```
-
+
## Requirements
| Name | Version |
@@ -104,4 +104,4 @@ No inputs.
## Outputs
No outputs.
-
+
diff --git a/variables.tf b/variables.tf
index 420248c744..edd0d938a6 100644
--- a/variables.tf
+++ b/variables.tf
@@ -50,6 +50,12 @@ variable "cluster_upgrade_policy" {
default = {}
}
+variable "cluster_zonal_shift_config" {
+ description = "Configuration block for the cluster zonal shift"
+ type = any
+ default = {}
+}
+
variable "cluster_additional_security_group_ids" {
description = "List of additional, externally created security group IDs to attach to the cluster control plane"
type = list(string)
@@ -142,6 +148,7 @@ variable "cluster_timeouts" {
default = {}
}
+# TODO - hard code to false on next breaking change
variable "bootstrap_self_managed_addons" {
description = "Indicates whether or not to bootstrap self-managed addons after the cluster has been created"
type = bool
diff --git a/versions.tf b/versions.tf
index 00d8754704..fc9dadd253 100644
--- a/versions.tf
+++ b/versions.tf
@@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = ">= 5.61"
+ version = ">= 5.75"
}
tls = {
source = "hashicorp/tls"