From 80f937e576fca859cbc63f432a3440a770952a51 Mon Sep 17 00:00:00 2001 From: Marcio Morales <marciogmorales@hotmail.com> Date: Wed, 29 Nov 2023 13:34:46 -0600 Subject: [PATCH 1/2] Amazon EKS - Terraform module refactoring Refactoring Amazon EKS terraform modules. --- .gitignore | 6 +- terraform/README.md | 92 ------- terraform/aws/eks-windows/README.md | 117 +++++++++ terraform/aws/eks-windows/locals.tf | 8 + terraform/aws/eks-windows/main.tf | 334 +++++++++++++++++++++++++ terraform/aws/eks-windows/outputs.tf | 7 + terraform/aws/eks-windows/variables.tf | 137 ++++++++++ terraform/aws/vpc/locals.tf | 4 + terraform/aws/vpc/main.tf | 122 +++++++++ terraform/aws/vpc/output.tf | 25 ++ terraform/aws/vpc/variable.tf | 26 ++ terraform/data.tf | 16 -- terraform/locals.tf | 31 --- terraform/main.tf | 166 ------------ terraform/provider.tf | 16 -- terraform/terraform.tf | 22 -- terraform/variables.tf | 24 -- 17 files changed, 785 insertions(+), 368 deletions(-) delete mode 100644 terraform/README.md create mode 100644 terraform/aws/eks-windows/README.md create mode 100644 terraform/aws/eks-windows/locals.tf create mode 100644 terraform/aws/eks-windows/main.tf create mode 100644 terraform/aws/eks-windows/outputs.tf create mode 100644 terraform/aws/eks-windows/variables.tf create mode 100644 terraform/aws/vpc/locals.tf create mode 100644 terraform/aws/vpc/main.tf create mode 100644 terraform/aws/vpc/output.tf create mode 100644 terraform/aws/vpc/variable.tf delete mode 100644 terraform/data.tf delete mode 100644 terraform/locals.tf delete mode 100644 terraform/main.tf delete mode 100644 terraform/provider.tf delete mode 100644 terraform/terraform.tf delete mode 100644 terraform/variables.tf diff --git a/.gitignore b/.gitignore index 41aa3d5..e169860 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,8 @@ kubernetes/ main -op-readiness -e2e.test test.tar.gz - -# Sonobuoy results sonobuoy-results/ +terraform/jumper/.terraform.lock.hcl -terraform/jumper/.terraform.lock.hcl +terraform/*.tfstate +terraform/*.tfstate.* +terraform/*terraform.lock* +*.DS_Store* + diff --git a/terraform/README.md b/terraform/README.md deleted file mode 100644 index be4ca2a..0000000 --- a/terraform/README.md +++ /dev/null @@ -1,92 +0,0 @@ -## Create a Windows Cluster - -In case you don't have a Windows cluster available, this project gives an option to bootstrap a new cluster -on AWS (initially) via Terraform using AWS managed clusters with EKS. Other projects exists in -case the user prefer to create the cluster locally with a robust machine, see [here](https://github.com/kubernetes-sigs/sig-windows-dev-tools). - -### Pre-requisites - -Terraform >= 1.1.0 -AWS Account with proper IAM permissions - -### Initializing modules - -Under the folder `./terraform` all the resources exists, to initizlie and download the used modules -call, terraform with init parameter: - -```shell -$ terraform init -backend false - -Initializing the backend... -Initializing modules... - -Initializing provider plugins... -- Reusing previous version of hashicorp/kubernetes from the dependency lock file -- Reusing previous version of hashicorp/cloudinit from the dependency lock file -- Reusing previous version of hashicorp/aws from the dependency lock file -- Reusing previous version of hashicorp/time from the dependency lock file -- Reusing previous version of hashicorp/tls from the dependency lock file -- Using previously-installed hashicorp/tls v4.0.4 -- Using previously-installed hashicorp/kubernetes v2.23.0 -- Using previously-installed hashicorp/cloudinit v2.3.2 -- Using previously-installed hashicorp/aws v5.26.0 -- Using previously-installed hashicorp/time v0.9.1 - -Terraform has been successfully initialized! - -You may now begin working with Terraform. Try running "terraform plan" to see -any changes that are required for your infrastructure. All Terraform commands -should now work. - -If you ever set or change modules or backend configuration for Terraform, -rerun this command to reinitialize your working directory. If you forget, other -commands will detect it and remind you to do so if necessary. -``` - -If the message `Terraform has been successfully initialized!` appears, proceed to the next -step, verify the version of the hashicorp plugins. - -### Verify the planning - -Terraform allows you to verify the resources on `dry-run` style, so you can double-check and verify if all -resources are being created in the DAG and managed correctly - -```shell -terraform plan -``` - -### Creating the cluster - -The new infrastructure is created using apply, based on the plan generated the DAG indicated all the AWS -resources created by this module. It includes a EKS cluster with 2 node groups: - -1. Linux node group with 3 nodes `t3.medium` using Amazon Linux -2. Windows node group with 1 node `t3.large` using Windows 2022 Core - -Addons are being installed on this clusters by default, kube-proxy, coredns and vpc-cni (with Windows support) - -To start creating, apply your plan with: - -```shell -terraform apply -... -Apply complete! Resources: 67 added, 0 changed, 0 destroyed. -``` - -## Notes - -There's **NO** persistence of the state, so a local `terraform.tfstate` file is created, keep it locally to manage -your cluster while you are working with it. - -To export the KubeConfig file and create new context for the new created cluster: - -```shell -aws eks update-kubeconfig --region us-east-1 --name eks-windows -``` - -## Resources - -A few other resources can be consulted in case of doubts or slight modification: - -* [Official EKS Documentation](https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html) -* [Running Windows Containers on AWS: A complete guide to successfully running Windows containers on Amazon ECS, EKS, and AWS Fargate](https://www.amazon.com/Running-Windows-Containers-AWS-successfully/dp/1804614130) \ No newline at end of file diff --git a/terraform/aws/eks-windows/README.md b/terraform/aws/eks-windows/README.md new file mode 100644 index 0000000..2040e75 --- /dev/null +++ b/terraform/aws/eks-windows/README.md @@ -0,0 +1,117 @@ +## Create a Windows Cluster + +In case you don't have an Amazon EKS with Windows nodes, this project gives an option to bootstrap a new Amazon EKS cluster. Other projects exists in +case the user prefer to create the cluster locally with a robust machine, see [here](https://github.com/kubernetes-sigs/sig-windows-dev-tools). + +### Pre-requisites + +Terraform >= 1.6.x +AWS Account with proper IAM permissions + +### Initializing modules + +Under the folder `./terraform/aws/eks-windows` all the resources exists, to initizlie and download the used modules +call, terraform with init parameter: + +```shell +$ terraform init -backend false + +Initializing the backend... +Initializing modules... + +Initializing provider plugins... +- Reusing previous version of hashicorp/aws from the dependency lock file +- Reusing previous version of hashicorp/kubernetes from the dependency lock file +- Reusing previous version of hashicorp/tls from the dependency lock file +- Using previously-installed hashicorp/kubernetes v2.24.0 +- Using previously-installed hashicorp/tls v4.0.5 +- Using previously-installed hashicorp/aws v5.27.0 + +Terraform has been successfully initialized! + +You may now begin working with Terraform. Try running "terraform plan" to see +any changes that are required for your infrastructure. All Terraform commands +should now work. + +If you ever set or change modules or backend configuration for Terraform, +rerun this command to reinitialize your working directory. If you forget, other +commands will detect it and remind you to do so if necessary. +``` + +If the message `Terraform has been successfully initialized!` appears, proceed to the next +step, verify the version of the hashicorp plugins. + +### Verify the planning + +Terraform allows you to verify the resources on `dry-run` style, so you can double-check and verify if all +resources are being created in the DAG and managed correctly + +```shell +terraform plan +``` + +### Creating the cluster + +The new infrastructure is created using apply, based on the plan generated the DAG indicated all the AWS +resources created by this module. It includes a EKS cluster with 2 node groups: + +1. Linux node group with 3 nodes `t3.medium` using Amazon Linux +2. Windows node group with 1 node `t3.medium` using Windows 2022 Core + +To start creating, apply your plan with: + +```shell +terraform apply +... +Apply complete! Resources: 67 added, 0 changed, 0 destroyed. +``` + +## Notes + +There's **NO** persistence of the state, so a local `terraform.tfstate` file is created, keep it locally to manage +your cluster while you are working with it. + +To export the KubeConfig file and create new context for the new created cluster: + +```shell +aws eks update-kubeconfig --region us-east-1 --name eks-windows +``` + +## Resources + +A few other resources can be consulted in case of doubts or slight modification: + +* [Official EKS Documentation](https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html) +* [Running Windows Containers on AWS: A complete guide to successfully running Windows containers on Amazon ECS, EKS, and AWS Fargate](https://www.amazon.com/Running-Windows-Containers-AWS-successfully/dp/1804614130) + + + +---- + + + +## Providers + +- hashicorp/aws | version = "~> 45.0" + +## Variables description +- **eks_cluster_name (string)**: Namne of the EKS cluster +- **endpoint_private_access (bool)**: Indicates whether or not the Amazon EKS private API server endpoint is enabled +- **endpoint_public_access (bool)**: Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true +- **public_access_cidrs (list(string))**: Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0. +- **enabled_cluster_log_types (list(string))**: A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [`api`, `audit`, `authenticator`, `controllerManager`, `scheduler`] +- **cluster_log_retention_period (number)**: Number of days to retain cluster logs. Requires `enabled_cluster_log_types` to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. +- **cluster_encryption_config_enabled (bool)**: Set to `true` to enable Cluster Encryption Configuration +- **cluster_encryption_config_kms_key_id (string)**: KMS Key ID to use for cluster encryption config +- **cluster_encryption_config_kms_key_enable_key_rotation (bool)**: Cluster Encryption Config KMS Key Resource argument - enable kms key rotation +- **cluster_encryption_config_kms_key_deletion_window_in_days (number)**: Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction +- **cluster_encryption_config_kms_key_policy (string)**: Cluster Encryption Config KMS Key Resource argument - key policy +- **cluster_encryption_config_resources (list(any))**: Cluster Encryption Config Resources to encrypt, e.g. ['secrets'] +- **eks_cluster_version (string)**: Version for the EKS cluster +- **launch_template_name (string)**: Name for the launch template +- **ec2_instance_types (string)**: EC2 instance type +- **eks_windows_workernode_instance_profile_name (string)**: Worker node instance profile name +- **alb_ingress_ports (list(number))**: List of ports opened from Internet to ALB +- **container_instances_ingress_ports (list(number))**: List of ports opened from ALB to Container Instances +- **kubelet_extra_args (string)**: This will make sure to taint your nodes at the boot time to avoid scheduling any existing resources in the new Windows worker nodes +- **map_users (list(object({})))**: Additional IAM users to add to the aws-auth configmap. diff --git a/terraform/aws/eks-windows/locals.tf b/terraform/aws/eks-windows/locals.tf new file mode 100644 index 0000000..e209842 --- /dev/null +++ b/terraform/aws/eks-windows/locals.tf @@ -0,0 +1,8 @@ +locals { + http_port = 80 + any_port = 0 + any_protocol = "-1" + tcp_protocol = "tcp" + all_ips_ipv4 = ["0.0.0.0/0"] + all_ips_ipv6 = ["::/0"] +} \ No newline at end of file diff --git a/terraform/aws/eks-windows/main.tf b/terraform/aws/eks-windows/main.tf new file mode 100644 index 0000000..53ca64b --- /dev/null +++ b/terraform/aws/eks-windows/main.tf @@ -0,0 +1,334 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 5.0" + } + } +} + +module "vpc" { + source = "../vpc" +} + +provider "kubernetes" { + host = data.aws_eks_cluster.eks_windows_cluster_data.endpoint + cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks_windows_cluster_data.certificate_authority[0].data) + token = data.aws_eks_cluster_auth.eks_windows_cluster_data.token +} + +### Data + +data "aws_eks_cluster" "eks_windows_cluster_ca" { + name = aws_eks_cluster.eks_windows.name +} + + +data "aws_eks_cluster" "eks_windows_cluster_data" { + name = aws_eks_cluster.eks_windows.name +} + +data "aws_eks_cluster_auth" "eks_windows_cluster_data" { + name = aws_eks_cluster.eks_windows.name +} + +data "tls_certificate" "eks_windows_cluster_tls" { + url = aws_eks_cluster.eks_windows.identity[0].oidc[0].issuer +} + +data "aws_iam_policy_document" "eks_windows_assume_role_policy" { + statement { + actions = ["sts:AssumeRoleWithWebIdentity"] + effect = "Allow" + + condition { + test = "StringEquals" + variable = "${replace(aws_iam_openid_connect_provider.eks_iam_openid.url, "https://", "")}:sub" + values = ["system:serviceaccount:kube-system:aws-node"] + } + + principals { + identifiers = [aws_iam_openid_connect_provider.eks_iam_openid.arn] + type = "Federated" + } + } +} + +data "aws_ami" "eks_optimized_ami" { + most_recent = true + owners = ["amazon"] + + filter { + name = "name" + values = ["Windows_Server-2022-English-Core-EKS_Optimized-*"] + } +} + +data "aws_caller_identity" "account_id" {} + +output "account_id" { + value = data.aws_caller_identity.account_id.account_id +} + +### Security Group + +resource "aws_security_group" "cluster_sg" { + name = "cluster_sg" + description = "Allow TLS inbound traffic" + vpc_id = module.vpc.vpc_id +} + +resource "aws_security_group" "windows_sg" { + name = "windows_sg" + description = "Allow TLS inbound traffic" + vpc_id = module.vpc.vpc_id + + egress { + from_port = 0 + to_port = 0 + protocol = -1 + cidr_blocks = ["0.0.0.0/0"] + } +} + +resource "aws_security_group_rule" "rule_worker_windows" { + type = "ingress" + from_port = 0 + to_port = 0 + protocol = -1 + source_security_group_id = aws_security_group.cluster_sg.id + security_group_id = aws_security_group.windows_sg.id +} + +resource "aws_security_group_rule" "rule_control_plane" { + type = "ingress" + from_port = 0 + to_port = 0 + protocol = -1 + security_group_id = aws_security_group.cluster_sg.id + source_security_group_id = aws_security_group.windows_sg.id +} + +### ECS IAM Roles and Instance Roles + +resource "aws_iam_openid_connect_provider" "eks_iam_openid" { + client_id_list = ["sts.amazonaws.com"] + thumbprint_list = [data.tls_certificate.eks_windows_cluster_tls.certificates[0].sha1_fingerprint] + url = aws_eks_cluster.eks_windows.identity[0].oidc[0].issuer +} + +#### EKS VPC CNI Role + +resource "aws_iam_role" "eks_vpc_cni_role" { + assume_role_policy = data.aws_iam_policy_document.eks_windows_assume_role_policy.json + name = "eks-vpc-cni-role" +} + +resource "aws_iam_role_policy_attachment" "eks_iam_role_attach_AmazonEKS_CNI_Policy" { + policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" + role = aws_iam_role.eks_vpc_cni_role.name +} + +#### EKS Cluster - Role + +resource "aws_iam_role" "eks_iam_role_cluster_service" { + name = "eks-cluster-service-role" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = "sts:AssumeRole" + Sid = "" + Effect = "Allow" + Principal = { + Service = "eks.amazonaws.com" + } + }, + ] + }) +} + +resource "aws_iam_role_policy_attachment" "eks_iam_role_cluster_service_attach" { + for_each = toset([ + "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy", + "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController" + ]) + role = aws_iam_role.eks_iam_role_cluster_service.name + policy_arn = each.value +} + +#### EKS Linux Node Group - Role +resource "aws_iam_role" "eks_node_group_role_linux" { + name = "eks-node-group-linux-role" + + assume_role_policy = jsonencode({ + Statement = [{ + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ec2.amazonaws.com" + } + }] + Version = "2012-10-17" + }) +} + +resource "aws_iam_role_policy_attachment" "eks_linux_node_group_role_attach" { + for_each = toset([ + "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy", + "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", + "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy", + "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" + ]) + role = aws_iam_role.eks_node_group_role_linux.name + policy_arn = each.value +} + +#### EKS Windows Node Group - Role +resource "aws_iam_role" "eks_node_group_role_windows" { + name = "eks-node-group-windows-role" + + assume_role_policy = jsonencode({ + Statement = [{ + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ec2.amazonaws.com" + } + }] + Version = "2012-10-17" + }) +} + +resource "aws_iam_role_policy_attachment" "eks_windows_node_group_role_attach" { + for_each = toset([ + "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy", + "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", + "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy", + "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" + ]) + role = aws_iam_role.eks_node_group_role_windows.name + policy_arn = each.value +} + +resource "aws_iam_instance_profile" "eks_windows_workernode_instance_profile" { + name = var.eks_windows_workernode_instance_profile_name + role = aws_iam_role.eks_node_group_role_windows.name +} + +### EKS Cluster + +resource "aws_eks_cluster" "eks_windows" { + name = var.eks_cluster_name + role_arn = aws_iam_role.eks_iam_role_cluster_service.arn + version = var.eks_cluster_version + + vpc_config { + subnet_ids = module.vpc.private_subnets_id + } +} + +#### Enable VPC CNI Windows Support and Prefix Delegation + +resource "kubernetes_config_map_v1_data" "amazon_vpc_cni_windows" { + depends_on = [ + aws_eks_cluster.eks_windows + ] + metadata { + name = "amazon-vpc-cni" + namespace = "kube-system" + } + + data = { + enable-windows-ipam : "true" + enable-windows-prefix-delegation : "true" + } + + force = "true" +} + +### AWS CONFIGMAP + +resource "kubernetes_config_map" "configmap" { + data = { + "mapRoles" = <<EOT +- groups: + - system:bootstrappers + - system:nodes + rolearn: arn:aws:iam::${data.aws_caller_identity.account_id.account_id}:role/eks-node-group-linux-role + username: system:node:{{EC2PrivateDNSName}} +- groups: + - eks:kube-proxy-windows + - system:bootstrappers + - system:nodes + rolearn: arn:aws:iam::${data.aws_caller_identity.account_id.account_id}:role/eks-node-group-windows-role + username: system:node:{{EC2PrivateDNSName}} +EOT + } + + metadata { + name = "aws-auth" + namespace = "kube-system" + } +} + +### EKS Linux Node Group + +resource "aws_eks_node_group" "node_group_linux" { + cluster_name = aws_eks_cluster.eks_windows.name + node_group_name = "linux-managed-nodegroup" + node_role_arn = aws_iam_role.eks_node_group_role_linux.arn + subnet_ids = module.vpc.private_subnets_id + depends_on = [ + aws_iam_role_policy_attachment.eks_linux_node_group_role_attach + ] + + scaling_config { + desired_size = 1 + max_size = 5 + min_size = 1 + } + + update_config { + max_unavailable = 2 + } + + tags = { + "name" = "eks-linux-node" + } +} + +### EKS Windows Node Group + +resource "aws_eks_node_group" "node_group_windows" { + cluster_name = aws_eks_cluster.eks_windows.name + node_group_name = "windows-managed-nodegroup" + node_role_arn = aws_iam_role.eks_node_group_role_windows.arn + ami_type = var.eks_windows_ami_version + instance_types = [var.ec2_instance_types] + subnet_ids = module.vpc.private_subnets_id + depends_on = [ + aws_iam_role_policy_attachment.eks_windows_node_group_role_attach + ] + + taint { + key = "windows" + value = "os" + effect = "NO_SCHEDULE" + } + + scaling_config { + desired_size = 1 + max_size = 5 + min_size = 1 + } + + update_config { + max_unavailable = 2 + } + + tags = { + "name" = "eks-windows-node" + } +} \ No newline at end of file diff --git a/terraform/aws/eks-windows/outputs.tf b/terraform/aws/eks-windows/outputs.tf new file mode 100644 index 0000000..bdab227 --- /dev/null +++ b/terraform/aws/eks-windows/outputs.tf @@ -0,0 +1,7 @@ +output "eks_cluster_name" { + value = aws_eks_cluster.eks_windows.name +} + +output "eks_cluster_status" { + value = aws_eks_cluster.eks_windows.status +} \ No newline at end of file diff --git a/terraform/aws/eks-windows/variables.tf b/terraform/aws/eks-windows/variables.tf new file mode 100644 index 0000000..4fdb11c --- /dev/null +++ b/terraform/aws/eks-windows/variables.tf @@ -0,0 +1,137 @@ +variable "eks_cluster_name" { + type = string + default = "eks-windows" + description = "Name of the EKS cluster" +} + +variable "endpoint_private_access" { + type = bool + default = true + description = "Indicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is false" +} + +variable "endpoint_public_access" { + type = bool + default = false + description = "Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true" +} + +variable "public_access_cidrs" { + type = list(string) + default = ["0.0.0.0/0"] + description = "Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0." +} + + +variable "enabled_cluster_log_types" { + type = list(string) + default = [] + description = "A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [`api`, `audit`, `authenticator`, `controllerManager`, `scheduler`]" +} + +variable "cluster_log_retention_period" { + type = number + default = 0 + description = "Number of days to retain cluster logs. Requires `enabled_cluster_log_types` to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html." +} + +variable "cluster_encryption_config_enabled" { + type = bool + default = true + description = "Set to `true` to enable Cluster Encryption Configuration" +} + +variable "cluster_encryption_config_kms_key_id" { + type = string + default = "" + description = "KMS Key ID to use for cluster encryption config" +} + +variable "cluster_encryption_config_kms_key_enable_key_rotation" { + type = bool + default = true + description = "Cluster Encryption Config KMS Key Resource argument - enable kms key rotation" +} + +variable "cluster_encryption_config_kms_key_deletion_window_in_days" { + type = number + default = 10 + description = "Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction" +} + +variable "cluster_encryption_config_kms_key_policy" { + type = string + default = null + description = "Cluster Encryption Config KMS Key Resource argument - key policy" +} + +variable "cluster_encryption_config_resources" { + type = list(any) + default = ["secrets"] + description = "Cluster Encryption Config Resources to encrypt, e.g. ['secrets']" +} + +variable "eks_cluster_version" { + type = string + default = "1.28" + description = "Version for the EKS cluster" +} + +variable "eks_windows_ami_version" { + type = string + default = "WINDOWS_CORE_2022_x86_64" + description = "Valid Values: AL2_x86_64 | AL2_x86_64_GPU | AL2_ARM_64 | CUSTOM | BOTTLEROCKET_ARM_64 | BOTTLEROCKET_x86_64 | BOTTLEROCKET_ARM_64_NVIDIA | BOTTLEROCKET_x86_64_NVIDIA | WINDOWS_CORE_2019_x86_64 | WINDOWS_FULL_2019_x86_64 | WINDOWS_CORE_2022_x86_64 | WINDOWS_FULL_2022_x86_64" +} + +variable "launch_template_name" { + type = string + default = "eks-windows-lt" + description = "Name for the launch template" +} + +variable "ec2_instance_types" { + type = string + default = "t3.medium" + description = "EC2 instance type" +} + +variable "eks_windows_workernode_instance_profile_name" { + type = string + default = "eks_windows_workernode_instance_profile" + description = "Worker node instance profile name" +} + +variable "alb_ingress_ports" { + type = list(number) + default = [80, 443] + description = "List of ports opened from Internet to ALB" +} + +variable "container_instances_ingress_ports" { + type = list(number) + default = [80, 443] + description = "List of ports opened from ALB to Container Instances" +} + +variable "kubelet_extra_args" { + type = string + default = "--register-with-taints='os=windows:NoSchedule'" + description = "This will make sure to taint your nodes at the boot time to avoid scheduling any existing resources in the new Windows worker nodes" +} + +variable "map_users" { + type = list(object({ + userarn = string + username = string + groups = list(string) + })) + + default = [ + { + userarn = "arn:aws:iam::0123456789:user/USER" + username = "momarcio" + groups = ["system:masters"] + }, + ] + description = "Additional IAM users to add to the aws-auth configmap." +} \ No newline at end of file diff --git a/terraform/aws/vpc/locals.tf b/terraform/aws/vpc/locals.tf new file mode 100644 index 0000000..7f51c50 --- /dev/null +++ b/terraform/aws/vpc/locals.tf @@ -0,0 +1,4 @@ +locals { + internet = "0.0.0.0/0" + +} diff --git a/terraform/aws/vpc/main.tf b/terraform/aws/vpc/main.tf new file mode 100644 index 0000000..7e004e7 --- /dev/null +++ b/terraform/aws/vpc/main.tf @@ -0,0 +1,122 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 5.0" + } + } +} + +## Data + +data "aws_availability_zones" "az" { + state = "available" + filter { + name = "opt-in-status" + values = ["opt-in-not-required"] + } +} + +## VPC + +resource "aws_vpc" "vpc" { + cidr_block = var.vpc_cidr_block + enable_dns_hostnames = true + enable_dns_support = true + tags = { + Name = "VPC for Amazon EKS cluster" + } +} + +## Private Subnets + +resource "aws_subnet" "private_subnets" { + vpc_id = aws_vpc.vpc.id + for_each = var.private_subnets + cidr_block = cidrsubnet(aws_vpc.vpc.cidr_block, 4, each.value) + availability_zone = each.key + + tags = { + Subnet = "Private Subnet ${each.key}-${each.value}" + Name = "Private Subnet / ${each.key}" + Tier = "Private" + "kubernetes.io/role/internal-elb" = 1 + } +} + +## Public Subnets + +resource "aws_subnet" "public_subnets" { + vpc_id = aws_vpc.vpc.id + for_each = var.public_subnets + cidr_block = cidrsubnet(aws_vpc.vpc.cidr_block, 4, each.value) + availability_zone = each.key + + tags = { + Subnet = "${each.key}-${each.value}" + Name = "Public Subnet / ${each.key}" + Tier = "Public" + "kubernetes.io/role/elb" = 1 + } +} + +## Internet Gateway + +resource "aws_internet_gateway" "internet_gateway" { + vpc_id = aws_vpc.vpc.id + tags = { + Name = "Internet Gateway" + } +} + +## Elastic IP for Nat Gateway + +resource "aws_eip" "eip_natgateway" { + domain = "vpc" + depends_on = [aws_internet_gateway.internet_gateway] + tags = { + Name = "Elastic IP for Nat Gateway" + } +} + +## Nat Gateway + +resource "aws_nat_gateway" "nat_gateway" { + allocation_id = aws_eip.eip_natgateway.id + # subnet_id = aws_subnet.public_subnets["us-east-1a"].id + subnet_id = aws_subnet.public_subnets[element(keys(aws_subnet.public_subnets), 0)].id #Accessing an specific value inside a for_each + depends_on = [aws_internet_gateway.internet_gateway] + tags = { + Name = "Nat Gateway" + } +} + +## Route Tables + +resource "aws_route_table" "private_subnets_route_table" { + vpc_id = aws_vpc.vpc.id + route { + cidr_block = local.internet + nat_gateway_id = aws_nat_gateway.nat_gateway.id + } +} + +resource "aws_route_table_association" "private_subnet_route_association" { + for_each = aws_subnet.private_subnets + subnet_id = each.value.id + route_table_id = aws_route_table.private_subnets_route_table.id +} + +resource "aws_route_table" "public_subnets_route_table" { + vpc_id = aws_vpc.vpc.id + route { + cidr_block = local.internet + gateway_id = aws_internet_gateway.internet_gateway.id + } +} + +resource "aws_route_table_association" "public_subnet_route_association" { + for_each = aws_subnet.public_subnets + subnet_id = each.value.id + route_table_id = aws_route_table.public_subnets_route_table.id +} \ No newline at end of file diff --git a/terraform/aws/vpc/output.tf b/terraform/aws/vpc/output.tf new file mode 100644 index 0000000..7184257 --- /dev/null +++ b/terraform/aws/vpc/output.tf @@ -0,0 +1,25 @@ +## VPC ID + +output "vpc_id" { + value = aws_vpc.vpc.id +} + +## Subnet IDs + +output "private_subnets_id" { + value = values(aws_subnet.private_subnets).*.id +} + +output "public_subnets_id" { + value = values(aws_subnet.public_subnets).*.id +} + +## Subnets CIDRs + +output "private_subnets_cidr" { + value = values(aws_subnet.private_subnets).*.cidr_block +} + +output "public_subnets_cidr" { + value = values(aws_subnet.public_subnets).*.cidr_block +} \ No newline at end of file diff --git a/terraform/aws/vpc/variable.tf b/terraform/aws/vpc/variable.tf new file mode 100644 index 0000000..ec7dfe1 --- /dev/null +++ b/terraform/aws/vpc/variable.tf @@ -0,0 +1,26 @@ +## VPC CIDR BLOCK +variable "vpc_cidr_block" { + default = "10.0.0.0/16" +} + +## Private Subnet CIDR BLOCK +variable "private_subnets" { + description = "Choose the AZs for the region of your choice" + type = map(number) + + default = { + "us-east-1a" = 1 + "us-east-1b" = 2 + } +} + +## Public Subnet CIDR BLOCK +variable "public_subnets" { + description = "Choose the AZs for the region of your choice" + type = map(number) + + default = { + "us-east-1a" = 3 + "us-east-1b" = 4 + } +} \ No newline at end of file diff --git a/terraform/data.tf b/terraform/data.tf deleted file mode 100644 index 1ad367b..0000000 --- a/terraform/data.tf +++ /dev/null @@ -1,16 +0,0 @@ -/* -Copyright 2023 The Kubernetes Authors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -data "aws_availability_zones" "available" {} - -data "aws_caller_identity" "current" {} diff --git a/terraform/locals.tf b/terraform/locals.tf deleted file mode 100644 index f1d50a7..0000000 --- a/terraform/locals.tf +++ /dev/null @@ -1,31 +0,0 @@ -/* -Copyright 2023 The Kubernetes Authors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -locals { - cluster_version = "1.28" - - azs = slice(data.aws_availability_zones.available.names, 0, 3) - vpc_cidr = "10.0.0.0/16" - - linux_node_group = "linux-node-group" - linux_instance_type = "t3.medium" - windows_node_group = "windows-node-group" - windows_ami_type = "WINDOWS_CORE_2022_x86_64" - windows_instance_type = "t3.large" - - tags = { - Cluster = var.cluster_name - GithubRepo = "sigs.k8s.io" - GithubOrg = "windows-operational-readiness" - } -} diff --git a/terraform/main.tf b/terraform/main.tf deleted file mode 100644 index bcd3c6e..0000000 --- a/terraform/main.tf +++ /dev/null @@ -1,166 +0,0 @@ -/* -Copyright 2023 The Kubernetes Authors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - - -################################################################################ -# IAM Permissions -################################################################################ - -resource "aws_iam_role_policy_attachment" "node_group_role_attach" { - for_each = toset([ - "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy", - "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", - "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy", - "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" - ]) - role = aws_iam_role.node_group_role.name - policy_arn = each.value -} - -resource "aws_iam_role" "node_group_role" { - name = "${local.linux_node_group}-role" - - assume_role_policy = jsonencode({ - Statement = [{ - Action = "sts:AssumeRole" - Effect = "Allow" - Principal = { - Service = "ec2.amazonaws.com" - } - }] - Version = "2012-10-17" - }) -} - -################################################################################ -# Supporting resources and networking -################################################################################ - -module "vpc" { - source = "terraform-aws-modules/vpc/aws" - version = "~> 4.0" - - name = "${var.cluster_name}-vpc" - cidr = local.vpc_cidr - - azs = local.azs - private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)] - public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)] - intra_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 52)] - - enable_nat_gateway = true - single_nat_gateway = true - - public_subnet_tags = { - "kubernetes.io/role/elb" = 1 - } - - private_subnet_tags = { - "kubernetes.io/role/internal-elb" = 1 - } - - tags = local.tags -} - -################################################################################ -# EKS Cluster main configuration -################################################################################ - -module "eks" { - source = "terraform-aws-modules/eks/aws" - cluster_name = var.cluster_name - cluster_version = local.cluster_version - cluster_endpoint_public_access = true - - vpc_id = module.vpc.vpc_id - subnet_ids = module.vpc.private_subnets - control_plane_subnet_ids = module.vpc.intra_subnets - - cluster_addons = { - coredns = { - most_recent = true - } - kube-proxy = { - most_recent = true - } - vpc-cni = { - most_recent = true - configuration_values = jsonencode({ - enableWindowsIpam : "true" - }) - } - } - - tags = local.tags -} - -################################################################################ -# Mixed Node group configuration -################################################################################ - -resource "aws_eks_node_group" "node_group_windows" { - node_group_name = local.windows_node_group - node_role_arn = aws_iam_role.node_group_role.arn - - cluster_name = module.eks.cluster_name - subnet_ids = module.vpc.private_subnets - depends_on = [ - aws_iam_role_policy_attachment.node_group_role_attach - ] - - ami_type = local.windows_ami_type - instance_types = [local.windows_instance_type] - - scaling_config { - desired_size = 1 - max_size = 5 - min_size = 1 - } - - update_config { - max_unavailable = 2 - } - - tags = merge( - { "node-group" : "windows" }, - local.tags, - ) -} - -resource "aws_eks_node_group" "node_group_linux" { - node_group_name = local.linux_node_group - node_role_arn = aws_iam_role.node_group_role.arn - - cluster_name = module.eks.cluster_name - subnet_ids = module.vpc.private_subnets - depends_on = [ - aws_iam_role_policy_attachment.node_group_role_attach - ] - - instance_types = [local.linux_instance_type] - - scaling_config { - desired_size = 3 - max_size = 5 - min_size = 1 - } - - update_config { - max_unavailable = 2 - } - - tags = merge( - { "node-group" : "linux" }, - local.tags, - ) -} diff --git a/terraform/provider.tf b/terraform/provider.tf deleted file mode 100644 index 0c62ac1..0000000 --- a/terraform/provider.tf +++ /dev/null @@ -1,16 +0,0 @@ -/* -Copyright 2023 The Kubernetes Authors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -provider "aws" { - region = var.aws_region -} diff --git a/terraform/terraform.tf b/terraform/terraform.tf deleted file mode 100644 index 693b653..0000000 --- a/terraform/terraform.tf +++ /dev/null @@ -1,22 +0,0 @@ -/* -Copyright 2023 The Kubernetes Authors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -terraform { - required_version = ">= 1.1.0" - required_providers { - aws = { - source = "hashicorp/aws" - version = "~> 5.26.0" - } - } -} diff --git a/terraform/variables.tf b/terraform/variables.tf deleted file mode 100644 index fc6868c..0000000 --- a/terraform/variables.tf +++ /dev/null @@ -1,24 +0,0 @@ -/* -Copyright 2023 The Kubernetes Authors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -variable "cluster_name" { - type = string - default = "eks-windows" - description = "EKS Cluster name" -} - -variable "aws_region" { - type = string - default = "us-east-1" - description = "EKS AWS Region" -} From faf7b83ba38a00b6adc3876a0763b09ceff6ffd7 Mon Sep 17 00:00:00 2001 From: Marcio Morales <marciogmorales@hotmail.com> Date: Wed, 29 Nov 2023 13:43:30 -0600 Subject: [PATCH 2/2] Refactored Amazon EKS terraform modules Refactored Amazon EKS terraform modules --- terraform/.gitignore | 2 -- terraform/aws/eks-windows/README.md | 34 +---------------------------- 2 files changed, 1 insertion(+), 35 deletions(-) delete mode 100644 terraform/.gitignore diff --git a/terraform/.gitignore b/terraform/.gitignore deleted file mode 100644 index 92c001d..0000000 --- a/terraform/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -terraform.tfstate* -.terraform* diff --git a/terraform/aws/eks-windows/README.md b/terraform/aws/eks-windows/README.md index 2040e75..e8bf354 100644 --- a/terraform/aws/eks-windows/README.md +++ b/terraform/aws/eks-windows/README.md @@ -82,36 +82,4 @@ aws eks update-kubeconfig --region us-east-1 --name eks-windows A few other resources can be consulted in case of doubts or slight modification: * [Official EKS Documentation](https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html) -* [Running Windows Containers on AWS: A complete guide to successfully running Windows containers on Amazon ECS, EKS, and AWS Fargate](https://www.amazon.com/Running-Windows-Containers-AWS-successfully/dp/1804614130) - - - ----- - - - -## Providers - -- hashicorp/aws | version = "~> 45.0" - -## Variables description -- **eks_cluster_name (string)**: Namne of the EKS cluster -- **endpoint_private_access (bool)**: Indicates whether or not the Amazon EKS private API server endpoint is enabled -- **endpoint_public_access (bool)**: Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true -- **public_access_cidrs (list(string))**: Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0. -- **enabled_cluster_log_types (list(string))**: A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [`api`, `audit`, `authenticator`, `controllerManager`, `scheduler`] -- **cluster_log_retention_period (number)**: Number of days to retain cluster logs. Requires `enabled_cluster_log_types` to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. -- **cluster_encryption_config_enabled (bool)**: Set to `true` to enable Cluster Encryption Configuration -- **cluster_encryption_config_kms_key_id (string)**: KMS Key ID to use for cluster encryption config -- **cluster_encryption_config_kms_key_enable_key_rotation (bool)**: Cluster Encryption Config KMS Key Resource argument - enable kms key rotation -- **cluster_encryption_config_kms_key_deletion_window_in_days (number)**: Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction -- **cluster_encryption_config_kms_key_policy (string)**: Cluster Encryption Config KMS Key Resource argument - key policy -- **cluster_encryption_config_resources (list(any))**: Cluster Encryption Config Resources to encrypt, e.g. ['secrets'] -- **eks_cluster_version (string)**: Version for the EKS cluster -- **launch_template_name (string)**: Name for the launch template -- **ec2_instance_types (string)**: EC2 instance type -- **eks_windows_workernode_instance_profile_name (string)**: Worker node instance profile name -- **alb_ingress_ports (list(number))**: List of ports opened from Internet to ALB -- **container_instances_ingress_ports (list(number))**: List of ports opened from ALB to Container Instances -- **kubelet_extra_args (string)**: This will make sure to taint your nodes at the boot time to avoid scheduling any existing resources in the new Windows worker nodes -- **map_users (list(object({})))**: Additional IAM users to add to the aws-auth configmap. +* [Running Windows Containers on AWS: A complete guide to successfully running Windows containers on Amazon ECS, EKS, and AWS Fargate](https://www.amazon.com/Running-Windows-Containers-AWS-successfully/dp/1804614130) \ No newline at end of file