[Bug]: When using Managed NG and Self-Managed NG, the Security Group does not allow Port 53 by default

[darrenlin]

Welcome to Amazon EKS Blueprints!

• Yes, I've searched similar issues on GitHub and didn't find any.

Amazon EKS Blueprints Release version

4.0.9

What is your environment, configuration and the example used?

I tried to create two node groups. One is managed node group, and another is self-managed node group (for local zones and outpost, but they are not required to reproduce the issue).

I am using the example below.

module "eks_blueprints" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints"

  # EKS CLUSTER
  cluster_version    = "1.22"
  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets
  cluster_name       = local.cluster_name

  # EKS MANAGED NODE GROUPS
  managed_node_groups = {
    managed_ng = {
      node_group_name = "managed-ng"
      instance_types  = ["m5.large"]
      subnet_ids      = module.vpc.private_subnets

      min_size = 1
    }
  }

  self_managed_node_groups = {
    self_mg_5 = {
      node_group_name    = "self-managed-ondemand"
      launch_template_os = "amazonlinux2eks"
      subnet_ids         = module.vpc.private_subnets
      min_size = 1

    }
  }
}

What did you do and What did you see instead?

I found that:

• self-managed node groups uses "EKS node shared security group" (sg-094ab4408afdddb9b)
• managed node groups uses "EKS created security group applied to ENI that is attached to EKS Control Plane master nodes, as well as any managed workloads." (sg-04f3aadaccae6341d)

➜  aws ec2 describe-security-groups --group-ids sg-094ab4408afdddb9b --query 'SecurityGroups[0].{Inbound:IpPermissions,Outbound:IpPermissionsEgress}' --output yaml
Inbound:
- FromPort: 10250
  IpProtocol: tcp
  IpRanges: []
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 10250
  UserIdGroupPairs:
  - Description: Cluster API to node kubelets
    GroupId: sg-09bfcd29517d60b58
    UserId: '091550601287'
- FromPort: 53
  IpProtocol: tcp
  IpRanges: []
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 53
  UserIdGroupPairs:
  - Description: Node to node CoreDNS
    GroupId: sg-094ab4408afdddb9b
    UserId: '091550601287'
- FromPort: 53
  IpProtocol: udp
  IpRanges: []
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 53
  UserIdGroupPairs:
  - Description: Node to node CoreDNS
    GroupId: sg-094ab4408afdddb9b
    UserId: '091550601287'
- FromPort: 443
  IpProtocol: tcp
  IpRanges: []
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 443
  UserIdGroupPairs:
  - Description: Cluster API to node groups
    GroupId: sg-09bfcd29517d60b58
    UserId: '091550601287'
Outbound:
- FromPort: 53
  IpProtocol: tcp
  IpRanges: []
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 53
  UserIdGroupPairs:
  - Description: Node to node CoreDNS
    GroupId: sg-094ab4408afdddb9b
    UserId: '091550601287'
- FromPort: 53
  IpProtocol: udp
  IpRanges: []
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 53
  UserIdGroupPairs:
  - Description: Node to node CoreDNS
    GroupId: sg-094ab4408afdddb9b
    UserId: '091550601287'
- FromPort: 123
  IpProtocol: udp
  IpRanges:
  - CidrIp: 0.0.0.0/0
    Description: Egress NTP/UDP to internet
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 123
  UserIdGroupPairs: []
- FromPort: 443
  IpProtocol: tcp
  IpRanges:
  - CidrIp: 0.0.0.0/0
    Description: Egress all HTTPS to internet
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 443
  UserIdGroupPairs:
  - Description: Node groups to cluster API
    GroupId: sg-09bfcd29517d60b58
    UserId: '091550601287'
- FromPort: 123
  IpProtocol: tcp
  IpRanges:
  - CidrIp: 0.0.0.0/0
    Description: Egress NTP/TCP to internet
  Ipv6Ranges: []
  PrefixListIds: []
  ToPort: 123
  UserIdGroupPairs: []

However, the security group rules does not allow UDP 53 data transfer, and will result errors for add-on installations and difficult to debug for new customers.

Additional Information

The security groups and rules are nicely created when I use eksctl to create self-managed node groups.

No response

[darrenlin]

I fixed the issue on my own using additional security group rules below:

module "eks_blueprints" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints"

  # EKS CLUSTER
  cluster_version    = "1.22"
  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets
  cluster_name       = local.cluster_name

  # EKS MANAGED NODE GROUPS
  managed_node_groups = {
    managed_ng = {
      node_group_name = "managed-ng"
      instance_types  = ["m5.large"]
      subnet_ids      = module.vpc.private_subnets

      min_size = 1
    }
  }

  self_managed_node_groups = {
    self_mg_5 = {
      node_group_name    = "self-managed-ondemand"
      launch_template_os = "amazonlinux2eks"
      subnet_ids         = module.vpc.private_subnets
      min_size           = 1

    }
  }

  node_security_group_additional_rules = {
    # Extend node-to-node security group rules. Recommended and required for the Add-ons
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    # Recommended outbound traffic for Node groups
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["0.0.0.0/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
    # Allows Control Plane Nodes to talk to Worker nodes on all ports. Added this to simplify the example and further avoid issues with Add-ons communication with Control plane.
    # This can be restricted further to specific port based on the requirement for each Add-on e.g., metrics-server 4443, spark-operator 8080, karpenter 8443 etc.
    # Change this according to your security requirements if needed
    ingress_cluster_to_node_all_traffic = {
      description              = "Cluster API to Nodegroup all traffic"
      protocol                 = "-1"
      from_port                = 0
      to_port                  = 0
      type                     = "ingress"
      source_security_group_id = module.eks_blueprints.cluster_security_group_id
    }
  }

resource "aws_security_group_rule" "allow_node_sg_to_cluster_sg" {
  description = "Self-Node Group to Cluster API/MNG all traffic"

  source_security_group_id = module.eks_blueprints.worker_node_security_group_id
  security_group_id        = module.eks_blueprints.cluster_primary_security_group_id
  type                     = "ingress"
  protocol                 = "-1"
  from_port                = 0
  to_port                  = 0

  depends_on = [
    module.eks_blueprints
  ]
}

resource "aws_security_group_rule" "allow_node_sg_from_cluster_sg" {
  description = "Cluster API/MNG to Self-Nodegroup all traffic"
  source_security_group_id = module.eks_blueprints.cluster_primary_security_group_id
  security_group_id        = module.eks_blueprints.worker_node_security_group_id
  type                     = "ingress"
  protocol                 = "-1"
  from_port                = 0
  to_port                  = 0

  depends_on = [
    module.eks_blueprints
  ]
}

[askulkarni2]

Hi @darrenlin, this is a known "issue" and is by design in the upstream module. As a security best practice only the ports required are opened. See

terraform-aws-eks-blueprints/examples/complete-kubernetes-addons/main.tf

Line 60 in 15e1575

#----------------------------------------------------------------------------------------------------------#

. Thanks for providing the workaround.

[darrenlin]

I think attaching the primary cluster security group will solve this issue
terraform-aws-modules/terraform-aws-eks#1952

Is it possible to add this as a variable of EKS Blueprint module?