feat: Update Terraform getting started guide to default to multi-cluster tagging scheme #1668

bryantbiggs · 2022-04-12T19:43:08Z

1. Issue, if available:
#1488

2. Description of changes:

We have updated the upstream IRSA module to allow users to control the IAM policy condition tag key in order to support multiple clusters in the same account, each with their own Karpenter provisioner. See more details/discussion near the bottom of Could not launch node, launching instances, with fleet error(s), UnauthorizedOperation: You are not authorized to perform this operation. #1488
A subsequent change was made to relax tagging requirements/restrictions for VPC subnets - fix: Avoid restricting Karpenter RunInstances subnets by tag key terraform-aws-modules/terraform-aws-iam#247. Users no longer need to adhere to strict subnet tagging and instead just supply the Karpenter provision with a subnetSelector that will match the subnets to use. Only launch template and security group are restricted by the Karpenter controller policy (and by extension, EC2 instances launched from the launch template)
Module used updated to latest and Kubernetes version used updated to 1.22

3. How was this change tested?

Code provided in Terraform getting started guide was used to provision a cluster and Karpenter components. The example deployment was used to validate Karpenter launching instances as requested

4. Does this change impact docs?

Yes, PR includes docs updates
Yes, issue opened: link to issue
No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

netlify · 2022-04-12T19:43:13Z

✅ Deploy Preview for karpenter-docs-prod ready!

Name	Link
🔨 Latest commit	`1f9beb4`
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/62964600365c84000868f0b6
😎 Deploy Preview	https://deploy-preview-1668--karpenter-docs-prod.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

bryantbiggs · 2022-04-12T19:43:44Z

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md

@@ -361,11 +362,11 @@ resource "kubectl_manifest" "karpenter_provisioner" {
        cpu: 1000
    provider:
      subnetSelector:
-        karpenter.sh/discovery: ${local.cluster_name}
+        karpenter.sh/discovery: ${module.eks.cluster_id}


local.cluster_name and module.eks.cluster_id are one in the same. Its better to use the module output though to ensure proper dependency ordering

dewjam · 2022-04-14T15:36:13Z

Planning to test this procedure today. I'll let you know what I find.

dewjam

I ran through these changes and am getting permissions denied errors. I'm fairly certain this is due to the tags which are set on the Subnets and Security groups when creating the EKS cluster.

Subnets and SecurityGroups have a label of karpenter.sh/discovery: <cluster_name>, while the condition in the IRSA module is using karpenter.sh/discovery/cluster_name/<cluster_name>: <cluster_name.

https://github.com/terraform-aws-modules/terraform-aws-iam/blob/master/modules/iam-role-for-service-accounts-eks/policies.tf#L548-L561

bryantbiggs · 2022-04-15T16:52:36Z

Let me take a closer look at this - I thought this would be a no-op change but looks like thats not the case

…ter tagging scheme

…subnet support

bryantbiggs · 2022-05-10T17:57:34Z

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md

  private_subnet_tags = {
-    "kubernetes.io/cluster/${local.cluster_name}" = "owned"
-    # Tags subnets for Karpenter auto-discovery
-    "karpenter.sh/discovery" = local.cluster_name


This karpenter unique tag requirement is lifted with terraform-aws-modules/terraform-aws-iam#247 - users just need to specify a subnetSelector that will match the subnets they intend to use. VPCs and Subnets are shared resources and we shouldn't impose specific tagging on those

bryantbiggs · 2022-05-10T17:58:42Z

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md

-  create_cluster_security_group = false
-  create_node_security_group    = false
+  node_security_group_additional_rules = {
+    ingress_nodes_karpenter_port = {


This has come up a bit so I thought it would be better to show the minimum additional security group rules required to run Karpenter. Hopefully this highlights to users that in addition to the minimum ports that Kubernetes/EKS requires, port 8443/TCP from the control plane to the node group is required

bryantbiggs · 2022-05-10T18:00:18Z

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md

      securityGroupSelector:
-        karpenter.sh/discovery: ${local.cluster_name}
+        karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}


The cluster unique key is directly related to #1488 - this allows users to have multiple clusters in a single account without tag conflicts and still maintain properly scoped permissions.

…clustername Reference: aws/karpenter-provider-aws#1668

njtran · 2022-05-31T15:58:45Z

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md


  cluster_name    = local.cluster_name
-  cluster_version = "1.21"
+  cluster_version = "1.22"


What's the ownership model of this file? How can we ensure this and L125 are up-to-date?

I can see the cluster_version being updated when the default cluster versions need to be updated, but for the TF version, do we need to specify a version if the terraform-aws-modules on L123 are referred to as latest?

Same goes for L237

If we can extract the HCL out of the README and put it into actual *.tf files somewhat similar to what was done for CloudFormation where the CloudFormation is injected via the template tags, that will give us 2 things:

Its easier to test and validate these changes because we can cd to the directory that holds the Terraform file and simply terraform init && terraform apply

Once the contents are back into Terraform files, we could enable Renovate which will track module updates and submit PRs to update Update Terraform terraform-aws-modules/iam/aws to v5 DrFaust92/terraform-kubernetes-ebs-csi-driver#78

In general though, the values are set statically to ensure that the guidance for users has been validated first. In theory, we can leave off line 128 and the underlying EKS module/service will pull the latest K8s version offered. However, this can be problematic if changes are required for supporting the latest version.

Theres also this that I created re: ownership #1574 - from an outsider perspective, all I really need is one thing: the IAM policy required to run Karpenter (the most minimal, restrictive, and finely scoped IAM policy). With that I can host that policy in the IRSA role for Karpenter and then find the appropriate location to host the example usage instead of always trying to update the getting started guides here for the various ways one can provision an EKS cluster+Karpenter

In general though, the values are set statically to ensure that the guidance for users has been validated first.

I agree this is definitely a good system. It's not easy to make sure it's always up to date and works, which gives value to hosting this in another repo like you've done. Was mostly curious if you had a good solution :)

njtran · 2022-05-31T16:03:56Z

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md

+  node_security_group_tags = {
+    # NOTE - if creating multiple security groups with this module, only tag the
+    # security group that Karpenter should utilize with the following tag
+    # (i.e. - at most, only one security group should have this tag in your account)
+    "karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
+  }


Can you include what the consequences of not following this note are?

Suggested change

node_security_group_tags = {

# NOTE - if creating multiple security groups with this module, only tag the

# security group that Karpenter should utilize with the following tag

# (i.e. - at most, only one security group should have this tag in your account)

"karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name

}

node_security_group_tags = {

# NOTE - if creating multiple security groups with this module, only tag the

# security group that Karpenter should utilize with the following tag

# (i.e. at most, only one security group should have this tag in your account

# as <insert_reason_for_note>)

"karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name

}

website/content/en/preview/getting-started/getting-started-with-terraform/_index.md

dewjam · 2022-06-15T14:53:39Z

I've tested this procedure and it works as expected. @bryantbiggs I think there is one comment from @njtran which could be addressed, otherwise it looks good to me.

bwagner5

lgtm

bryantbiggs requested a review from a team as a code owner April 12, 2022 19:43

bryantbiggs requested a review from suket22 April 12, 2022 19:43

bryantbiggs commented Apr 12, 2022

View reviewed changes

dewjam reviewed Apr 15, 2022

View reviewed changes

bryantbiggs marked this pull request as draft April 15, 2022 16:52

bryantbiggs added 2 commits May 10, 2022 09:48

feat: Update Terraform getting started guide to default to multi-clus…

6eb75b1

…ter tagging scheme

feat: Update from testing to ensure multi-cluster support and shared …

61385c0

…subnet support

bryantbiggs commented May 10, 2022

View reviewed changes

bryantbiggs marked this pull request as ready for review May 10, 2022 18:00

bryantbiggs mentioned this pull request May 22, 2022

feat: support cluster_security_group_tags and node_security_group_tags aws-ia/terraform-aws-eks-blueprints#557

Merged

6 tasks

armujahid added a commit to armujahid/terraform-aws-eks-blueprints that referenced this pull request May 23, 2022

examples/karpenter: make karpenter discovery tag unique by appending …

26ca8e8

…clustername Reference: aws/karpenter-provider-aws#1668

armujahid added a commit to armujahid/terraform-aws-eks-blueprints that referenced this pull request May 23, 2022

examples/karpenter: make karpenter discovery tag unique by appending …

77d4ced

…clustername Reference: aws/karpenter-provider-aws#1668

njtran reviewed May 31, 2022

View reviewed changes

bryantbiggs added 2 commits May 31, 2022 12:43

Merge branch 'main' of github.com:aws/karpenter into feat/custom-tag-key

633fc9d

chore: updates for latest and PR feedback

1f9beb4

bwagner5 approved these changes Jul 11, 2022

View reviewed changes

bwagner5 merged commit 3643910 into aws:main Jul 11, 2022

bryantbiggs deleted the feat/custom-tag-key branch July 11, 2022 21:12

This was referenced Jul 13, 2022

Fix failing lint checks #2128

Closed

feat: Adding Custom AMIFamily and validations #2154

Merged

suket22 mentioned this pull request Jul 21, 2022

docs: Add docs for customAMIs, tests for UserData #2169

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Update Terraform getting started guide to default to multi-cluster tagging scheme #1668

feat: Update Terraform getting started guide to default to multi-cluster tagging scheme #1668

bryantbiggs commented Apr 12, 2022 •

edited

Loading

netlify bot commented Apr 12, 2022 •

edited

Loading

bryantbiggs Apr 12, 2022

dewjam commented Apr 14, 2022

dewjam left a comment

bryantbiggs commented Apr 15, 2022

bryantbiggs May 10, 2022

bryantbiggs May 10, 2022

dewjam May 17, 2022

bryantbiggs May 10, 2022 •

edited

Loading

njtran May 31, 2022

njtran May 31, 2022

bryantbiggs May 31, 2022

bryantbiggs May 31, 2022 •

edited

Loading

njtran May 31, 2022

njtran May 31, 2022

dewjam commented Jun 15, 2022

bwagner5 left a comment

feat: Update Terraform getting started guide to default to multi-cluster tagging scheme #1668

feat: Update Terraform getting started guide to default to multi-cluster tagging scheme #1668

Conversation

bryantbiggs commented Apr 12, 2022 • edited Loading

netlify bot commented Apr 12, 2022 • edited Loading

✅ Deploy Preview for karpenter-docs-prod ready!

Choose a reason for hiding this comment

dewjam commented Apr 14, 2022

dewjam left a comment

Choose a reason for hiding this comment

bryantbiggs commented Apr 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs May 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs May 31, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dewjam commented Jun 15, 2022

bwagner5 left a comment

Choose a reason for hiding this comment

bryantbiggs commented Apr 12, 2022 •

edited

Loading

netlify bot commented Apr 12, 2022 •

edited

Loading

bryantbiggs May 10, 2022 •

edited

Loading

bryantbiggs May 31, 2022 •

edited

Loading