docs: add example for node draining

examples/node_draining/README.md

@@ -0,0 +1,63 @@
+## EKS node drainer with terraform
+
+Changing the AMI version of worker group or changing the kubernetes version will cause recreation of the nodes.
+By default the nodes won't drain themselves before they got removed. So we have some downtime.
+Node groups are a good alternative but not sufficient for us as we require to [set custom security groups](https://github.com/aws/containers-roadmap/issues/609).
+
+We are adding a termination lifecycle to run kubectl drain before we shutdown nodes.
+
+The serverless python drain function is from https://github.com/aws-samples/amazon-k8s-node-drainer
+and translated to terraform. 
+
+**STEPS**
+* build python zip https://docs.aws.amazon.com/lambda/latest/dg/python-package.html
+```
+# script need to be adapted to your system (python 3 version)
+# you will find better solution to deploy serverless function but it serves the example purpose
+./build.sh
+```
+* apply
+```
+terraform init
+terraform apply
+# will fail once because the subnets are not yet in the data filter
+# solving subnet dependencies needs to happen on other layer but not important for this example
+terraform apply
+```
+
+### Testing seamless worker upgrade
+
+* update kubeconfig and deploy example grafana with pod disruption budget
+```
+aws eks update-kubeconfig --name $CLUSTER_NAME --alias drainer
+
+# optional install latest cni plugin to ensure we can destroy cluster clean
+# https://github.com/terraform-aws-modules/terraform-aws-eks/issues/285
+# https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html
+kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.6/config/v1.6/aws-k8s-cni.yaml
+
+# install metrics server to watch node resource allocation
+helm upgrade --install grafana stable/grafana --set podDisruptionBudget.minAvailable=1 --set replicas=2 --set persistence.enabled=true --set persistence.type=statefulset
+
+# check that volumes are allocated in different regions
+kubectl get pv -o custom-columns=PVC-NAME:.spec.claimRef.name,REGION:.metadata.labels
+
+kubectl get pods
+```
+* change version number of the ami version of nodes to see node drainer in action
+* you can also verify the output in cloudwatch of the lambda function
+```
+terraform apply -var ami_version=v20200609
+```
+* now the nodes will not get deleted before they have been drained completely
+* the drainer will respect pod disruption budget in our example that's one running grafana replica
+
+### Drawbacks
+* terminating instances will just continue after the lifecycle timeout is reaching regardless of failure during draining 
+
+### Info
+* this example shows an example for a single AZ workergroup which is necessary if you are using EBS volumes with Statefulsets.
+    * not really necessary for node draining remove if you want
+* the drainer works also in combination with cluster-autoscaler
+    * how to use cluster-autoscaler is already well documented in [examples/irsa](../irsa)
+    * a full example with no guarantees can we found at [eks-node-drainer](https://github.com/karlderkaefer/eks-node-drainer)

examples/node_draining/build.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+cd drainer || exit 1
+mkdir -p dist
+cp -R src/*.py dist/
+cd src || exit 1
+python3 -m venv v-env
+source v-env/bin/activate
+pip install -r requirements.txt
+deactivate
+cp -R v-env/lib/python3.5/site-packages/* ../dist/
+cd ../.. || return

examples/node_draining/drainer.tf

@@ -0,0 +1,116 @@
+data "archive_file" "node_drainer" {
+  count       = var.drainer_enabled ? 1 : 0
+  type        = "zip"
+  source_dir  = "${path.module}/drainer/dist"
+  output_path = "${path.module}/lambda_function.zip"
+}
+
+resource "aws_iam_role" "node_drainer" {
+  count              = var.drainer_enabled ? 1 : 0
+  name               = "NodeDrainerRole"
+  assume_role_policy = data.aws_iam_policy_document.node_drainer_assume_role[0].json
+}
+
+data "aws_iam_policy_document" "node_drainer_assume_role" {
+  count = var.drainer_enabled ? 1 : 0
+  statement {
+    sid    = "AssumeRolePolicy"
+    effect = "Allow"
+    actions = [
+      "sts:AssumeRole"
+    ]
+    principals {
+      type        = "Service"
+      identifiers = ["lambda.amazonaws.com"]
+    }
+  }
+}
+
+data "aws_iam_policy_document" "node_drainer" {
+  count = var.drainer_enabled ? 1 : 0
+  statement {
+    sid    = "LoggingPolicy"
+    effect = "Allow"
+    actions = [
+      "logs:CreateLogGroup",
+      "logs:CreateLogStream",
+      "logs:PutLogEvents",
+    ]
+    resources = [
+      "arn:aws:logs:*:*:*"
+    ]
+  }
+  statement {
+    sid    = "AutoscalePolicy"
+    effect = "Allow"
+    actions = [
+      "autoscaling:CompleteLifecycleAction",
+      "ec2:DescribeInstances",
+      "eks:DescribeCluster",
+      "sts:GetCallerIdentity",
+    ]
+    resources = [
+      "*"
+    ]
+  }
+}
+
+resource "aws_iam_role_policy" "node_drainer" {
+  count  = var.drainer_enabled ? 1 : 0
+  role   = aws_iam_role.node_drainer[0].id
+  policy = data.aws_iam_policy_document.node_drainer[0].json
+}
+
+resource "aws_lambda_function" "node_drainer" {
+  count         = var.drainer_enabled ? 1 : 0
+  filename      = data.archive_file.node_drainer[0].output_path
+  function_name = var.drainer_lambda_function_name
+  role          = aws_iam_role.node_drainer[0].arn
+  handler       = "handler.lambda_handler"
+  memory_size   = 300
+  timeout       = var.drainer_lambda_timeout
+
+  source_code_hash = filebase64sha256(data.archive_file.node_drainer[0].output_path)
+
+  runtime = "python3.7"
+
+  environment {
+    variables = {
+      CLUSTER_NAME = var.cluster_name
+    }
+  }
+  depends_on = [
+    aws_iam_role.node_drainer,
+    aws_cloudwatch_log_group.node_drainer,
+    data.archive_file.node_drainer,
+  ]
+}
+
+# This is to optionally manage the CloudWatch Log Group for the Lambda Function.
+# If skipping this resource configuration, also add "logs:CreateLogGroup" to the IAM policy below.
+resource "aws_cloudwatch_log_group" "node_drainer" {
+  count             = var.drainer_enabled ? 1 : 0
+  name              = "/aws/lambda/${var.drainer_lambda_function_name}"
+  retention_in_days = 14
+}
+
+resource "aws_lambda_permission" "node_drainer" {
+  count         = var.drainer_enabled ? 1 : 0
+  action        = "lambda:InvokeFunction"
+  function_name = aws_lambda_function.node_drainer[0].function_name
+  principal     = "events.amazonaws.com"
+}
+
+resource "aws_cloudwatch_event_rule" "terminating_events" {
+  count       = var.drainer_enabled ? 1 : 0
+  name        = "asg-terminate-events-${var.cluster_name}"
+  description = "Capture all terminating autoscaling events for cluster ${var.cluster_name}"
+
+  event_pattern = templatefile("${path.module}/event-rule.tpl", { cluster_name = var.cluster_name })
+}
+
+resource "aws_cloudwatch_event_target" "terminating_events" {
+  count = var.drainer_enabled ? 1 : 0
+  rule  = aws_cloudwatch_event_rule.terminating_events[0].name
+  arn   = aws_lambda_function.node_drainer[0].arn
+}

examples/node_draining/drainer/__init__.py

@@ -0,0 +1,4 @@
+import os
+import sys
+
+sys.path.append(os.path.dirname(os.path.realpath(__file__)))