EKS Capsule Demo

EKS Capsule Demo Table of Contents

• What is Capsule you ask?
• What Problem are we Trying to Solve?
• Demo Assumptions
• Cluster and IAM Resource Creation
• Cluster Configuration
• Testing Global Permissions
  • Test Host Duplication Restrictions

This is a demo of Capsule multi-tenancy management running on an AWS EKS cluster.

What is Capsule you ask?

The Capsule Documentation Overview section puts it quite succinctly:

"Capsule implements a multi-tenant and policy-based environment in your Kubernetes cluster. It is designed as a micro-services-based ecosystem with the minimalist approach, leveraging only on upstream Kubernetes."

The Capsule policy engine allows for a wide range of controls over what resources individual tenants can access such as storageclasses, nodegroups, ingresses, to resources limits, etc. It can assign additional metadata on tenant created objects.

What Problem are we Trying to Solve?

Often developers inadvertently create ingresses with host entries identical to those already in use somewhere in the cluster. This can become a problem in a multi-tenant Kubernetes environment. This demo shows one tool you can use to eliminate that problem across the entire cluster.

Demo Assumptions

This demo is written making the following assumptions:

1. You are running macOS or Linux

2. You have an AWS account and permissions to fully manage:

   1. CloudFormation stacksets

   2. EKS clusters

   3. IAM policies, roles, and users

3. You have the following CLI tools installed and available in your $PATH

   1. AWS CLI (With the default profile in the ~/.aws/ files configured with credentials that have the above permissions)

   2. eksctl CLI tool to easily create/delete EKS clusters

   3. git

   4. helm

   5. kubectl

Cluster and IAM Resource Creation

Clone this repo and cd into the new directory

git clone https://github.com/cgahlon/eks-capsule-demo.git
cd eks-capsule-demo

Clear the AWS CLI environment variables to avoid identity confusion. This demo is presented to use only the contents of ~/.aws/ files for authentication. Revising authentication methods used in this demo and providing examples is on my to-do list.

unset AWS_PROFILE
unset AWS_SECRET_ACCESS_KEY
unset AWS_ACCESS_KEY_ID

Create a small test cluster using this command.

NOTE

This step can take 20 about minutes on average. It varies based on time of day/load in the AWS region it is being deployed in. While you wait, read the next step’s instructions. It will save you some time.

eksctl create cluster --name=test-k8s --managed --node-type=t3.small --node-volume-size=20 --kubeconfig=kubeconfig.conf

Create a test IAM user and associated permissions. Optionally, you can run this in another terminal while waiting for the cluster and nodegroup to be created.

eval aws cloudformation deploy --capabilities CAPABILITY_NAMED_IAM --parameter-overrides "ClusterName=test-k8s" --stack-name "test-k8s-users" --template-file cloudformation/cluster-users.cf

Cluster Configuration

Create an identity mapping in the kube-system/aws-auth configmap for the 'alice' user.

export AWS_CLOUDFORMATION_DETAILS=$(aws cloudformation describe-stacks --stack-name "test-k8s-users" --output json)
export ALICE_ROLE_ARN=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"RoleAliceArn\") .OutputValue")
export ALICE_USER_ACCESSKEY=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"AccessKeyAlice\") .OutputValue")
export ALICE_USER_SECRETACCESSKEY=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"SecretAccessKeyAlice\") .OutputValue")

eksctl create iamidentitymapping --cluster="test-k8s" --arn="${ALICE_ROLE_ARN}" --username alice --group capsule.clastix.io

Append the test user’s credentials and profile information to the AWS CLI config files.

file: ~/.aws/config

cat >> ~/.aws/config << EOF
[profile alice]
role_arn=${ALICE_ROLE_ARN}
source_profile=alice
EOF

file: ~/.aws/credentials

cat >> ~/.aws/credentials << EOF
[alice]
aws_access_key_id=${ALICE_USER_ACCESSKEY}
aws_secret_access_key=${ALICE_USER_SECRETACCESSKEY}
EOF

Deploy Capsule using Helm

helm repo add clastix https://clastix.github.io/charts
helm repo update
helm upgrade --install --namespace capsule-system --create-namespace capsule clastix/capsule --kubeconfig=kubeconfig.conf

Create a Capsule tenant

kubectl apply -f manifests/capsule/tenant.yaml --kubeconfig="kubeconfig.conf"

Testing Global Permissions

Tenant users will have permission to create new namespaces but cannot create other resources outside of those namespaces

AWS_PROFILE=alice kubectl auth can-i create namespaces --kubeconfig="kubeconfig.conf"
AWS_PROFILE=alice kubectl auth can-i create ingresses --kubeconfig="kubeconfig.conf"

Test Host Duplication Restrictions

Create a namespace as 'alice' and deploy the demo-app ingress in it

AWS_PROFILE=alice kubectl create ns oil-production --kubeconfig="kubeconfig.conf"
AWS_PROFILE=alice kubectl apply -n oil-production -f manifests/demo-apps/app-oil-production.yaml --kubeconfig="kubeconfig.conf"

Later that week Alice tries to create another ingress in the "oil-development" namespace. However, she copied and pasted the ingress manifest from production and forgot to change the ingress host entry. Capsule will block the creation of an ingress with a duplicate host based on the scope defined in the tenant config.

AWS_PROFILE=alice kubectl create namespace oil-development --kubeconfig="kubeconfig.conf"
AWS_PROFILE=alice kubectl -n oil-development apply -f manifests/demo-apps/app-oil-development.yaml --kubeconfig="kubeconfig.conf"

When you try to apply the manifests/demo-apps/app-oil-development.yaml file you end up getting denied with a message similar to this:

Error from server (Forbidden): error when creating "manifests/demo-apps/app-oil-development.yaml": admission webhook "ingress.capsule.clastix.io" denied the request: hostname web.oil.acmecorp.com is already used across the cluster: please, reach out to the system administrators

Even cluster admins are not allowed to duplicate host names in a capsule tenant’s namespaces. That is, those namespaces with a "capsule.clastix.io/tenant=<TENANT_NAME>" label.

kubectl apply -f manifests/demo-apps/app-oil-admin.yaml -n oil-development --kubeconfig="kubeconfig.conf"

Error from server (Forbidden): error when creating "manifests/demo-apps/app-oil-admin.yaml": admission webhook "ingress.capsule.clastix.io" denied the request: hostname web.oil.acmecorp.com is already used across the cluster: please, reach out to the system administrators

NOTE

This restriction on cluster admins is only scoped to namespaces automatically labeled by Capsule with capsule.clastix.io/tenant=<TENANT_NAME> when a tenant creates the NS. Any unlabeled namespace can have an ingress with a duplicate host.

References / Credits

Many of the commands and code in this demo are derived from code in the Clastix/Capsule general tutorial and their EKS specific examples.

• Capsule Tutorial

• Capsule on AWS EKS Guide

• Capsule on GitHub