403-admission-policy

Admission Control for Kubernetes on AWS

Table of Contents

• Goals
• Prerequisites
• Overview
• Step 1: Create Cluster with External Admission Controllers via kops
• Step 2: Deploy OPA on Kubernetes
  • 2.1 Generate service-certificate for OPA
  • 2.2: Deploy OPA on Kubernetes
• Step 3: Enforce Kubernetes Admission Control with OPA
  • 3.1 Load a policy into OPA
  • 3.2 Check that the policy is working
• Wrap Up

Doc Writer <[email protected]> v1.0, 2018-01-17

Kubernetes Admission Controllers perform semantic validation of resources during create, update, and delete operations. In Kubernetes 1.8+, you can use the Open Policy Agent and Kubernetes External Admission Webhooks to enforce custom admission control policies without recompiling or reconfiguring the Kubernetes API server.

The Open Policy Agent is a general-purpose policy engine that aims to policy-enable other projects or services. It provides a rich, declarative policy language where policy decisions may be booleans (e.g. allow/deny), strings (e.g. hostnames), numbers (e.g. ratelimits), arrays (e.g. destination hosts for a pod), or dictionaries (e.g. JSON patch changes to a pod definition).

For example, using OPA with Kubernetes you could enforce any of the following policies:

• All images come from an AWS repository (other than a whitelist)

• Images are pulled from the same AWS repository in the same region as the cluster

• Each namespace is controlled by the users in different AWS IAM groups

Goals

This tutorial shows how to use OPA to enforce custom policies on resources in Kubernetes running on AWS. For the purpose of this tutorial, you will define a policy that requires all pod images to come from an AWS image repository.

Prerequisites

This tutorial requires a Kubernetes 1.8 cluster. Keep in mind that External Admission Webhook support in Kubernetes is currently in alpha.

Overview

1. Create the Kubernetes cluster with kops and enable the External Admission Webhooks

2. Install the OpenPolicyAgent as an External Admission Webhook

3. Write admission control policies with the OpenPolicyAgent

Step 1: Create Cluster with External Admission Controllers via kops

Create a Kubernetes 1.8 cluster with kops as shown in the Cluster Install instructions.

Update the Cluster spec to enable webhooks. (Remember that when you set up the cluster you chose the name example.cluster.k8s.local. )

kops edit example.cluster.k8s.local

  kubeAPIServer:
    admissionControl:
      - NamespaceLifecycle
      - LimitRanger
      - ServiceAccount
      - PersistentVolumeLabel
      - DefaultStorageClass
      - DefaultTolerationSeconds
      - GenericAdmissionWebhook
      - ResourceQuota
    runtimeConfig:
      admissionregistration.k8s.io/v1alpha1: "true"

Then update kops.

kops update example.cluster.k8s.local --yes

Step 2: Deploy OPA on Kubernetes

2.1 Generate service-certificate for OPA

First, create the required OpenSSL configuration files. Put these in your current directory; we will only use them once to generate certificates. You may discard them after running the openssl commands below.

server.conf

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth

Now create local files that contain the CA and server key pair.

Create a certificate authority:

openssl genrsa -out ca.key 2048

openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"

Create a server certiticate. IMPORTANT: the CN must match the name of the service used to expose the external admission controller.

openssl genrsa -out server.key 2048

openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf

2.2: Deploy OPA on Kubernetes

First, create a namespace to deploy OPA into.

kubectl create namespace opa

Create a Service to expose the OPA API. The Kubernetes API server will lookup the Service and execute webhook requests against it.

opa-admission-controller-service.yaml:

kind: Service
apiVersion: v1
metadata:
  name: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443

kubectl create -f opa-admission-controller-service.yaml -n opa

Next, create Secrets containing the TLS credentials for OPA:

kubectl create secret generic opa-ca --from-file=ca.crt -n opa
kubectl create secret tls opa-server --cert=server.crt --key=server.key -n opa

Finally, create the Deployment to run OPA as an Admission Controller. The deployment contains two containers: opa and kube-mgmt. opa by itself is a general-purpose policy engine and knows nothing about Kubernetes. kube-mgmt is a collection of Kubernetes-specific code that helps OPA interact with kubernetes.

opa-admission-controller-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  name: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        - name: opa
          image: openpolicyagent/opa:0.5.13
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:443"
            - "--insecure-addr=127.0.0.1:8181"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.5
          args:
            - "--replicate=v1/pods"
            - "--pod-name=$(MY_POD_NAME)"
            - "--pod-namespace=$(MY_POD_NAMESPACE)"
            - "--register-admission-controller"
            - "--admission-controller-ca-cert-file=/certs/ca.crt"
            - "--admission-controller-service-name=opa"
            - "--admission-controller-service-namespace=$(MY_POD_NAMESPACE)"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-ca
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server
        - name: opa-ca
          secret:
            secretName: opa-ca

kubectl create -f opa-admission-controller-deployment.yaml -n opa

When OPA starts, the sidecar (kube-mgmt) will register it as an External Admission Controller. To verify that registration succeeded, query the Kubernetes API for the list of External Admission Controllers.

kubectl describe externaladmissionhookconfigurations admission.openpolicyagent.org

Finally, you can follow the OPA logs to see the webhook requests being issued by the Kubernetes API server:

kubectl logs -l app=opa -c opa -n opa

Step 3: Enforce Kubernetes Admission Control with OPA

3.1 Load a policy into OPA

To test admission control, create a policy that requires all images to come from an AWS repository. For details on the policy language, see the Open Policy Agent documentation.

Note

Below replace the Amazon account ID 123456789 with your own account if your want the pod to actually come up. If you just want to see the admission controller in action, you can leave it with the fake ID. That account ID also appears in the image names when you create pods below; just make sure the account IDs are the same.

image_source.rego:

package system

# Deny requests that include container images not from ECR.
deny[explanation] {
    image_name = input.spec.object.Spec.Containers[_].Image
    image_name_parts = split(image_name, "/")
    repo_name = image_name_parts[0]
    not startswith(repo_name, "12345678.dkr.ecr.us-west-2.amazonaws.com")
    explanation = sprintf("image '%v' not from AWS ECR", [image_name])
}


# main is entry point to policy.
# Boilerplate required by admission webhook.
# Actual policy decision is `status`, which takes the form
#   {"allowed": BOOLEAN, "status": {"reason": STRING}}
main = {
    "apiVersion": "admission.k8s.io/v1alpha1",
    "kind": "AdmissionReview",
    "status": {"allowed": allowed, "status": {"reason": reason}}
}

# Boilerplate: construct 'reason' and 'allowed' variables.
#  Real policy is the collection of 'deny' statements above.
#  If not denied, allow.
reason = msg {
    msg = concat(", ", deny)
}
default allowed = true
allowed = false { n = count(deny); n > 0 }

Store the policy in Kubernetes as a ConfigMap.

kubectl create configmap image-source --from-file=image_source.rego -n opa

The OPA sidecar will notice the ConfigMap and automatically load the contained policy into OPA.

3.2 Check that the policy is working

To verify that your policy is working, create separate test pods.

nginx-pod.yaml:

kind: Pod
apiVersion: v1
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - image: nginx
    name: nginx

Note	Below replace the Amazon account ID 123456789 with your own account if your want the pod to actually come up. If you just want to see the admission controller in action, you can leave it with the fake ID.

amazon-linux-pod.yaml:

kind: Pod
apiVersion: v1
metadata:
  name: amazon-linux-pod
  labels:
    app: amazon-linux
spec:
  containers:
  - image: 123456789.dkr.ecr.us-west-2.amazonaws.com/amazon-linux
    name: amazon-linux

Verify that you can create an amazon-linux pod.

kubectl create -f amazon-linux-pod.yaml

Verify that you CANNOT create an nginx pod and receive the appropriate error message.

kubectl create -f nginx-pod.yaml

Error from server (image 'nginx' not from AWS ECR): error when creating "nginx-pod.yaml":

This example shows how to ensure ALL images come from an AWS repository. But in reality you might have a collection of images like nginx that can come from outside of AWS. Or maybe you only want to apply the policy to certain Kubernetes namespaces. OPA’s policy language is flexible enough to add image whitelists and control the applicable namespaces. Just modify your policy locally, update the ConfigMap, and kube-mgmt will update OPA with your changes.

Wrap Up

Congratulations for finishing the tutorial!

This tutorial showed how you can leverage OPA to enforce admission control decisions in Kubernetes clusters without modifying or recompiling any Kubernetes components. Furthermore, once Kubernetes is configured to use OPA as an External Admission Controller, policies can be modified on-the-fly to satisfy changing operational requirements.

You are now ready to continue on with the workshop!

Go to Developer Index	Go to Operations Index