kube-aws-approver.md

Kubernetes AWS approver

A custom node CertificateSigningRequest approver for kubernetes clusters running on AWS.

Why do I want this?

kubelet TLS bootstrapping allows kubelet to create CertificateSigningRequest for its client certs. These CertificateSigningRequest need to be approved so as to be signed by kubernetes internal certificate signer controller.

Currently you can either manually approve each node that tries to join the cluster. Or, instruct kubernetes internal certificate approver to approve all certificates from system:bootstrappers group. The human intervention required for manual approval is not ideal in auto scaling environments. And the default auto approver performs very basic checks that might not be very secure.

kube-aws-approver goes a step further from default auto approver and uses the CertificateSigningRequest generated by node to extract information about the node requesting the certificate, and tries to validate it against AWS API before approving the CertificateSigningRequest.

How does it work?

kube-aws-approver matches incoming CertificateSigningRequest against two critera:

Criteria 1: Node already belonging to the cluster requests for client certificate renewal.

• The CertificateRequest Organization is equal to system:nodes
• The CertificateRequest DNSNames list is empty
• The CertificateRequest EmailAddresses list is empty
• The CertificateRequest IPAddresses list is empty
• The CertificateRequest CommonName has system:node: prefix
• The CertificateSigningRequest Username matches the CertificateRequest CommonName, i.e. the node provided its client certificate to create this CertificateSigningRequest.
• The CertificateSigningRequest Groups has system:nodes
• There exists an instance in running state corresponding to the nodename from the CertificateRequest CommonName
• A node with the nodename from CertificateRequest CommonName is part of the cluster and in Ready state.
• The instance should belong to an AutoScalingGroup.
• The AutoSaclingGroup belongs to the list of whitelisted AutoScalingGroups provided to kube-aws-approver at startup.

Only when a CertificateSigningRequest from a node looking to renew its client certificate meets all the above requirements, the CertificateSigningRequest is Approved more

Criteria 2: A new node trying to join the cluster requests for a client certificate.

• The CertificateRequest Organization is equal to system:nodes
• The CertificateRequest DNSNames list is empty
• The CertificateRequest EmailAddresses list is empty
• The CertificateRequest IPAddresses list is empty
• The CertificateRequest CommonName has system:node: prefix
• The CertificateSigningRequest Groups has system:bootstrappers
• There exists an instance in running state corresponding to the nodename from the CertificateRequest CommonName
• The CertificateSigningRequest Username shoudd be of the form system:bootstrappers:<aws-instance-id> How you might achieve this?
• The instance-id extracted from CertificateSigningRequest Username matches the instance-id extracted from AWS API for the nodename in CertificateRequest CommonName
• The node with nodename from CertificateRequest CommonName is not part of the cluster.
• The instance should belong to an AutoScalingGroup.
• The AutoSaclingGroup belongs to the list of whitelisted AutoScalingGroups provided to kube-aws-approver at startup.

Only when a CertificateSigningRequest from a new node looking for its client certificate meets all the above requirements, the CertificateSigningRequest is Approved more

kube-aws-approver only approves a CertificateSigningRequest that meets one of these two criteria. It never explicitly declines any CertificateSigningRequest. The approved CertificateSigningRequest is then signed by the kube-controller-manager and can be used by the node as client certificate.

CertificateSigningRequest Approval process

The following example go code shows how the kube-aws-approver marks a CertificateSigningRequest approved.

...
...
// csr is the CertificateSigningRequest(https://godoc.org/k8s.io/api/certificates/v1beta1#CertificateSigningRequest) object received by the approver.
csr.Status.Conditions = append(csr.Status.Conditions, CertificateSigningRequestCondition{
    Type:    CertificateApproved,
    Reason:  "AutoApproved",
    Message: successMessage,
})
_, err = client.CertificatesV1beta1().CertificateSigningRequests().UpdateApproval(csr)

...
...

How to run?

Running kube-aws-approver outside the target cluster

1. Contacting the cluster

--kubeconfig should point to a valid kubeconfig file that has access CertificateSigningRequest.

2. Contacting the AWS API

--region-name should be set to the AWS region where the instances and autoscalinggroups are running.

The kube-aws-approver uses the following priority for the source of credentials to contact AWS API. ENV variables > ~/.aws/ config file > EC2Role provided credentials.

3. Setting the ASG whitelist

--allowed-asgs should contain a comma separated list of the names of AutoScalingGroups that instances requesting to join the cluster can belong to.

4. Running the approver

TODO(abhinav): run as a container.

Running kube-aws-approver inside the target cluster

1. Contacting the cluster

Either --kubeconfig should point to a valid kubeconfig file that has access CertificateSigningRequest. Or If the --kubeconfig is empty, kube-aws-approver will use InCluster Config to contact the apiserver.

2. Contacting the AWS API

Either --region-name should be set to the AWS region where the instances and autoscalinggroups are running. Or If the --region-name is empty, kube-aws-approver will fetch the AWS region from instance metadata service.

The kube-aws-approver uses the following priority for the source of credentials to contact AWS API. ENV variables > ~/.aws/ config file > EC2Role provided credentials. Recommended approach is to use EC2Role as credentials.

3. Setting the ASG whitelist

--allowed-asgs should contain a comma separated list of the names of AutoScalingGroups that instances requesting to join the cluster can belong to.

4. Running the approver

TODO(abhinav): run as a deployment, with service account, cluster role bindings and config map for allowed ASGs.