Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Protect the Kubelet API #208

Closed
wants to merge 2 commits into from
Closed

Protect the Kubelet API #208

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
12ce8c5
Protect the Kubelet API
bgeesaman Sep 2, 2017
0aaf79e
Fix RBAC for Kubelet and add ClusterRole/Bindings
bgeesaman Sep 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/08-bootstrapping-kubernetes-controllers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ ExecStart=/usr/local/bin/kube-apiserver \\
--etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \\
--event-ttl=1h \\
--experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
--insecure-bind-address=0.0.0.0 \\
--insecure-bind-address=127.0.0.1 \\
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the secure port is already listening, this is just used for the locally running control plane services.

--kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
--kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
--kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
Expand Down Expand Up @@ -118,7 +118,7 @@ ExecStart=/usr/local/bin/kube-controller-manager \\
--cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
--cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
--leader-elect=true \\
--master=http://${INTERNAL_IP}:8080 \\
--master=http://127.0.0.1:8080 \\
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modify to look locally instead of on the INTERNAL_IP.

--root-ca-file=/var/lib/kubernetes/ca.pem \\
--service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \\
--service-cluster-ip-range=10.32.0.0/16 \\
Expand All @@ -144,7 +144,7 @@ Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
--leader-elect=true \\
--master=http://${INTERNAL_IP}:8080 \\
--master=http://127.0.0.1:8080 \\
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modify to look locally instead of on the INTERNAL_IP.

--v=2
Restart=on-failure
RestartSec=5
Expand Down
57 changes: 55 additions & 2 deletions docs/09-bootstrapping-kubernetes-workers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,8 @@ Requires=crio.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
--anonymous-auth=false \\
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not automatically consider clients to the Kubelet as system:anonymous in case someone unknowingly makes the system:anonymous role have actual permissions.

--authorization-mode=Webhook \\
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ask the API server via the SubjectAccessReview mechanism if the subject is allowed to access the Kubelet API for a given resource and verb.

--allow-privileged=true \\
--cluster-dns=10.32.0.10 \\
--cluster-domain=cluster.local \\
Expand All @@ -199,6 +201,7 @@ ExecStart=/usr/local/bin/kubelet \\
--register-node=true \\
--require-kubeconfig \\
--runtime-request-timeout=10m \\
--client-ca-file=/var/lib/kubernetes/ca.pem \\
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give the kubelet the ability to validate certificates against the CA to know the subject to send for SubjectAccessReview.

--tls-cert-file=/var/lib/kubelet/${HOSTNAME}.pem \\
--tls-private-key-file=/var/lib/kubelet/${HOSTNAME}-key.pem \\
--v=2
Expand Down Expand Up @@ -258,15 +261,65 @@ sudo systemctl start crio kubelet kube-proxy

> Remember to run the above commands on each worker node: `worker-0`, `worker-1`, and `worker-2`.

## Verification
## Implement RBAC for Kubelet Authorization

Login to one of the controller nodes:

```
gcloud compute ssh controller-0
```

List the registered Kubernetes nodes:
Define a ```clusterrole``` with the proper permissions for kubelet API access and a ```clusterrolebinding``` to allow the ```kubernetes``` user to use that ```clusterrole```.
```
cat > kubelet-rbac.yaml << EOF
---
apiVersion: v1
kind: List
metadata: {}
items:
- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
verbs:
- "*"
- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF
```

Create the ```clusterrole``` and ```clusterrolebinding``` in the cluster.
```
kubectl create -f kubelet-rbac.yaml
```

## Verification

While still logged into one of the controller nodes, list the registered Kubernetes nodes:

```
kubectl get nodes
Expand Down