-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document RBAC set up for ingress key cert protection. #755
Changes from 5 commits
07add4d
3c91c35
bd00f69
02f66cb
722943b
5795f37
ff6b74c
abd1066
5992c3d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -269,6 +269,204 @@ rules. | |
curl -I -k https://$INGRESS_HOST/status/200 | ||
``` | ||
|
||
1. Configuring RBAC for ingress key/cert. | ||
|
||
There are service accounts which can access this ingress key/cert, and this leads to risks of | ||
leaking key/cert. We can set up Role-Based Access Control ("RBAC") to protect it. | ||
[istio/install/kubernetes/istio.yaml](https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml) | ||
defines ClusterRoles and ClusterRoleBindings which allow service accounts in namespace istio-system | ||
to access all secret resources. We need to update or replace these RBAC set up to only allow | ||
istio-ingress-service-account to access ingress key/cert. | ||
|
||
We can use kubectl to list all secrets in namespace istio-system that we need to protect using RBAC. | ||
|
||
```bash | ||
kubectl get secrets -n istio-system | ||
``` | ||
This produces the following output: | ||
|
||
```bash | ||
NAME TYPE DATA AGE | ||
istio-ca-secret istio.io/ca-root 2 7d | ||
istio-ingress-certs kubernetes.io/tls 2 7d | ||
istio-ingress-service-account-token-kwr85 kubernetes.io/service-account-token 3 7d | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we remove this one as well? And istio-ca-secret? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
istio.istio-ingress-service-account istio.io/key-and-cert 3 7d | ||
...... | ||
``` | ||
|
||
1. Update RBAC set up for istio-pilot-service-account and istio-mixer-istio-service-account. | ||
|
||
Delete existing ClusterRoleBindings and ClusterRole. | ||
|
||
```bash | ||
kubectl delete ClusterRoleBinding istio-pilot-admin-role-binding-istio-system | ||
kubectl delete ClusterRoleBinding istio-mixer-admin-role-binding-istio-system | ||
kubectl delete ClusterRole istio-mixer-istio-system | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a similar role for istio-pilot-istio-system? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there any way to recover the deletion here at the end of this file? The idea is to clean up the change in this file after everything is done. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. And what privileges will pilot/mixer lose after this change? I am afraid this will break existing system. Let me add some pilot and mixer guy. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. istio-pilot-istio-system is also bound to account istio-ingress-service-account by istio-ingress-admin-role-binding-istio-system. We will delete istio-pilot-istio-system in next step, after deleting that ClusterRoleBinding object. According to https://github.com/istio/istio/blob/11dc2fb937a23157215fffa06921cf507d27f757/install/kubernetes/istio.yaml#L22 |
||
``` | ||
|
||
Create istio-pilot-mixer-istio-system.yaml, which allows istio-pilot-service-account and | ||
istio-mixer-service-account to read all kubernetes.io/service-account-token, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think anyone needs to read There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree. That line is removed. |
||
istio.io/key-and-cert, and istio.io/ca-root types of secret instances. | ||
|
||
```bash | ||
kind: ClusterRole | ||
apiVersion: rbac.authorization.k8s.io/v1beta1 | ||
metadata: | ||
name: istio-pilot-mixer-istio-system | ||
rules: | ||
- apiGroups: [""] # "" indicates the core API group | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here and below. Please avoid showing unrelated content. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
resources: ["secrets"] | ||
resourceNames: ["istio-ca-service-account-token-rl4xm"] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems quite dangerous to hard-code names with randomized tokens ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a kubernetes.io/service-account-token, and it is removed. |
||
verbs: ["get", "list", "watch"] | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio.istio-ca-service-account"] | ||
verbs: ["get", "list", "watch"] | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio-ca-secret"] | ||
verbs: ["get", "list", "watch"] | ||
...... | ||
--- | ||
kind: ClusterRoleBinding | ||
apiVersion: rbac.authorization.k8s.io/v1beta1 | ||
metadata: | ||
name: istio-pilot-mixer-admin-role-binding-istio-system | ||
subjects: | ||
- kind: ServiceAccount | ||
name: istio-pilot-service-account | ||
namespace: istio-system | ||
- kind: ServiceAccount | ||
name: istio-mixer-service-account | ||
namespace: istio-system | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So we're binding the same role to both mixer SA and pilot SA. Is this desirable? IIRC mixer and pilot need different permissions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, they need different permissions. I have create separate ClusterRoles for each of them. |
||
roleRef: | ||
kind: ClusterRole | ||
name: istio-pilot-mixer-istio-system | ||
apiGroup: rbac.authorization.k8s.io | ||
``` | ||
|
||
```bash | ||
kubectl apply -f istio-pilot-mixer-istio-system.yaml | ||
``` | ||
|
||
1. Update RBAC set up for istio-ingress-service-account. | ||
|
||
Delete existing ClusterRoleBinding and ClusterRole. | ||
|
||
```bash | ||
kubectl delete clusterrolebinding istio-ingress-admin-role-binding-istio-system | ||
kubectl delete ClusterRole istio-pilot-istio-system | ||
``` | ||
|
||
Create istio-ingress-istio-system.yaml which allows istio-ingress-service-account to read | ||
istio-ingress-certs as well as other secret instances. | ||
|
||
```bash | ||
kind: ClusterRole | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
metadata: | ||
name: istio-ingress-istio-system | ||
rules: | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio-ca-service-account-token-rl4xm"] | ||
verbs: ["get", "list", "watch"] | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio.istio-ca-service-account"] | ||
verbs: ["get", "list", "watch"] | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio-ca-secret"] | ||
verbs: ["get", "list", "watch"] | ||
...... | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio-ingress-certs"] | ||
verbs: ["get", "list", "watch"] | ||
--- | ||
kind: ClusterRoleBinding | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
metadata: | ||
name: istio-ingress-admin-role-binding-istio-system | ||
subjects: | ||
- kind: ServiceAccount | ||
name: istio-ingress-service-account | ||
namespace: istio-system | ||
roleRef: | ||
kind: ClusterRole | ||
name: istio-ingress-istio-system | ||
apiGroup: rbac.authorization.k8s.io | ||
``` | ||
|
||
```bash | ||
kubectl apply -f istio-ingress-istio-system.yaml | ||
``` | ||
|
||
1. Update RBAC set up for istio-ca-service-account. | ||
|
||
Create istio-ca-istio-system.yaml, which updates existing ClusterRole istio-ca-istio-system that | ||
allows istio-ca-service-account to read, create and modify all istio.io/key-and-cert, | ||
kubernetes.io/service-account-token and istio.io/ca-root types of secrets. | ||
|
||
```bash | ||
kind: ClusterRole | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
metadata: | ||
name: istio-ca-istio-system | ||
rules: | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio-ca-service-account-token-rl4xm"] | ||
verbs: ["get", "list", "watch", "create", "update"] | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio.istio-ca-service-account"] | ||
verbs: ["get", "list", "watch", "create", "update"] | ||
- apiGroups: [""] # "" indicates the core API group | ||
resources: ["secrets"] | ||
resourceNames: ["istio-ca-secret"] | ||
verbs: ["get", "list", "watch", "create", "update"] | ||
...... | ||
``` | ||
|
||
```bash | ||
kubectl apply -f istio-ca-istio-system.yaml | ||
``` | ||
1. Verify that the new ClusterRoles work as expected. | ||
|
||
```bash | ||
kubectl auth can-i get secret/istio-ingress-certs --as system:serviceaccount:istio-system:istio-ingress-service-account -n istio-system | ||
``` | ||
whose output should be | ||
```bash | ||
yes | ||
``` | ||
In this command, we can replace verb "get" with "list" or "watch", and the output should always | ||
be "yes". Now let us test with other service accounts. | ||
|
||
```bash | ||
kubectl auth can-i get secret/istio-ingress-certs --as system:serviceaccount:istio-system:istio-pilot-service-account -n istio-system | ||
``` | ||
whose output should be | ||
```bash | ||
no - Unknown user "system:serviceaccount:istio-system:istio-pilot-service-account" | ||
``` | ||
In this command, we can replace service account with istio-mixer-service-account, or | ||
istio-ca-service-account, we can also replace verb "get" with "watch" or "list", and the | ||
output should look similarly. | ||
|
||
Accessibility to secret resources except istio-ingress-certs should remain the same for | ||
istio-ca-service-account, istio-ingress-service-account, istio-pilot-service-account and | ||
istio-mixer-service-account. | ||
```bash | ||
kubectl auth can-i get secret/istio-ca-service-account-token-r14xm --as system:serviceaccount:istio-system:istio-ca-service-account -n istio-system | ||
``` | ||
whose output should be | ||
```bash | ||
yes | ||
``` | ||
|
||
## Configuring ingress for gRPC | ||
|
||
The ingress controller currently doesn't support `.` characters in the `path` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't have to display everything, please focus on ingress related one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Thanks!