From 6e275d207c4cf24dfb94ee46b52a84ab2be3c1ab Mon Sep 17 00:00:00 2001 From: bsctl Date: Mon, 28 Jun 2021 08:45:57 +0200 Subject: [PATCH 1/6] docs: add few test cases --- .../mtb/block-access-to-cluster-resources.md | 48 ++++++ .../block-access-to-multitenant-resources.md | 153 ++++++++++++++++++ .../block-access-to-other-tenant-resources.md | 97 +++++++++++ docs/operator/mtb/sig-multitenancy-bench.md | 30 ++++ 4 files changed, 328 insertions(+) create mode 100644 docs/operator/mtb/block-access-to-cluster-resources.md create mode 100644 docs/operator/mtb/block-access-to-multitenant-resources.md create mode 100644 docs/operator/mtb/block-access-to-other-tenant-resources.md create mode 100644 docs/operator/mtb/sig-multitenancy-bench.md diff --git a/docs/operator/mtb/block-access-to-cluster-resources.md b/docs/operator/mtb/block-access-to-cluster-resources.md new file mode 100644 index 00000000..62b1353f --- /dev/null +++ b/docs/operator/mtb/block-access-to-cluster-resources.md @@ -0,0 +1,48 @@ +# Block access to cluster resources + +**Profile Applicability:** L1 + +**Type:** Configuration Check + +**Category:** Control Plane Isolation + +**Description:** Tenants should not be able to view, edit, create, or delete cluster (non-namespaced) resources such Node, ClusterRole, ClusterRoleBinding, etc. + +**Rationale:** Access controls should be configured for tenants so that a tenant cannot list, create, modify or delete cluster resources + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - < +``` +Each command must return `no` + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +``` \ No newline at end of file diff --git a/docs/operator/mtb/block-access-to-multitenant-resources.md b/docs/operator/mtb/block-access-to-multitenant-resources.md new file mode 100644 index 00000000..6b7bc440 --- /dev/null +++ b/docs/operator/mtb/block-access-to-multitenant-resources.md @@ -0,0 +1,153 @@ +# Block access to multitenant resources + +**Profile Applicability:** L1 + +**Type:** Behavioral + +**Category:** Tenant Isolation + +**Description:** Each tenant namespace may contain resources setup by the cluster administrator for multi-tenancy, such as role bindings, and network policies. Tenants should not be allowed to modify the namespaced resources created by the cluster administrator for multi-tenancy. However, for some resources such as network policies, tenants can configure additional instances of the resource for their workloads. + +**Rationale:** Tenants can escalate priviliges and impact other tenants if they are able to delete or modify required multi-tenancy resources such as namespace resource quotas or default network policy. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - < 7m5s +capsule-oil-1 7m5s +``` + +As tenant owner try to modify or delete one of the networkpolicies + +```bash +kubectl --kubeconfig alice delete networkpolicies capsule-oil-0 +``` + +You should receive an error message denying the edit/delete request + +```bash +Error from server (Forbidden): networkpolicies.networking.k8s.io "capsule-oil-0" is forbidden: User "oil" cannot delete resource "networkpolicies" in API group "networking.k8s.io" in the namespace "oil-production" +``` + +As tenant owner, you can create an additional networkpolicy inside the namespace + +```yaml +kubectl create -f - << EOF +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: hijacking + namespace: oil-production +spec: + egress: + - to: + - ipBlock: + cidr: 0.0.0.0/0 + podSelector: {} + policyTypes: + - Egress +EOF +``` + +However, due the additive nature of networkpolicies, the `DENY ALL` policy set by the cluster admin, prevents the hijacking. + +As tenant owner list RBAC permissions set by Capsule + +```bash +kubectl --kubeconfig alice get rolebindings +NAME ROLE AGE +namespace-deleter ClusterRole/capsule-namespace-deleter 11h +namespace:admin ClusterRole/admin 11h +``` + +As tenant owner, try to change/delete the rolebindings in order to escalate permissions + +```bash +kubectl --kubeconfig alice edit/delete rolebinding namespace:admin +``` + +You should receive an error message: + +``` +error: rolebindings.rbac.authorization.k8s.io "namespace:admin" could not be patched: +rolebindings.rbac.authorization.k8s.io "namespace:admin" is forbidden: +... +``` + +However, the tenant owner can create and assign permissions inside namespace she owns + +```yaml +kubectl create -f - << EOF +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + labels: + name: oil-robot:admin + namespace: oil-production +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: admin +subjects: +- kind: ServiceAccount + name: default + namespace: oil-production +EOF +``` + + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +``` \ No newline at end of file diff --git a/docs/operator/mtb/block-access-to-other-tenant-resources.md b/docs/operator/mtb/block-access-to-other-tenant-resources.md new file mode 100644 index 00000000..d052be0f --- /dev/null +++ b/docs/operator/mtb/block-access-to-other-tenant-resources.md @@ -0,0 +1,97 @@ +# Block access to other tenant resources + +**Profile Applicability:** L1 + +**Type:** Behavioral + +**Category:** Tenant Isolation + +**Description:** Each tenant has its own set of resources, such as namespaces, service accounts, secrets, pods, services, etc. Tenants should not be allowed to access eachother's resources. + +**Rationale:** Tenant's resources must be not accessible by other tenants. + +**Audit:** + +As cluster admin, create a couple of tenants + +```yaml +kubectl create -f - < N.B. At time of writing, the MTB are in development and not ready for usage. Strictly speaking, we do not claim an official conformance to MTB, but just to adhere to the multi-tenancy requirements and best practices promoted by MTB. + +|MTB Benchmark |MTB Profile|Capsule Version|Conformance|Notes | +|--------------|-----------|---------------|-----------|-------| +|[Block access to cluster resources](block-access-to-cluster-resources.md)|L1|v0.1.0|✓|---| +|[Block access to multitenant resources](block-access-to-multitenant-resources.md)|L1|v0.1.0|✓|---| +|[Block access to other tenant resources](block-access-to-other-tenant-resources.md)|L1|v0.1.0|✓|MTB draft| +|Block add capabilities|L1|v0.1.0|✓|---| +|Require always imagePullPolicy|L1|v0.1.0|✓|---| +|Require run as non-root user|L1|v0.1.0|✓|---| +|Block privileged containers|L1|v0.1.0|✓|---| +|Block privilege escalation|L1|v0.1.0|✓|---| +|Configure namespace resource quotas|L1|v0.1.0|✓|---| +|Configure namespace object limits|L1|v0.1.0|✓|---| +|Block use of host path volumes|L1|v0.1.0|✓|---| +|Block use of NodePort services|L1|v0.1.0|✓|---| +|Block use of host networking and ports|L1|v0.1.0|✓|---| +|Block use of host PID|L1|v0.1.0|✓|---| +|Block use of host IPC|L1|v0.1.0|✓|---| +|Block modification of resource quotas|L1|v0.1.0|✓|---| +|Require PersistentVolumeClaim for storage|L1|v0.1.0|✓|MTB draft| +|Require PV reclaim policy of delete|L1|v0.1.0|✓|MTB draft| +|Block use of existing PVs|L1|v0.1.0|✓|MTB draft| +|Block network access across tenant namespaces|L1|v0.1.0|✓|MTB draft| +|Allow self-service management of Network PoliciesL2|v0.1.0|✓|---| +|Allow self-service management of RolesL2|v0.1.0|✓|MTB draft| +|Allow self-service management of Roles Bindings|L2|v0.1.0|✓|MTB draft| From d78f7393bdbaaffd6cb9b27fb09a2eddaaa03db6 Mon Sep 17 00:00:00 2001 From: bsctl Date: Fri, 16 Jul 2021 11:45:51 +0200 Subject: [PATCH 2/6] docs: additional test cases --- .../block-access-to-multitenant-resources.md | 3 +- docs/operator/mtb/block-add-capabilities.md | 118 ++++++++++++++++++ .../mtb/require-always-imagepullpolicy.md | 71 +++++++++++ docs/operator/mtb/sig-multitenancy-bench.md | 4 +- 4 files changed, 193 insertions(+), 3 deletions(-) create mode 100644 docs/operator/mtb/block-add-capabilities.md create mode 100644 docs/operator/mtb/require-always-imagepullpolicy.md diff --git a/docs/operator/mtb/block-access-to-multitenant-resources.md b/docs/operator/mtb/block-access-to-multitenant-resources.md index 6b7bc440..8e978e56 100644 --- a/docs/operator/mtb/block-access-to-multitenant-resources.md +++ b/docs/operator/mtb/block-access-to-multitenant-resources.md @@ -75,7 +75,8 @@ kubectl --kubeconfig alice delete networkpolicies capsule-oil-0 You should receive an error message denying the edit/delete request ```bash -Error from server (Forbidden): networkpolicies.networking.k8s.io "capsule-oil-0" is forbidden: User "oil" cannot delete resource "networkpolicies" in API group "networking.k8s.io" in the namespace "oil-production" +Error from server (Forbidden): networkpolicies.networking.k8s.io "capsule-oil-0" is forbidden: +User "oil" cannot delete resource "networkpolicies" in API group "networking.k8s.io" in the namespace "oil-production" ``` As tenant owner, you can create an additional networkpolicy inside the namespace diff --git a/docs/operator/mtb/block-add-capabilities.md b/docs/operator/mtb/block-add-capabilities.md new file mode 100644 index 00000000..cfb85293 --- /dev/null +++ b/docs/operator/mtb/block-add-capabilities.md @@ -0,0 +1,118 @@ +# Block add capabilities + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Control Plane Isolation + +**Description:** Control Linux capabilities. + +**Rationale:** Linux allows defining fine-grained permissions using capabilities. With Kubernetes, it is possible to add capabilities for pods that escalate the level of kernel access and allow other potentially dangerous behaviors. + +**Audit:** + +As cluster admin, define a `PodSecurityPolicy` with `allowedCapabilities` and map the policy to a tenant: + +```yaml +kubectl create -f - << EOF +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: tenant +spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + # The default set of capabilities are implicitly allowed + # The empty set means that no additional capabilities may be added beyond the default set + allowedCapabilities: [] + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + fsGroup: + rule: RunAsAny +EOF +``` + +> Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod and see new capabilities cannot be added in the tenant namespaces + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-settime-cap + namespace: + labels: +spec: + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] + securityContext: + capabilities: + add: + - SYS_TIME +EOF +``` + +You should have the pod blocked by PodSecurityPolicy. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +``` \ No newline at end of file diff --git a/docs/operator/mtb/require-always-imagepullpolicy.md b/docs/operator/mtb/require-always-imagepullpolicy.md new file mode 100644 index 00000000..f8c4a321 --- /dev/null +++ b/docs/operator/mtb/require-always-imagepullpolicy.md @@ -0,0 +1,71 @@ +# Require always imagePullPolicy + +**Profile Applicability:** L1 + +**Type:** Configuration Check + +**Category:** Data Isolation + +**Description:** Set the image pull policy to Always for tenant workloads. + +**Rationale:** Tenants have to be assured that their private images can only be used by those who have the credentials to pull them. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil +spec: + imagePullPolicies: + - Always + owners: + - kind: User + name: alice +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, creates a pod in the tenant namespace having `imagePullPolicies=IfNotPresent` + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: nginx + namespace: oil-production +spec: + containers: + - name: nginx + image: nginx:latest + imagePullPolicy: IfNotPresent +EOF +``` + +You should receive an error message denying the request: + +``` +Error from server +(ImagePullPolicy IfNotPresent for container nginx is forbidden, use one of the followings: Always): error when creating "STDIN": admission webhook "pods.capsule.clastix.io" denied the request: +ImagePullPolicy IfNotPresent for container nginx is forbidden, use one of the followings: Always +``` + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +``` \ No newline at end of file diff --git a/docs/operator/mtb/sig-multitenancy-bench.md b/docs/operator/mtb/sig-multitenancy-bench.md index 5bea8619..28331a30 100644 --- a/docs/operator/mtb/sig-multitenancy-bench.md +++ b/docs/operator/mtb/sig-multitenancy-bench.md @@ -8,8 +8,8 @@ Actually, there's no yet a real standard for the multi-tenancy model in Kubernet |[Block access to cluster resources](block-access-to-cluster-resources.md)|L1|v0.1.0|✓|---| |[Block access to multitenant resources](block-access-to-multitenant-resources.md)|L1|v0.1.0|✓|---| |[Block access to other tenant resources](block-access-to-other-tenant-resources.md)|L1|v0.1.0|✓|MTB draft| -|Block add capabilities|L1|v0.1.0|✓|---| -|Require always imagePullPolicy|L1|v0.1.0|✓|---| +|[Block add capabilities](block-add-capabilities.md)|L1|v0.1.0|✓|---| +|[Require always imagePullPolicy](require-always-imagepullpolicy.md)|L1|v0.1.0|✓|---| |Require run as non-root user|L1|v0.1.0|✓|---| |Block privileged containers|L1|v0.1.0|✓|---| |Block privilege escalation|L1|v0.1.0|✓|---| From 7bd41c9ad377095da90e1aa46c684008783735fd Mon Sep 17 00:00:00 2001 From: bsctl Date: Fri, 16 Jul 2021 20:22:23 +0200 Subject: [PATCH 3/6] docs: additional test cases --- docs/operator/mtb/block-add-capabilities.md | 3 + .../block-modification-of-resource-quotas.md | 69 +++++++++ .../mtb/block-privilege-escalation.md | 115 +++++++++++++++ .../mtb/block-privileged-containers.md | 116 +++++++++++++++ .../block-use-of-host-networking-and-ports.md | 134 ++++++++++++++++++ .../mtb/block-use-of-host-path-volumes.md | 127 +++++++++++++++++ .../mtb/configure-namespace-object-limits.md | 66 +++++++++ .../configure-namespace-resource-quotas.md | 65 +++++++++ .../mtb/require-run-as-non-root-user.md | 119 ++++++++++++++++ docs/operator/mtb/sig-multitenancy-bench.md | 18 +-- 10 files changed, 823 insertions(+), 9 deletions(-) create mode 100644 docs/operator/mtb/block-modification-of-resource-quotas.md create mode 100644 docs/operator/mtb/block-privilege-escalation.md create mode 100644 docs/operator/mtb/block-privileged-containers.md create mode 100644 docs/operator/mtb/block-use-of-host-networking-and-ports.md create mode 100644 docs/operator/mtb/block-use-of-host-path-volumes.md create mode 100644 docs/operator/mtb/configure-namespace-object-limits.md create mode 100644 docs/operator/mtb/configure-namespace-resource-quotas.md create mode 100644 docs/operator/mtb/require-run-as-non-root-user.md diff --git a/docs/operator/mtb/block-add-capabilities.md b/docs/operator/mtb/block-add-capabilities.md index cfb85293..3259bbed 100644 --- a/docs/operator/mtb/block-add-capabilities.md +++ b/docs/operator/mtb/block-add-capabilities.md @@ -64,6 +64,7 @@ apiVersion: capsule.clastix.io/v1beta1 kind: Tenant metadata: name: oil + namespace: oil-production spec: owners: - kind: User @@ -115,4 +116,6 @@ As cluster admin, delete all the created resources ```bash kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp ``` \ No newline at end of file diff --git a/docs/operator/mtb/block-modification-of-resource-quotas.md b/docs/operator/mtb/block-modification-of-resource-quotas.md new file mode 100644 index 00000000..3875ba7d --- /dev/null +++ b/docs/operator/mtb/block-modification-of-resource-quotas.md @@ -0,0 +1,69 @@ +# Block modification of resource quotas + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Tenant Isolation + +**Description:** Tenants should not be able to modify the resource quotas defined in their namespaces + +**Rationale:** Resource quotas must be configured for isolation and fairness between tenants. Tenants should not be able to modify existing resource quotas as they may exhaust cluster resources and impact other tenants. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - < Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod or container that sets `allowPrivilegeEscalation=true` in its `securityContext`. + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-priviliged-mode + namespace: oil-production + labels: +spec: + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] + securityContext: + allowPrivilegeEscalation: true +EOF +``` + +You should have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/block-privileged-containers.md b/docs/operator/mtb/block-privileged-containers.md new file mode 100644 index 00000000..a698c39a --- /dev/null +++ b/docs/operator/mtb/block-privileged-containers.md @@ -0,0 +1,116 @@ +# Block privileged containers + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Control Plane Isolation + +**Description:** Control container permissions. + +**Rationale:** By default a container is not allowed to access any devices on the host, but a “privileged” container can access all devices on the host. A process within a privileged container can also get unrestricted host access. Hence, tenants should not be allowed to run privileged containers. + +**Audit:** + +As cluster admin, define a `PodSecurityPolicy` that sets `privileged=false` and map the policy to a tenant: + +```yaml +kubectl create -f - << EOF +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: tenant +spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + fsGroup: + rule: RunAsAny +EOF +``` + +> Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil + namespace: oil-production +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod or container that sets privileges in its `securityContext`. + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-priviliged-mode + namespace: + labels: +spec: + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] + securityContext: + privileged: true +EOF +``` + +You should have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/block-use-of-host-networking-and-ports.md b/docs/operator/mtb/block-use-of-host-networking-and-ports.md new file mode 100644 index 00000000..301dda33 --- /dev/null +++ b/docs/operator/mtb/block-use-of-host-networking-and-ports.md @@ -0,0 +1,134 @@ +# Block use of host networking and ports + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Host Isolation + +**Description:** Tenants should not be allowed to use host networking and host ports for their workloads. + +**Rationale:** Using `hostPort` and `hostNetwork` allows tenants workloads to share the host networking stack allowing potential snooping of network traffic across application pods. + +**Audit:** + +As cluster admin, define a `PodSecurityPolicy` that restricts `hostPort` and `hostNetwork` and map the policy to a tenant: + +```yaml +kubectl create -f - << EOF +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: tenant +spec: + privileged: false + hostNetwork: false + hostPorts: [] # empty means no allowed host ports + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + fsGroup: + rule: RunAsAny +EOF +``` + +> Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil + namespace: oil-production +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod using `hostNetwork` + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-hostnetwork + namespace: oil-production +spec: + hostNetwork: true + containers: + - name: nginx + image: nginx:latest + ports: + - containerPort: 80 +EOF +``` + +As tenant owner, create a pod defining a container using `hostPort` + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-hostport + namespace: oil-production +spec: + containers: + - name: nginx + image: nginx:latest + ports: + - containerPort: 80 + hostPort: 9090 +EOF +``` + +In both the cases, you should have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/block-use-of-host-path-volumes.md b/docs/operator/mtb/block-use-of-host-path-volumes.md new file mode 100644 index 00000000..0622cc2d --- /dev/null +++ b/docs/operator/mtb/block-use-of-host-path-volumes.md @@ -0,0 +1,127 @@ +# Block use of host path volumes + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Host Protection + +**Description:** Tenants should not be able to mount host volumes and directories. + +**Rationale:** The use of host volumes and directories can be used to access shared data or escalate priviliges and also creates a tight coupling between a tenant workload and a host. + +**Audit:** + +As cluster admin, define a `PodSecurityPolicy` that restricts `hostPath` volumes and map the policy to a tenant: + +```yaml +kubectl create -f - << EOF +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: tenant +spec: + privileged: false + volumes: # hostPath is not permitted + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'persistentVolumeClaim' + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + fsGroup: + rule: RunAsAny +EOF +``` + +> Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil + namespace: oil-production +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod defining a volume of type `hostpath`. + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-hostpath-volume + namespace: oil-production +spec: + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] + volumeMounts: + - mountPath: /tmp + name: volume + volumes: + - name: volume + hostPath: + # directory location on host + path: /data + type: Directory +EOF +``` + +You should have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/configure-namespace-object-limits.md b/docs/operator/mtb/configure-namespace-object-limits.md new file mode 100644 index 00000000..0c851b58 --- /dev/null +++ b/docs/operator/mtb/configure-namespace-object-limits.md @@ -0,0 +1,66 @@ +# Configure namespace object limits + +**Profile Applicability:** L1 + +**Type:** Configuration + +**Category:** Fairness + +**Description:** Namespace resource quotas should be used to allocate, track and limit the number of objects, of a particular type, that can be created within a namespace. + +**Rationale:** Resource quotas must be configured for each tenant namespace, to guarantee isolation and fairness across tenants. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - < Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod or container that does not set `runAsNonRoot` to `true` in its `securityContext`, and `runAsUser` must not be set to 0. + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-run-as-root + namespace: oil-production +spec: + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] +EOF +``` + +You should have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/sig-multitenancy-bench.md b/docs/operator/mtb/sig-multitenancy-bench.md index 28331a30..44ddc09d 100644 --- a/docs/operator/mtb/sig-multitenancy-bench.md +++ b/docs/operator/mtb/sig-multitenancy-bench.md @@ -10,17 +10,17 @@ Actually, there's no yet a real standard for the multi-tenancy model in Kubernet |[Block access to other tenant resources](block-access-to-other-tenant-resources.md)|L1|v0.1.0|✓|MTB draft| |[Block add capabilities](block-add-capabilities.md)|L1|v0.1.0|✓|---| |[Require always imagePullPolicy](require-always-imagepullpolicy.md)|L1|v0.1.0|✓|---| -|Require run as non-root user|L1|v0.1.0|✓|---| -|Block privileged containers|L1|v0.1.0|✓|---| -|Block privilege escalation|L1|v0.1.0|✓|---| -|Configure namespace resource quotas|L1|v0.1.0|✓|---| -|Configure namespace object limits|L1|v0.1.0|✓|---| -|Block use of host path volumes|L1|v0.1.0|✓|---| -|Block use of NodePort services|L1|v0.1.0|✓|---| -|Block use of host networking and ports|L1|v0.1.0|✓|---| +|[Require run as non-root user](require-run-as-non-root-user.md)|L1|v0.1.0|✓|---| +|[Block privileged containers](block-privileged-containers.md)|L1|v0.1.0|✓|---| +|[Block privilege escalation](block-privilege-escalation.md)|L1|v0.1.0|✓|---| +|[Configure namespace resource quotas](configure-namespace-resource-quotas.md)|L1|v0.1.0|✓|---| +|[Block modification of resource quotas](block-modification-of-resource-quotas.md)|L1|v0.1.0|✓|---| +|[Configure namespace object limits](configure-namespace-object-limits.md)|L1|v0.1.0|✓|---| +|[Block use of host path volumes](block-use-of-host-path-volumes.md)|L1|v0.1.0|✓|---| +|[Block use of host networking and ports](block-use-of-host-networking-and-ports.md)|L1|v0.1.0|✓|---| |Block use of host PID|L1|v0.1.0|✓|---| |Block use of host IPC|L1|v0.1.0|✓|---| -|Block modification of resource quotas|L1|v0.1.0|✓|---| +|Block use of NodePort services|L1|v0.1.0|✓|---| |Require PersistentVolumeClaim for storage|L1|v0.1.0|✓|MTB draft| |Require PV reclaim policy of delete|L1|v0.1.0|✓|MTB draft| |Block use of existing PVs|L1|v0.1.0|✓|MTB draft| From b7b101809201001f2c04aee4ba0a8c6ef2e04dae Mon Sep 17 00:00:00 2001 From: bsctl Date: Sat, 17 Jul 2021 14:05:51 +0200 Subject: [PATCH 4/6] docs: add further test cases --- ...-service-management-of-network-policies.md | 77 +++++++++++ ...self-service-management-of-rolebindings.md | 58 ++++++++ .../allow-self-service-management-of-roles.md | 58 ++++++++ .../block-access-to-multitenant-resources.md | 2 +- .../block-access-to-other-tenant-resources.md | 4 +- docs/operator/mtb/block-add-capabilities.md | 2 +- ...network-access-across-tenant-namespaces.md | 107 +++++++++++++++ .../mtb/block-privilege-escalation.md | 2 +- .../mtb/block-privileged-containers.md | 2 +- ...lock-use-of-existing-persistent-volumes.md | 40 ++++++ docs/operator/mtb/block-use-of-host-ipc.md | 115 ++++++++++++++++ .../block-use-of-host-networking-and-ports.md | 4 +- .../mtb/block-use-of-host-path-volumes.md | 4 +- docs/operator/mtb/block-use-of-host-pid.md | 115 ++++++++++++++++ .../mtb/block-use-of-nodeport-services.md | 75 +++++++++++ .../mtb/require-always-imagepullpolicy.md | 2 +- ...quire-persistentvolumeclaim-for-storage.md | 124 ++++++++++++++++++ .../mtb/require-reclaim-policy-of-delete.md | 87 ++++++++++++ .../mtb/require-run-as-non-root-user.md | 2 +- docs/operator/mtb/sig-multitenancy-bench.md | 20 +-- 20 files changed, 880 insertions(+), 20 deletions(-) create mode 100644 docs/operator/mtb/allow-self-service-management-of-network-policies.md create mode 100644 docs/operator/mtb/allow-self-service-management-of-rolebindings.md create mode 100644 docs/operator/mtb/allow-self-service-management-of-roles.md create mode 100644 docs/operator/mtb/block-network-access-across-tenant-namespaces.md create mode 100644 docs/operator/mtb/block-use-of-existing-persistent-volumes.md create mode 100644 docs/operator/mtb/block-use-of-host-ipc.md create mode 100644 docs/operator/mtb/block-use-of-host-pid.md create mode 100644 docs/operator/mtb/block-use-of-nodeport-services.md create mode 100644 docs/operator/mtb/require-persistentvolumeclaim-for-storage.md create mode 100644 docs/operator/mtb/require-reclaim-policy-of-delete.md diff --git a/docs/operator/mtb/allow-self-service-management-of-network-policies.md b/docs/operator/mtb/allow-self-service-management-of-network-policies.md new file mode 100644 index 00000000..50ff37ad --- /dev/null +++ b/docs/operator/mtb/allow-self-service-management-of-network-policies.md @@ -0,0 +1,77 @@ +# Allow self-service management of Network Policies + +**Profile Applicability:** L2 + +**Type:** Behavioral + +**Category:** Self-Service Operations + +**Description:** Tenants should be able to perform self-service operations by creating own network policies in their namespaces. + +**Rationale:** Enables self-service management of network-policies. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - < 7m5s +``` + +As tenant owner check for permissions to manage networkpolicy for each verb + +```bash +kubectl --kubeconfig alice auth can-i get networkpolicies +kubectl --kubeconfig alice auth can-i create networkpolicies +kubectl --kubeconfig alice auth can-i update networkpolicies +kubectl --kubeconfig alice auth can-i patch networkpolicies +kubectl --kubeconfig alice auth can-i delete networkpolicies +kubectl --kubeconfig alice auth can-i deletecollection networkpolicies +``` + +Each command must return 'yes' + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +``` \ No newline at end of file diff --git a/docs/operator/mtb/allow-self-service-management-of-rolebindings.md b/docs/operator/mtb/allow-self-service-management-of-rolebindings.md new file mode 100644 index 00000000..baa4828d --- /dev/null +++ b/docs/operator/mtb/allow-self-service-management-of-rolebindings.md @@ -0,0 +1,58 @@ +# Allow self-service management of Role Bindings + +**Profile Applicability:** L2 + +**Type:** Behavioral + +**Category:** Self-Service Operations + +**Description:** Tenants should be able to perform self-service operations by creating own rolebindings in their namespaces. + +**Rationale:** Enables self-service management of roles. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - < Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil + namespace: oil-production +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod mounting the host IPC namespace. + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-host-ipc + namespace: oil-production +spec: + hostIPC: true + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] +EOF +``` + +You must have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/block-use-of-host-networking-and-ports.md b/docs/operator/mtb/block-use-of-host-networking-and-ports.md index 301dda33..82164d39 100644 --- a/docs/operator/mtb/block-use-of-host-networking-and-ports.md +++ b/docs/operator/mtb/block-use-of-host-networking-and-ports.md @@ -22,6 +22,8 @@ metadata: name: tenant spec: privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false hostNetwork: false hostPorts: [] # empty means no allowed host ports runAsUser: @@ -122,7 +124,7 @@ spec: EOF ``` -In both the cases, you should have the pod blocked by `PodSecurityPolicy`. +In both the cases above, you must have the pod blocked by `PodSecurityPolicy`. **Cleanup:** As cluster admin, delete all the created resources diff --git a/docs/operator/mtb/block-use-of-host-path-volumes.md b/docs/operator/mtb/block-use-of-host-path-volumes.md index 0622cc2d..95db6bd2 100644 --- a/docs/operator/mtb/block-use-of-host-path-volumes.md +++ b/docs/operator/mtb/block-use-of-host-path-volumes.md @@ -22,6 +22,8 @@ metadata: name: tenant spec: privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false volumes: # hostPath is not permitted - 'configMap' - 'emptyDir' @@ -115,7 +117,7 @@ spec: EOF ``` -You should have the pod blocked by `PodSecurityPolicy`. +You must have the pod blocked by `PodSecurityPolicy`. **Cleanup:** As cluster admin, delete all the created resources diff --git a/docs/operator/mtb/block-use-of-host-pid.md b/docs/operator/mtb/block-use-of-host-pid.md new file mode 100644 index 00000000..b7ef20c2 --- /dev/null +++ b/docs/operator/mtb/block-use-of-host-pid.md @@ -0,0 +1,115 @@ +# Block use of host PID + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Host Isolation + +**Description:** Tenants should not be allowed to share the host process ID (PID) namespace. + +**Rationale:** The `hostPID` setting allows pods to share the host process ID namespace allowing potential privilege escalation. Tenant pods should not be allowed to share the host PID namespace. + +**Audit:** + +As cluster admin, define a `PodSecurityPolicy` that restricts `hostPID` usage and map the policy to a tenant: + +```yaml +kubectl create -f - << EOF +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: tenant +spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + hostPID: false + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + fsGroup: + rule: RunAsAny +EOF +``` + +> Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil + namespace: oil-production +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod mounting the host PID namespace. + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-host-pid + namespace: oil-production +spec: + hostPID: true + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] +EOF +``` + +You must have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/block-use-of-nodeport-services.md b/docs/operator/mtb/block-use-of-nodeport-services.md new file mode 100644 index 00000000..7d27434d --- /dev/null +++ b/docs/operator/mtb/block-use-of-nodeport-services.md @@ -0,0 +1,75 @@ +# Block use of NodePort services + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** Host Isolation + +**Description:** Tenants should not be able to create services of type NodePort. + +**Rationale:** the service type `NodePorts` configures host ports that cannot be secured using Kubernetes network policies and require upstream firewalls. Also, multiple tenants cannot use the same host port numbers. + +**Audit:** + +As cluster admin, create a tenant + +```yaml +kubectl create -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil +spec: + enableNodePorts: false + owners: + - kind: User + name: alice +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, creates a service in the tenant namespace having service type of `NodePort` + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Service +metadata: + name: nginx + labels: + namespace: oil-production +spec: + ports: + - protocol: TCP + port: 8080 + targetPort: 80 + selector: + run: nginx + type: NodePort +EOF +``` + +You must receive an error message denying the request: + +``` +Error from server +Error from server (NodePort service types are forbidden for the tenant: +error when creating "STDIN": admission webhook "services.capsule.clastix.io" denied the request: +NodePort service types are forbidden for the tenant: please, reach out to the system administrators +``` + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +``` \ No newline at end of file diff --git a/docs/operator/mtb/require-always-imagepullpolicy.md b/docs/operator/mtb/require-always-imagepullpolicy.md index f8c4a321..caa45756 100644 --- a/docs/operator/mtb/require-always-imagepullpolicy.md +++ b/docs/operator/mtb/require-always-imagepullpolicy.md @@ -55,7 +55,7 @@ spec: EOF ``` -You should receive an error message denying the request: +You must receive an error message denying the request: ``` Error from server diff --git a/docs/operator/mtb/require-persistentvolumeclaim-for-storage.md b/docs/operator/mtb/require-persistentvolumeclaim-for-storage.md new file mode 100644 index 00000000..97a629f1 --- /dev/null +++ b/docs/operator/mtb/require-persistentvolumeclaim-for-storage.md @@ -0,0 +1,124 @@ +# Require PersistentVolumeClaim for storage + +**Profile Applicability:** L1 + +**Type:** Behavioral Check + +**Category:** na + +**Description:** Tenants should not be able to use all volume types except `PersistentVolumeClaims`. + +**Rationale:** In some scenarios, it would be required to disallow usage of any core volume types except PVCs. + +**Audit:** + +As cluster admin, define a `PodSecurityPolicy` allowing only `PersistentVolumeClaim` volumes and map the policy to a tenant: + +```yaml +kubectl create -f - << EOF +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: tenant +spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + volumes: + - 'persistentVolumeClaim' + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + fsGroup: + rule: RunAsAny +EOF +``` + +> Note: make sure `PodSecurityPolicy` Admission Control is enabled on the APIs server: `--enable-admission-plugins=PodSecurityPolicy` + +Then create a ClusterRole using or granting the said item + +```yaml +kubectl create -f - << EOF +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: tenant:psp +rules: +- apiGroups: ['policy'] + resources: ['podsecuritypolicies'] + resourceNames: ['tenant'] + verbs: ['use'] +EOF +``` + +And assign it to the tenant + +```yaml +kubectl apply -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil + namespace: oil-production +spec: + owners: + - kind: User + name: alice + additionalRoleBindings: + - clusterRoleName: tenant:psp + subjects: + - kind: "Group" + apiGroup: "rbac.authorization.k8s.io" + name: "system:authenticated" +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, create a pod defining a volume of any of the core type except `PersistentVolumeClaim`. For example: + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +apiVersion: v1 +kind: Pod +metadata: + name: pod-with-hostpath-volume + namespace: oil-production +spec: + containers: + - name: busybox + image: busybox:latest + command: ["/bin/sleep", "3600"] + volumeMounts: + - mountPath: /tmp + name: volume + volumes: + - name: volume + hostPath: + # directory location on host + path: /data + type: Directory +EOF +``` + +You must have the pod blocked by `PodSecurityPolicy`. + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete PodSecurityPolicy tenant +kubectl --kubeconfig cluster-admin delete ClusterRole tenant:psp +``` diff --git a/docs/operator/mtb/require-reclaim-policy-of-delete.md b/docs/operator/mtb/require-reclaim-policy-of-delete.md new file mode 100644 index 00000000..284d2725 --- /dev/null +++ b/docs/operator/mtb/require-reclaim-policy-of-delete.md @@ -0,0 +1,87 @@ +# Require PV reclaim policy of delete + +**Profile Applicability:** L1 + +**Type:** Configuration Check + +**Category:** Data Isolation + +**Description:** Force a tenant to use a Storage Class with `reclaimPolicy=Delete`. + +**Rationale:** Tenants have to be assured that their Persisten Volumes cannot be reclaimed by other tenants. + +**Audit:** + +As cluster admin, create a Storage Class with `reclaimPolicy=Delete` + +```yaml +kubectl create -f - << EOF +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: delete-policy +reclaimPolicy: Delete +provisioner: clastix.io/nfs +EOF +``` + +As cluster admin, create a tenant and assign the above Storage Class + +```yaml +kubectl create -f - << EOF +apiVersion: capsule.clastix.io/v1beta1 +kind: Tenant +metadata: + name: oil +spec: + owners: + - kind: User + name: alice + storageClasses: + allowed: + - delete-policy +EOF + +./create-user.sh alice oil +``` + +As tenant owner, run the following command to create a namespace in the given tenant + +```bash +kubectl --kubeconfig alice create ns oil-production +kubectl --kubeconfig alice config set-context --current --namespace oil-production +``` + +As tenant owner, creates a Persistent Volum Claim in the tenant namespace missing the Storage Class or using any other Storage Class: + +```yaml +kubectl --kubeconfig alice apply -f - << EOF +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: pvc + namespace: oil-production +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 12Gi +EOF +``` + +You must receive an error message denying the request: + +``` +Error from server (A valid Storage Class must be used, one of the following (delete-policy)): +error when creating "STDIN": admission webhook "pvc.capsule.clastix.io" denied the request: +A valid Storage Class must be used, one of the following (delete-policy) +``` + +**Cleanup:** +As cluster admin, delete all the created resources + +```bash +kubectl --kubeconfig cluster-admin delete tenant oil +kubectl --kubeconfig cluster-admin delete storageclass delete-policy +``` \ No newline at end of file diff --git a/docs/operator/mtb/require-run-as-non-root-user.md b/docs/operator/mtb/require-run-as-non-root-user.md index 4d768ea5..9f25d83e 100644 --- a/docs/operator/mtb/require-run-as-non-root-user.md +++ b/docs/operator/mtb/require-run-as-non-root-user.md @@ -107,7 +107,7 @@ spec: EOF ``` -You should have the pod blocked by `PodSecurityPolicy`. +You must have the pod blocked by `PodSecurityPolicy`. **Cleanup:** As cluster admin, delete all the created resources diff --git a/docs/operator/mtb/sig-multitenancy-bench.md b/docs/operator/mtb/sig-multitenancy-bench.md index 44ddc09d..d2cceb24 100644 --- a/docs/operator/mtb/sig-multitenancy-bench.md +++ b/docs/operator/mtb/sig-multitenancy-bench.md @@ -18,13 +18,13 @@ Actually, there's no yet a real standard for the multi-tenancy model in Kubernet |[Configure namespace object limits](configure-namespace-object-limits.md)|L1|v0.1.0|✓|---| |[Block use of host path volumes](block-use-of-host-path-volumes.md)|L1|v0.1.0|✓|---| |[Block use of host networking and ports](block-use-of-host-networking-and-ports.md)|L1|v0.1.0|✓|---| -|Block use of host PID|L1|v0.1.0|✓|---| -|Block use of host IPC|L1|v0.1.0|✓|---| -|Block use of NodePort services|L1|v0.1.0|✓|---| -|Require PersistentVolumeClaim for storage|L1|v0.1.0|✓|MTB draft| -|Require PV reclaim policy of delete|L1|v0.1.0|✓|MTB draft| -|Block use of existing PVs|L1|v0.1.0|✓|MTB draft| -|Block network access across tenant namespaces|L1|v0.1.0|✓|MTB draft| -|Allow self-service management of Network PoliciesL2|v0.1.0|✓|---| -|Allow self-service management of RolesL2|v0.1.0|✓|MTB draft| -|Allow self-service management of Roles Bindings|L2|v0.1.0|✓|MTB draft| +|[Block use of host PID](block-use-of-host-pid.md)|L1|v0.1.0|✓|---| +|[Block use of host IPC](block-use-of-host-ipc.md)|L1|v0.1.0|✓|---| +|[Block use of NodePort services](block-use-of-nodeport-services.md)|L1|v0.1.0|✓|---| +|[Require PersistentVolumeClaim for storage](require-persistentvolumeclaim-for-storage.md)|L1|v0.1.0|✓|MTB draft| +|[Require PV reclaim policy of delete](require-reclaim-policy-of-delete.md)|L1|v0.1.0|✓|MTB draft| +|[Block use of existing PVs](block-use-of-existing-persistent-volumes.md)|L1|v0.1.0|✓|MTB draft| +|[Block network access across tenant namespaces](block-network-access-across-tenant-namespaces.md)|L1|v0.1.0|✓|MTB draft| +|[Allow self-service management of Network Policies](allow-self-service-management-of-network-policies.md)|L2|v0.1.0|✓|---| +|[Allow self-service management of Roles](allow-self-service-management-of-roles.md)|L2|v0.1.0|✓|MTB draft| +|[Allow self-service management of Role Bindings](allow-self-service-management-of-rolebindings.md)|L2|v0.1.0|✓|MTB draft| From 85de087786c1e7e267c94309d181bab6bbfd6c0b Mon Sep 17 00:00:00 2001 From: bsctl Date: Sat, 17 Jul 2021 14:29:59 +0200 Subject: [PATCH 5/6] docs: update links in documentation --- docs/operator/overview.md | 38 +++----------------------------------- 1 file changed, 3 insertions(+), 35 deletions(-) diff --git a/docs/operator/overview.md b/docs/operator/overview.md index c986e24e..a901d3fd 100644 --- a/docs/operator/overview.md +++ b/docs/operator/overview.md @@ -1,41 +1,9 @@ # Kubernetes multi-tenancy made simple -**Capsule** helps to implement a multi-tenancy and policy-based environment in your Kubernetes cluster. It is not intended to be yet another _PaaS_, instead, it has been designed as a micro-services based ecosystem with minimalist approach, leveraging only on upstream Kubernetes. - -# What's the problem with the current status? -Kubernetes introduces the _Namespace_ object type to create logical partitions of the cluster as isolated *slices*. However, implementing advanced multi-tenancy scenarios, it becomes soon complicated because of the flat structure of Kubernetes namespaces and the impossibility to share resources among namespaces belonging to the same tenant. To overcome this, cluster admins tend to provision a dedicated cluster for each groups of users, teams, or departments. As an organization grows, the number of clusters to manage and keep aligned becomes an operational nightmare, described as the well know phenomena of the _clusters sprawl_. - -# Entering Caspule -Capsule takes a different approach. In a single cluster, it aggregates multiple namespaces in a lightweight abstraction called _Tenant_. Within each tenant, users are free to create their namespaces and share all the assigned resources while a Policy Engine keeps different tenants isolated from each other. The _Network and Security Policies_, _Resource Quota_, _Limit Ranges_, _RBAC_, and other policies defined at the tenant level are automatically inherited by all the namespaces in the tenant. And users are free to operate their tenants in authonomy, without the intervention of the cluster administrator. - -# Features -## Self-Service -Leave to developers the freedom to self-provision their cluster resources according to the assigned boundaries. - -## Preventing Clusters Sprawl -Share a single cluster with multiple teams, groups of users, or departments by saving operational and management efforts. - -## Governance -Leverage Kubernetes Admission Controllers to enforce the industry security best practices and meet legal requirements. - -## Resources Control -Take control of the resources consumed by users while preventing them to overtake. - -## Native Experience -Provide multi-tenancy with a native Kubernetes experience without introducing additional management layers, plugins, or customised binaries. - -## GitOps ready -Capsule is completely declarative and GitOps ready. - -## Bring your own device (BYOD) -Assign to tenants a dedicated set of compute, storage, and network resources and avoid the noisy neighbors' effect. - -# Common use cases for Capsule -Please, refer to the corresponding [section](./use-cases/overview.md) in the project documentation for a detailed list of common use cases that Capsule can address. - -# What’s next -Have a fun with Capsule: +**Capsule** helps to implement a multi-tenancy and policy-based environment in your Kubernetes cluster. Have a fun with Capsule: * [Getting Started](./getting-started.md) * [Use Cases](./use-cases/overview.md) +* [SIG Multi-tenancy benchmark](./mtb/sig-multitenancy-bench.md) +* [Run on Managed Kubernetes Services](./managed-kubernetes/getting-started-amazon-eks.md) * [Contributing](./contributing.md) * [References](./references.md) \ No newline at end of file From b881f0a655388161fc476eebd8aa1294019cc1ff Mon Sep 17 00:00:00 2001 From: bsctl Date: Fri, 23 Jul 2021 18:38:45 +0200 Subject: [PATCH 6/6] docs: fix minor issues --- ...-service-management-of-network-policies.md | 2 +- .../mtb/block-access-to-cluster-resources.md | 65 +++++++++++++++++++ .../block-access-to-multitenant-resources.md | 9 +-- 3 files changed, 71 insertions(+), 5 deletions(-) diff --git a/docs/operator/mtb/allow-self-service-management-of-network-policies.md b/docs/operator/mtb/allow-self-service-management-of-network-policies.md index 50ff37ad..94e1ff61 100644 --- a/docs/operator/mtb/allow-self-service-management-of-network-policies.md +++ b/docs/operator/mtb/allow-self-service-management-of-network-policies.md @@ -26,7 +26,7 @@ spec: name: alice networkPolicies: items: - ingress: + - ingress: - from: - namespaceSelector: matchLabels: diff --git a/docs/operator/mtb/block-access-to-cluster-resources.md b/docs/operator/mtb/block-access-to-cluster-resources.md index 62b1353f..65606509 100644 --- a/docs/operator/mtb/block-access-to-cluster-resources.md +++ b/docs/operator/mtb/block-access-to-cluster-resources.md @@ -40,6 +40,71 @@ kubectl --kubeconfig alice auth can-i ``` Each command must return `no` +**Exception:** + +It should, but it does not: + +```bash +kubectl --kubeconfig alice auth can-i create selfsubjectaccessreviews +yes +kubectl --kubeconfig alice auth can-i create selfsubjectrulesreviews +yes +kubectl --kubeconfig alice auth can-i create namespaces +yes +``` + +Any kubernetes user can create `SelfSubjectAccessReview` and `SelfSubjectRulesReviews` to checks whether he/she can perform an action. First two exceptions are not an issue. + +```bash +kubectl --anyuser auth can-i --list +Resources Non-Resource URLs Resource Names Verbs +selfsubjectaccessreviews.authorization.k8s.io [] [] [create] +selfsubjectrulesreviews.authorization.k8s.io [] [] [create] + [/api/*] [] [get] + [/api] [] [get] + [/apis/*] [] [get] + [/apis] [] [get] + [/healthz] [] [get] + [/healthz] [] [get] + [/livez] [] [get] + [/livez] [] [get] + [/openapi/*] [] [get] + [/openapi] [] [get] + [/readyz] [] [get] + [/readyz] [] [get] + [/version/] [] [get] + [/version/] [] [get] + [/version] [] [get] + [/version] [] [get] +``` + +In order to enable namespace self-service provisioning, Capsule intentionally gives permissions to create namespaces to all users belonging to the Capsule group: + +```bash +kubectl describe clusterrolebindings capsule-namespace-provisioner +Name: capsule-namespace-provisioner +Labels: +Annotations: +Role: + Kind: ClusterRole + Name: capsule-namespace-provisioner +Subjects: + Kind Name Namespace + ---- ---- --------- + Group capsule.clastix.io + +kubectl describe clusterrole capsule-namespace-provisioner +Name: capsule-namespace-provisioner +Labels: +Annotations: +PolicyRule: + Resources Non-Resource URLs Resource Names Verbs + --------- ----------------- -------------- ----- + namespaces [] [] [create] +``` + +Capsule controls self-service namespace creation by limiting the number of namespaces the user can create by the `tenant.spec.namespaceQuota option`. + **Cleanup:** As cluster admin, delete all the created resources diff --git a/docs/operator/mtb/block-access-to-multitenant-resources.md b/docs/operator/mtb/block-access-to-multitenant-resources.md index f3ca91e9..9db18e5a 100644 --- a/docs/operator/mtb/block-access-to-multitenant-resources.md +++ b/docs/operator/mtb/block-access-to-multitenant-resources.md @@ -116,12 +116,13 @@ As tenant owner, try to change/delete the rolebindings in order to escalate per kubectl --kubeconfig alice edit/delete rolebinding namespace:admin ``` -You must receive an error message: +The rolebindings is immediately recreated by Capsule: ``` -error: rolebindings.rbac.authorization.k8s.io "namespace:admin" could not be patched: -rolebindings.rbac.authorization.k8s.io "namespace:admin" is forbidden: -... +kubectl --kubeconfig alice get rolebindings +NAME ROLE AGE +namespace-deleter ClusterRole/capsule-namespace-deleter 11h +namespace:admin ClusterRole/admin 2s ``` However, the tenant owner can create and assign permissions inside namespace she owns