-
Notifications
You must be signed in to change notification settings - Fork 616
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refer to authorisation model in RFC-0001
Signed-off-by: Stefan Prodan <[email protected]>
- Loading branch information
1 parent
a67d193
commit e0bc754
Showing
1 changed file
with
37 additions
and
103 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,10 @@ | ||
| # RFC-0001 Flux Multi-Tenancy | ||
| # RFC-0004 Flux Multi-Tenancy | ||
|
|
||
| **Status:** provisional | ||
|
|
||
| **Creation date:** 2021-11-15 | ||
|
|
||
| **Last update:** 2021-12-17 | ||
|
|
||
| ## Summary | ||
|
|
||
|
|
@@ -7,10 +13,10 @@ models for multi-tenancy, and gives reference implementations for those models. | |
|
|
||
| ## Motivation | ||
|
|
||
| To this point, the Flux project has provided [examples of | ||
| multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy/tree/v0.1.0), but not explained exactly | ||
| To this point, the Flux project has provided [examples of multi-tenancy][mt], but not explained exactly | ||
| how they relate to Flux's authorisation model. This RFC explains two multi-tenancy implementations, | ||
| their security properties, and how they are implemented within the authorisation model. | ||
| their security properties, and how they are implemented within the authorisation model | ||
| as defined in [RFC-0001](https://github.com/fluxcd/flux2/pull/2212). | ||
|
|
||
| ### Goals | ||
|
|
||
|
|
@@ -22,7 +28,7 @@ their security properties, and how they are implemented within the authorisation | |
| ### Non-Goals | ||
|
|
||
| - Give an exhaustive account of multi-tenancy implementations in general. | ||
| - Provide an [end-to-end workflow](](https://github.com/fluxcd/flux2-multi-tenancy/tree/v0.1.0)) of | ||
| - Provide an [end-to-end workflow][mt] of | ||
| how to set up multi-tenancy with Flux. | ||
|
|
||
| ## Introduction | ||
|
|
@@ -54,14 +60,18 @@ to a cluster. These controllers are subject to authorisation on two counts: | |
| - when accessing Kubernetes resources that are needed for a | ||
| particular "apply" operation -- for example, a secret referenced in | ||
| the field `.spec.valuesFrom` in a `HelmRelease`; | ||
| - when creating, updating and deleting Kubernetes resources in the process of applying a piece of | ||
| configuration. | ||
| - when creating, watching, updating and deleting Kubernetes resources | ||
| in the process of applying a piece of configuration. | ||
|
|
||
| To give users control over this authorisation, these two controllers will _impersonate_ (assume the | ||
| identity of) a service account mentioned in the apply specification (e.g., [the field | ||
| `.spec.serviceAccountName` in a `Kustomization` object][serviceAccountName]) for both accessing | ||
| resources and applying configuration. This lets a user constrain the operations mentioned above with | ||
| RBAC. | ||
| identity of) a service account mentioned in the apply specification (e.g., the field | ||
| `.spec.serviceAccountName` in a [`Kustomization` object][kcsa] | ||
| or in a [`HelmRelease` object][hcsa]) for both accessing resources and applying configuration. | ||
| This lets a user constrain the operations mentioned above with RBAC. | ||
|
|
||
| As stated in [RFC-0003](https://github.com/fluxcd/flux2/pull/2093), | ||
| the platform admins can configure Flux to enforce service account impersonation | ||
| by setting a default service account name when `.spec.serviceAccountName` is not specified. | ||
|
|
||
| #### Remote apply | ||
|
|
||
|
|
@@ -72,9 +82,6 @@ the client used to apply the specified set of configuration. The effect of this | |
| configuration will be applied as the user given in the kubeconfig; often this is a user with the | ||
| `cluster-admin` role bound to it, but not necessarily so. | ||
|
|
||
| [serviceAccountName]: https://fluxcd.io/docs/components/kustomize/api/#kustomize.toolkit.fluxcd.io/v1beta2.KustomizationSpec | ||
| [kubeconfig]: https://fluxcd.io/docs/components/kustomize/api/#kustomize.toolkit.fluxcd.io/v1beta2.KubeConfig | ||
|
|
||
| ## Assumptions made by the multi-tenancy models | ||
|
|
||
| ### User Roles | ||
|
|
@@ -146,9 +153,9 @@ To restrict the reconciliation of tenant's sources, a Kubernetes service account | |
| in Flux `Kustomizations` and `HelmReleases` under `.spec.serviceAccountName`. Please consult the Flux | ||
| documentation for more details: | ||
|
|
||
| - [Kustomization API: Role-based access control](https://fluxcd.io/docs/components/kustomize/kustomization/#role-based-access-control) | ||
| - [HelmRelease API: Role-based access control](https://fluxcd.io/docs/components/helm/helmreleases/#role-based-access-control) | ||
| - [Flux multi-tenancy example repository](https://github.com/fluxcd/flux2-multi-tenancy) | ||
| - [Kustomization API: Role-based access control][kcsa] | ||
| - [HelmRelease API: Role-based access control][hcsa] | ||
| - [Flux multi-tenancy example repository][mt] | ||
|
|
||
| Note that with soft multi-tenancy, true tenant isolation requires security measures beyond Kubernetes RBAC. | ||
| Please refer to the Kubernetes [security considerations documentation](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/#security-considerations) | ||
|
|
@@ -167,100 +174,22 @@ The Flux CLI offers an easy way of generating all the Kubernetes manifests neede | |
| - `flux create source git` command generates the configuration that tells Flux which repositories belong to tenants. | ||
| - `flux create kustomization` command generates the configuration that tells Flux how to reconcile the manifests found in the tenants repositories. | ||
|
|
||
| All the above commands have an `--export` flag for generating the Kubernetes resources in YAML format. | ||
| The platform admins should place the generated manifests in the repository that defines the cluster(s) desired state. | ||
|
|
||
| Here is an example of the generated manifests: | ||
|
|
||
| ```yaml | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: tenant1 | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: flux | ||
| namespace: tenant1 | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: RoleBinding | ||
| metadata: | ||
| name: flux | ||
| namespace: tenant1 | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: cluster-admin | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: flux | ||
| namespace: tenant1 | ||
| --- | ||
| apiVersion: source.toolkit.fluxcd.io/v1beta1 | ||
| kind: GitRepository | ||
| metadata: | ||
| name: tenant1 | ||
| namespace: tenant1 | ||
| spec: | ||
| interval: 5m0s | ||
| ref: | ||
| branch: main | ||
| secretRef: | ||
| name: tenant1-git-auth | ||
| url: ssh://[email protected]/org/tenant1 | ||
| --- | ||
| apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 | ||
| kind: Kustomization | ||
| metadata: | ||
| name: tenant1 | ||
| namespace: tenant1 | ||
| spec: | ||
| interval: 10m0s | ||
| path: ./ | ||
| prune: true | ||
| serviceAccountName: flux | ||
| sourceRef: | ||
| kind: GitRepository | ||
| name: tenant1 | ||
| ``` | ||
| Note that the [cluster-admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) | ||
| role is used in a `RoleBinding`; this only gives full control over every resource in the role binding's namespace. | ||
|
|
||
| Once the tenants main repositories are registered on the cluster(s), the tenants can configure their app delivery | ||
| in Git using Kubernetes namespace-scoped resources such as `Deployments`, `Services`, Flagger `Canaries`, | ||
| Flux `GitRepositories`, `Kustomizations`, `HelmRepositories`, `HelmReleases`, `ImageUpdateAutomations`, | ||
| `Alerts`, `Receivers`, etc. | ||
|
|
||
| #### Caveats | ||
|
|
||
| As of v0.23.0, Flux does not enforce a service account to be specified on Flux `Kustomizations` and `HelmReleases`. | ||
| When a service account is not specified, Flux defaults to cluster-admin. | ||
| In order to enforce the tenant isolation, an admission controller such as Kyverno or OPA Gatekeeper must be used | ||
| to make the `.spec.serviceAccountName` a required field for the Flux custom resources created by tenants. | ||
|
|
||
| We provide an [example](https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml) | ||
| for enforcing service accounts using a Kyverno cluster policy. | ||
|
|
||
| As of v0.23.0, Flux allows for `Kustomizations` and `HelmReleases` to reference sources | ||
| (`GitRepositories`, `HelmRepositories` and `Buckets`) across namespaces. | ||
| In order to prevent tenants from accessing each other sources, an admission controller such as Kyverno or OPA Gatekeeper | ||
| must be used to block cross-namespace references. | ||
|
|
||
| We provide an [example](https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno-policies/flux-multi-tenancy.yaml) | ||
| for blocking source cross-namespace references using a Kyverno cluster policy. | ||
|
|
||
| ### Hard Multi-Tenancy | ||
|
|
||
| With hard multi-tenancy, the platform admins use Kubernetes Cluster API to create dedicated clusters for each tenant. | ||
| The Flux instance installed on the management cluster is responsible | ||
| for reconciling the cluster definitions belonging to tenants. | ||
| With hard multi-tenancy, the platform admins create dedicated clusters for each tenant. | ||
|
|
||
| When the tenants's clusters are created with Kubernetes Cluster API, the Flux instance | ||
| installed on the management cluster is responsible for reconciling the cluster | ||
| definitions belonging to tenants. | ||
|
|
||
| To enable GitOps for the tenant's clusters, the platform admins can configure the Flux instance running on the | ||
| management cluster to connect to the tenant's cluster using the `kubeConfig` generated by the Cluster API provider. | ||
| management cluster to connect to the tenant's cluster using the kubeconfig generated by the Cluster API provider | ||
| or by creating kubeconfig secrets for the clusters created by other means than Cluster API. | ||
|
|
||
| To configure Flux reconciliation of remote clusters, a Kubernetes secret containing a `kubeConfig` can be specified | ||
| in Flux `Kustomizations` and `HelmReleases` under `.spec.kubeConfig.secretRef`. Please consult the Flux API | ||
|
|
@@ -278,7 +207,7 @@ When using a Kubernetes Cluster API provider, the `kubeConfig` secret is automat | |
| make use of it without any manual actions. For clusters created by other means than Cluster API, the | ||
| platform team has to create the `kubeConfig` secrets to allow Flux access to the remote clusters. | ||
|
|
||
| As of Flux v0.23.0, we don't provide any guidance for cluster admins on how to generate the `kubeConfig` secrets. | ||
| As of Flux v0.24 (Nov 2021), we don't provide any guidance for cluster admins on how to generate the `kubeConfig` secrets. | ||
|
|
||
| ## Implementation History | ||
|
|
||
|
|
@@ -289,3 +218,8 @@ As of Flux v0.23.0, we don't provide any guidance for cluster admins on how to g | |
| [fluxcd/flux2-multi-tenancy](https://github.com/fluxcd/flux2-multi-tenancy). | ||
| - Soft multi-tenancy [CVE-2021-41254](https://github.com/fluxcd/kustomize-controller/security/advisories/GHSA-35rf-v2jv-gfg7) | ||
| "Privilege escalation to cluster admin on multi-tenant environments" was fixed in flux2 **v0.15.0**. | ||
|
|
||
| [mt]: https://github.com/fluxcd/flux2-multi-tenancy/tree/v0.1.0 | ||
| [kcsa]: https://fluxcd.io/docs/components/kustomize/kustomization/#role-based-access-control | ||
| [hcsa]: https://fluxcd.io/docs/components/helm/helmreleases/#role-based-access-control | ||
| [kubeconfig]: https://fluxcd.io/docs/components/kustomize/api/#kustomize.toolkit.fluxcd.io/v1beta2.KubeConfig | ||