[RFC] Define Flux tenancy models

[stefanprodan]

The main goal of this RFC is to define the Kubernetes tenancy models supported by Flux.

This PR attempts to document the status quo, and should provide clarity of what multi-tenancy capabilities Flux has. It also functions as a base for rewriting the loose proposal in #582 into well scoped RFCs.

[phillebaba]

Does this mean that tenants should not configure their own sources? In the tenants section it does however state "Register their sources with Flux". I might just be misinterpreting the meaning.

[stefanprodan]

This is about the cluster control plane desired state as in cluster-wide resources, controllers, etc.

[stefanprodan]

I've added a new section "Tenants Onboarding". Hopefully this clarifies that tenants can add their app repos to their main repo which is registered by admins.

[squaremo]

I was initially confused by this as well. It might pay to spend some paras up front explaining which git repositories are assumed to exist, and how they are used (i.e., what they contain).

[makkes]

I wonder if it would make sense to introduce a definition of the "control plane" in a separate paragraph at the beginning of the RFC somewhere. I'm thinking of everything that is either shared among tenants or created as part of the on-boarding of a tenant; the Flux instance itself, components such as Gatekeeper and resources such as ServiceAccounts.

It might also be helpful to explain the repo hierarchy: Each tenant has a root repo that's created by cluster admins and as many subsequent repos maintained by themselves.

[souleb]

It is said here above that cluster-admin Onboard tenants by registering their Git repositories with Flux. This might need a clarification on separation of concern

[stefanprodan]

I've added a new section "Tenants Onboarding". Hopefully this clarifies that tenants can add their app repos to their main repo which is registered by admins.

[yebyen]

This is a great overview of the current state-of-the-art with lots of good references for follow-up education. 👍 LGTM with changes, a few typos corrected.

[squaremo]

It's not uncommon to have a "memorandum" RFC which describes the status quo, rather than proposing a new design. It seems like a needless indirection to use an RFC to propose new documentation, giving the content verbatim, though. I would expect either

• an RFC which will stand itself as the description of the status quo at some point (and, e.g., documentation can point at it, or paraphrase it while still relevant); or,
• a PR with the new documentation.

Given the goal of building up to new designs, I think the first is the appropriate form here (and would require only a little adaptation).

[jonathan-innis]

LGTM 🚀

[nikkomiu]

Looks good! (I'm with the MS/Azure group)

[squaremo]

Overall comment: 💯 👏 for the effort to definitively set out tenancy models for Flux. I think the content could be more pointed in how it does that, by

• being clear about which bits are definitions or assumptions;
• describing the models more directly -- in some places, the text lapses into being "how to" rather than being definitive.

[squaremo]

These aren't the only ways to set up a multi-tenant Flux system though, are they? This feels like it's partly a guide to good practice, rather than a reference. In which case, the language could be more like

• Define two models for multi-tenancy, "soft multi-tenancy" and "hard multi-tenancy"
• Explain when each is appropriate
• Describe a reference implementation of each model with Flux

(this distinguishes between definitions, which are normative; and implementations, of which there may be variations).

[squaremo]

Is this a strict requirement of the model? Or could the kubeConfig secrets come from some other mechanism, e.g., if clusters are constructed with terraform, or with clicking buttons.

[squaremo]

I'm not clear on whether applying things remotely is required for the hard multi-tenancy, or kind of a mixed-in concern (if you're giving each tenant a cluster, you probably have a management cluster, so let's base that model on that assumption ...). Could you provide some justification in the text for this approach? Or explicitly give it as an assumption.

[makkes]

D2iQ actually currenty implements hard multi-tenancy without kubeConfig but instead we have controllers install Flux and create the sync resources on each tenant cluster. So I suppose kubeConfig is one of several ways to enforce hard multi-tenancy.

[squaremo]

This and the examples following are "how to set up multi-tenancy" rather than describing the model or implementation. Demonstrating how to set it up is not a goal, in the text as it stands -- neither are describing the model or its implementation, but according to the PR title, perhaps they should be.

I suggest reworking this section to describe what the soft-tenancy model requires of RBAC (things like "each tenant namespace has a service account, with these bindings"). Telling people how to make it so conveniently, as you have here, is useful as extra information, but informative rather than definitive.

[pjbgf]

Expanding the RBAC recommendations here would be really useful.
It would be good to ensure we cater for protecting the tenant's service account from being misused.
Here's some ideas:

A) Vanila K8S

The Platform Admin would pre-create all namespaces the tenant will use ahead of time, setting access via rolebindings for all the tenant's namespaces.
All Flux objects are created at the tenant flux namespace.

flux-tenant-alpha
├── flux-tenant-alpha-serviceaccount
├── flux-tenant-alpha-rolebinding
├── podinfo-helmrepository
└── podinfo-helmrelease
apps
├── flux-tenant-alpha-rolebinding
└── podinfo

B) HNC

The Platform Admin pre-creates the tenant top level namespace, with its service account and rolebindings.
All Flux objects are created at the tenant top level namespace.
Tenants can create subnamespaces and deploy apps to it.

flux-tenant-alpha
├── flux-tenant-alpha-serviceaccount
├── flux-tenant-alpha-rolebinding
├── podinfo-helmrepository
├── podinfo-helmrelease
└── [ns] apps
         ├── flux-tenant-alpha-rolebinding
         └── podinfo

In both cases, the "deployment" service account is never placed on a namespace that is shared with other applications.

If a tenant's flux namespace needs to have mixed use (shared between applications and flux components), it would require admission controllers to block the misuse of the tenant's service account.

C) Vanila K8S + Admission Controllers

flux-tenant-alpha
├── flux-tenant-alpha-serviceaccount (i.e. kyverno policy to block misuse of this service account)
├── flux-tenant-alpha-rolebinding
├── podinfo-helmrepository
└── podinfo-helmrelease
└── podinfo

[squaremo]

The paras here are a nice, and concise, explanation 💟

[pjbgf]

+1

[squaremo]

The text above says they come from Cluster API.

[makkes]

Suggested change

	The documentation [here](https://fluxcd.io/docs/) describes the security model of Flux.
	The documentation [here](https://fluxcd.io/docs/security/) describes the security model of Flux.

Isn't this the more concrete page? The main one doesn't mention security.

[makkes]

I recall someone (maybe it was @stefanprodan) telling me there shouldn't be multiple instances of Flux running on a single cluster (which could help in isolating tenants). Maybe that notion should be part of this doc as some kind of "official guidance"?

[hiddeco]

There are configuration options in which this theoretically still is a solution, but need to adhere to a set of rules that do not apply to most.

[makkes]

Suggested change

	The tenancy models assume two types of user: platform admins and tenants.
	The existing Flux tenancy models assume two types of user: platform admins and tenants.

Not sure if that's the intention here but I figure a bit of clarification of which tenancy model we're talking about here might be helpful.

[makkes]

I wonder if it would make sense to introduce a definition of the "control plane" in a separate paragraph at the beginning of the RFC somewhere. I'm thinking of everything that is either shared among tenants or created as part of the on-boarding of a tenant; the Flux instance itself, components such as Gatekeeper and resources such as ServiceAccounts.

It might also be helpful to explain the repo hierarchy: Each tenant has a root repo that's created by cluster admins and as many subsequent repos maintained by themselves.

[makkes]

D2iQ actually currenty implements hard multi-tenancy without kubeConfig but instead we have controllers install Flux and create the sync resources on each tenant cluster. So I suppose kubeConfig is one of several ways to enforce hard multi-tenancy.

[makkes]

We should mention here that hard multi-tenancy can be combined with soft multi-tenancy to get around this limitation.

[csand-msft]

AKS multi-tenancy doc: https://docs.microsoft.com/azure/aks/operator-best-practices-cluster-isolation

[pjbgf]

On the four sources below (Kubernetes, GCP, Azure and AWS) only AWS uses the terms soft and hard for multi-tenancy. It would be useful to expand slightly here to clarify what we mean by it, which may speak to the RFC's goal of "Explain when each model is appropriate.".

Some ideas:

	Soft Multi-tenancy	Hard Multi-tenancy
Tenants may share cluster with other tenants	Yes	No
Tenants may share cluster with the flux management instance	Yes	No
Tenants access to cluster-wide resources	Limited	Unrestricted

[pjbgf]

+1

[pjbgf]

I wonder whether we need a small multi-tenancy security section on its own, as similar points may be valid for hard multi-tenancy - although at a lower level of the stack.

The key point being that flux support several multi-tenancy use cases, but the Platform Admin is ultimately the responsible for ensuring the correct level of isolation is enforced between the tenants, based on their own security requirements.

[pjbgf]

Expanding the RBAC recommendations here would be really useful.
It would be good to ensure we cater for protecting the tenant's service account from being misused.
Here's some ideas:

A) Vanila K8S

The Platform Admin would pre-create all namespaces the tenant will use ahead of time, setting access via rolebindings for all the tenant's namespaces.
All Flux objects are created at the tenant flux namespace.

flux-tenant-alpha
├── flux-tenant-alpha-serviceaccount
├── flux-tenant-alpha-rolebinding
├── podinfo-helmrepository
└── podinfo-helmrelease
apps
├── flux-tenant-alpha-rolebinding
└── podinfo

B) HNC

The Platform Admin pre-creates the tenant top level namespace, with its service account and rolebindings.
All Flux objects are created at the tenant top level namespace.
Tenants can create subnamespaces and deploy apps to it.

flux-tenant-alpha
├── flux-tenant-alpha-serviceaccount
├── flux-tenant-alpha-rolebinding
├── podinfo-helmrepository
├── podinfo-helmrelease
└── [ns] apps
         ├── flux-tenant-alpha-rolebinding
         └── podinfo

In both cases, the "deployment" service account is never placed on a namespace that is shared with other applications.

If a tenant's flux namespace needs to have mixed use (shared between applications and flux components), it would require admission controllers to block the misuse of the tenant's service account.

C) Vanila K8S + Admission Controllers

flux-tenant-alpha
├── flux-tenant-alpha-serviceaccount (i.e. kyverno policy to block misuse of this service account)
├── flux-tenant-alpha-rolebinding
├── podinfo-helmrepository
└── podinfo-helmrelease
└── podinfo