Adding Capsule to existing cluster

[barrydobson]

Hi,
I've spent quite a lot of time look at the documentation, and other issues/use cases in order to understand how to use Capsule for my use case, but I'm unable to find any details.

I have an EKS which I'm looking to add Capsule to. This cluster is fully managed by FluxV2, both Kustomize and Helm controllers. The cluster has quite a bit of 'system' pods that are running. E.G. prometheus to gather metrics, Flux for deployments, and various other things like Fluent bit and Istio.

These are added as part of a cluster bootstrapping process, which would also add the Tenant resource and namespaces that tenants would use. Tenants will then be able to manage application deployments using the existing Flux controllers.

The problem I've got is that once I add Capsule to the cluster, Flux is unable to reconcile anything as they are not assigned to a tenant.

Is anyone else using a similar pattern and can help me understand how to use Capsule this way?

[oliverbaehler]

Hi, I can help you out. We are using the same pattern without any problems.

The problem I've got is that once I add Capsule to the cluster, Flux is unable to reconcile anything as they are not assigned to a tenant.

What kind of RBAC are you using for Flux. Is Flux in a Tenant as Well? We don't deploy Flux into a Capsule Tenant.

[barrydobson]

Hi @oliverbaehler Thanks for your response.

I am hoping not to have flux in a tenant. I would like certain namespaces to be unmanaged by Capsule, these would be namespaces containing things such as monitoring, flux etc. Once deployed I want to create tenants, then have flux watch the tenant repository and be able to deploy into that tenant.

What I've found is that once Capsule is on there, it seems to want to control everything, so flux fails with a message about not belonging to a tenant.

For flux, I'm using the standard RBAC. each controller is running with a service account, which has the cluster role binding to cluster admin

[prometherion]

Hey @barrydobson, thanks for opening your first Capsule issue!

The problem I've got is that once I add Capsule to the cluster, Flux is unable to reconcile anything as they are not assigned to a tenant.

At first sight, it seems you're using the entry system:serviceaccounts in the CapsuleConfiguration/spec/groups key: is that correct?

[barrydobson]

my capsule usergroups are

- "capsule.clastix.io"
- "system:serviceaccounts:flux-system"

[oliverbaehler]

Well since you consider all the SAs in the namespace flux-system: are they owners of the tenants they should reconcile stuff in?

We deploy a dedicated serviceaccount (eg. Called orchestrator in the namespace tenants. Then we add system:serviceaccounts:tenants:orchestrator as capsule group. This sa we use to deploy actual tenants and make him owner of each capsule tenant, so he can bootstrap additional namespaces and stuff.

This way flux does not interfer with the capsule tenants. Hope this helps

[barrydobson]

This is kind of what I've been trying. I have flux-system namespace, with the the service accounts, but reconciliation always fails because of the no tenant issue.

[prometherion]

If the Flux ServiceAccount deployed in flux-system should be able to create Namespace resources, obviously, it must be added to the Capsule user group array.

With that said, to deploy Namespaces, it must have one or more Tenant resources assigned, otherwise, the webhook will prevent any creation operation: it's like Alice, if they ain't any Tenant assigned, no self-service Namespace for them.

You can try to overcome this adding the Flux ServiceAccount as a Tenant owner for each Tenant it has to manage, YMMV.

apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: my-tenant
spec:
  owners:
  - kind: User
    name: barrydobson
  - kind: ServiceAccount
    name: system:serviceaccounts:tenants:orchestrator

With this approach, if the Namespace is missing, Flux will be allowed to create it and get it controlled by the Tenant my-tenant.

---

If Flux must, instead, only perform some reconciliation as creating Deployment, Services, and other basic resources in the Namespace resources managed by a Tenant, I think you can avoid setting its ServiceAccount in the Capsule user groups option: obviously, required RBAC must be put in place to allow it to manage everything, but I'm expecting its already cluster-admin.

[barrydobson]

I've done some more testing with this over the weekend.
I've managed to have flux work as expected on namespaces which are not a tenant by adding the service account to the capsule user groups setting.
The net issue I'm facing is that when flux deploys a tenant, then creates a namespace, it's not being picked up as tenant. Therefore none of the rules are being applied.

I'm not sure if I'm missing something, or the documentation is missing something?

It mentions a section on a GitOps use case, such as assigning the SA 'robot' as a tenant owner but falls short of any other detail that's required to have it actually work.

Thanks for you help

[MaxFedotov]

hi @barrydobson. Is you flux account added as tenant owner to the tenant where you try to create namespace?

[oliverbaehler]

Also note that the Kustomization/Helmrelease must impersonate as a serviceAccount which is considered for tenants.

Note that we can't reference sa across namespaces, so the kustomization would look something like this:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: tenant-dev-ux
  namespace: tenants
spec:
  serviceAccountName: orchestrator
...

This assumes that system:serviceaccounts:tenants:orchestrator as capsuleGroup. Then it should already work. Bootstraping additional namespaces is via giving this sa owner access to the tenant.

[barrydobson]

Thanks for all of the suggestions, I think I'm making progress.

Flux is now able to provision inside a tenant. The last major blocker I have is that now flux won't reconcile any namespace that isn't a tenant, for example, it's own namespace.

[oliverbaehler]

For anyone else reading this, we also had to deploy an additional clusterRole/Binding so that serviceAccounts could actually deploy namespaces via Kustomizations (this requires patch privileges). Otherwise it fails like so:

flux reconcile kustomization harbor -n tenants
► annotating Kustomization harbor in tenants namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✗ Kustomization reconciliation failed: Namespace/sre-harbor dry-run failed, reason: Forbidden, error: namespaces "sre-harbor" is forbidden: User "system:serviceaccount:tenants:sre-system-deploy" cannot patch resource "namespaces" in API group "" in the namespace "sre-harbor"

So we added the following, which grants patch rights to namespaces for all tech accounts for all tenants:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: capsule-namespace-patcher
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: capsule-namespace-patcher
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: capsule-namespace-patcher
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:tenants

Once I have some time I would consider contributing a flux guide for capsule. So that the onboarding is easier for flux users with capsule. However it's always an architecture question, based on the repository setup etc.

[prometherion]

Thanks for this, @oliverbaehler, really appreciated it!

Once I have some time I would consider contributing a flux guide for capsule

This would be absolutely great, what about creating an issue to track it down?

[oliverbaehler]

@barrydobson Progress on #528 might be interesting for you

[prometherion]

Closing this, we've introduced full support for Flux v2.