Support Any Additional resource

[oliverbaehler]

Describe the feature

I would like to have a generic approach on resources, which are created for each new namespace belonging to a tenant. The same behavior that .spec.additionalRoleBindings and .spec.networkPolicies already implement, but for any kubernetes resource.

As of why:

We are currently moving to Cilium Network Policies, and we would require each namespace to implement those policies (same could also apply for calico resources or really anything else). So instead of writing an own workaround, it would make sense to be able to declare something like this:

---
apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: gas
spec:
  additionalResources:
    -  apiVersion: "cilium.io/v2"
       kind: CiliumNetworkPolicy 
       metadata:
         name: "l3-rule"
      spec:
        endpointSelector:
          matchLabels:
             role: backend
       ingress:
       - fromEndpoints:
         - matchLabels:
             role: frontend

What would the new user story look like?

How would the new interaction with Capsule look like? E.g.

1. A new Tenant is created with additional resources
2. The resources could be validated on tenant creation and reject the tenant creation if there's an error, if not the tenant is created.
3. A new User creates/assigns a new namespace to the previously created tenant.
4. the additional resources are added to the namespace once the operator has reconciled.
5. Everyone is happy.

Expected behavior

A clear and concise description of what you expect to happen.

When I have defined .spec.additionalResources on a tenant, those resources are created for each namespace that is assigned or created in that tenant.

[bsctl]

@oliverbaehler thanks for suggesting this improvement. I think this will require some sort of refactoring of the code. Let's to see what @prometherion says.

[prometherion]

I'm a bit late on this, thanks for the patience!

This feature would be awesome and I couldn't agree more, despite some limitations I'd like to evaluate with the community.

We have to keep in mind that CRDs are, essentially, OpenAPI v3 specifications: we got a schema that must stick to it and that's great, I love it when everything is strictly typed since it's less error-prone and safer, especially at compile time.
The first challenge would be how to structure the /spec/additionalResources field: although from a Go struct type point of view a []interface{} type could do the trick, the resulting OpenAPI v3 wouldn't be so happy (ndr: not sure, should be tested), and even if it wouldn't be the case, we would have to deal with another issue, how to apply these resources.

I'm really happy with the unversioned client set offered by controller-runtime since we just need to register the schemes and everything is done under the hoods, and this simplicity is a trade-off on the flexibility required by this feature request: that would be impossible using this client, or at least would be required the dynamic client that we're not using.

The dynamic client setup looks easy as pie, also considering we already have whatever is needed: I'm more concerned regarding the naming of the resources that must be prefixed to avoid collision (especially for the cluster-scoped resources) and the set of the Namespace field.

tl;dr; we would be required to deal with https://k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.Unstructured type that seems able to set Namespace and Name.

A lot of words and a simple question: should we implement this on our own or, rather, take advantage of HNC? We would just need to point the Tenant to a template Namespace and that controller would start replicating the resources without hacking so much our code-base, although adding a tough dependency since an external component would be required, not sure if we can grab their code and start it using our Manager, need to be investigated.

[oliverbaehler]

Another Solution to this would be just using generate Functionalities of e.g Kyverno (https://kyverno.io/docs/writing-policies/generate/). You could write policies which would generate the resources based on namespace labels. As far as I am concerned this would also cover the use-case. Now you would have to assume that kyverno is already installed and therefor would have kinda a dependency. But I would also say that such use cases occur in complexer setup where the probability is high, that such policy management is already deployed. This would not required to hack your codebase and also would not require a template namespace you mentioned. But it would mostly not fit in the incrementing name convention etc. But i don't know if that's a big thing.

Actually Kyverno implements a clone feature which kinda does the same thing as your proposed with the template namespace:

https://kyverno.io/docs/writing-policies/generate/#clone-a-configmap-and-propagate-changes

I am mentioning that because it could make sense to not inflate the capsule project to a all use-case resolving solution. But that's up to you guys if you think that feature should be covered by capsule.

Some Examples.

Let's say you would want to add a ciliumPolicy for each Namespace that's part of the capsule

policy.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenancy-cilium-inject
spec:
  rules:
  - name: tenancy-cilium-inject
    match:
      resources:
        kinds:
        - Namespace
      selector:
        matchExpressions:
          - key: "tenant.label"
            operator: Exists
    exclude:
      resources:
        namespaces:
        - capsule-system
    generate:
      synchronize: true
      kind: CiliumNetworkPolicy 
      data:
        apiVersion: "cilium.io/v2"
        metadata:
          name: "l3-rule"
          namespace: "{{request.object.metadata.name}}"
        spec:
          endpointSelector:
            matchLabels:
              role: backend
          ingress:
           - fromEndpoints:
             - matchLabels:
                 role: frontend

i didn't test it, but just that you get the idea of what i am talking.

[sathieu]

@prometherion Kiosk has this feature with very powerful templating and helm: https://github.com/loft-sh/kiosk#5-working-with-templates

There is no validation:

https://github.com/loft-sh/kiosk/blob/2aa40678f3a0726f1d2dd7dc31cfc9475e249023/pkg/apis/config/v1alpha1/template_types.go#L54-L59

[oliverbaehler]

@prometherion Still the same thoughts on this? I mean we could start a very simple feature where it just renders all the resources and validation for the basic kubernetes fields (kind,metadata) and then go from there

[oliverbaehler]

After some thoughts i may have an extra idea for this feature.

What if we would also add a selector, with which we could target only specific namespaces which create the additional resource (if no selector set create in all)

So the an extra resource would look something like this

apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: gas
spec:
  additionalResources:
    - selector:
        # For any namespace that matches the regexp 
        regex: "^.*-reserved$"
        # For any namespace the has the team "marketing" label
        matchLabels:
           team: "marketing"
      template: | 
         apiVersion: "cilium.io/v2"
         kind: CiliumNetworkPolicy 
         metadata:
           name: "l3-rule"
         spec:
           endpointSelector:
             matchLabels:
               role: backend
         ingress:
         - fromEndpoints:
           - matchLabels:
               role: frontend

This might also help with building a tenant based rbac, since you could assign rolebinding based on the namespace's attributes. I don't know if that would help your team @MaxFedotov.

It's just an idea, not yet exactly sure how much sense it makes.

[prometherion]

@oliverbaehler JFI there's a feature request (#525 ) that is pretty similar to this. Could you provide feedback on that?