feat: support additional resources across tenant namespaces

This attempts to address  projectcapsule#525 which is related to projectcapsule#416.

problem this commit is trying to solve
----

```
Alice would like to create several resources on each of their Namespaces.
Upon the creation of each Namespace, they have to create the desired resources on each Namespace
```

On the above linked tickets there are two proposed approaches

approach 01
----

Create a new resource `TenantResource`  something like this

```yaml
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
  name: database
  namespace: solar-management
spec:
  resyncPeriod: 60s
  additionalResources:
    namespaceSelector:
      matchLabels:
        tier: one
    items:
    - apiVersion: presslabs.io/v1beta1
      kind: MySQL
      spec:
        foo: bar
  clusterRoles:
    namespaceSelector:
      matchLabels:
        tier: one
    items:
      - name: database-admin
        subjects:
        - kind: Group
          name: developers
```

approach 02
-----

Extend `Tenant` to  support addtional resources

```yaml
apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: gas
spec:
  additionalResources:
    -  apiVersion: "cilium.io/v2"
       kind: CiliumNetworkPolicy
       metadata:
         name: "l3-rule"
      spec:
        endpointSelector:
          matchLabels:
             role: backend
       ingress:
       - fromEndpoints:
         - matchLabels:
             role: frontend
```

This commit implements approach `02` due to the following reasons

- The namespaces belong to the tenant already, extra `TenantResource` seems redundant given that the lifecycle of the additional resources are tied to the `Tenant`.

How does the crd look like now ?
----

```yaml
apiVersion: capsule.clastix.io/v1beta1
kind: Tenant
metadata:
  name: oil
spec:
  resyncPeriod: 60s
  additionalResources:
    namespaceSelector:
      matchLabels:
        tier: one
    items:
      - |
        apiVersion: v1
        kind: Pod
        metadata:
          name: nginx
          labels:
            app.kubernetes.io/name: proxy
        spec:
          containers:
            - name: nginx
              image: nginx:11.14.2
              ports:
                - containerPort: 80
                  name: http-web-svc
      - |
        apiVersion: v1
        kind: Service
        metadata:
          name: nginx-service
          labels:
            app.kubernetes.io/name: proxy
        spec:
          selector:
            app.kubernetes.io/name: proxy
          ports:
            - name: name-of-service-port
              protocol: TCP
              port: 80
              targetPort: http-web-svc
  owners:
    - name: alice
      kind: User
```

The difference with the proposed crd is, items are strings of k8s objects in yaml
format. I ran through decoding issues when I tried to use `Unstructured`  so I
decided to settle on strings instead.

How it works ?
----

We search for namespaces  specified by `namespaceSelector` that are owned by the
tenant. For each matched namespace, apply all resources specified in `additionalResources.items`
on any error, reschedule next reconciliation by `resyncPeriod`.

What is missing ?
----

- [ ] Tests
- [ ] What happens when a tenant is deleted ?
- [ ] What happens when a tenant is deleted ?
- [ ] Does `additionalRoleBindings` cover for  `clusterRoles` defined in approach 01?

I will wait for feedback/discussion on how to proceed from here.

Makefile

@@ -214,4 +214,4 @@ e2e/%: ginkgo
 		capsule \
 		./charts/capsule
 	$(GINKGO) -v -tags e2e ./e2e
-	kind delete cluster --name capsule
+	kind delete cluster --name capsule

api/v1beta1/additional_resources.go

@@ -0,0 +1,11 @@
+package v1beta1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+type AdditionalResources struct {
+	NamespaceSelector metav1.LabelSelector `json:"namespaceSelector"`
+	// list of k8s resources in yaml format
+	Items []string `json:"items"`
+}

api/v1beta1/tenant_types.go

@@ -9,6 +9,13 @@ import (
 
 // TenantSpec defines the desired state of Tenant
 type TenantSpec struct {
+	// Duration for reconciliation loop to poll and ensure AdditionalResources are
+	// in correct state.
+	//+kubebuilder:default="60s"
+	ResyncPeriod *metav1.Duration `json:"resyncPeriod,omitempty"`
+	// AdditionalResources specifies additional resources that will be created with
+	// the tenant
+	AdditionalResources *AdditionalResources `json:"additionalResources,omitempty"`
 	// Specifies the owners of the Tenant. Mandatory.
 	Owners OwnerListSpec `json:"owners"`
 	// Specifies options for the Namespaces, such as additional metadata or maximum number of namespaces allowed for that Tenant. Once the namespace quota assigned to the Tenant has been reached, the Tenant owner cannot create further namespaces. Optional.

charts/capsule/crds/capsuleconfiguration-crd.yaml

@@ -14,43 +14,43 @@ spec:
     singular: capsuleconfiguration
   scope: Cluster
   versions:
-    - name: v1alpha1
-      schema:
-        openAPIV3Schema:
-          description: CapsuleConfiguration is the Schema for the Capsule configuration API
-          properties:
-            apiVersion:
-              description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
-              type: string
-            kind:
-              description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
-              type: string
-            metadata:
-              type: object
-            spec:
-              description: CapsuleConfigurationSpec defines the Capsule configuration
-              properties:
-                forceTenantPrefix:
-                  default: false
-                  description: Enforces the Tenant owner, during Namespace creation, to name it using the selected Tenant name as prefix, separated by a dash. This is useful to avoid Namespace name collision in a public CaaS environment.
-                  type: boolean
-                protectedNamespaceRegex:
-                  description: Disallow creation of namespaces, whose name matches this regexp
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: CapsuleConfiguration is the Schema for the Capsule configuration API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: CapsuleConfigurationSpec defines the Capsule configuration
+            properties:
+              forceTenantPrefix:
+                default: false
+                description: Enforces the Tenant owner, during Namespace creation, to name it using the selected Tenant name as prefix, separated by a dash. This is useful to avoid Namespace name collision in a public CaaS environment.
+                type: boolean
+              protectedNamespaceRegex:
+                description: Disallow creation of namespaces, whose name matches this regexp
+                type: string
+              userGroups:
+                default:
+                - capsule.clastix.io
+                description: Names of the groups for Capsule users.
+                items:
                   type: string
-                userGroups:
-                  default:
-                    - capsule.clastix.io
-                  description: Names of the groups for Capsule users.
-                  items:
-                    type: string
-                  type: array
-              type: object
-          type: object
-      served: true
-      storage: true
+                type: array
+            type: object
+        type: object
+    served: true
+    storage: true
 status:
   acceptedNames:
     kind: ""
     plural: ""
   conditions: []
-  storedVersions: []
+  storedVersions: []