Assign Priority Classes to tenants

[bsctl]

Describe the feature

In kubernetes, pods can have priority. Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible.

In a multi-tenant cluster where not all users are trusted, a tenant owner could create Pods at the highest possible priorities, causing other Pods to be evicted/not get scheduled.

It would be nice the cluster admin to assign one or more Priority Classes to a tenant. The Capsule Operator (i.e. its policy engine component) can control the admitted Priority Classes as for Storage and Ingress Classes.

What would the new user story look like?

The cluster admin creates Priority Classes with a given priority number

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: oil
value: 1000000
globalDefault: false

and then assigns it to a tenant

kind: Tenant
apiVersion: capsule.clastix.io/v1alpha1
metadata:
  name: oil
spec:
  priorityClasses:
    allowed:
      - oil
      - default
    allowedRegex: ''

Expected behavior

The tenant admin can only deploy pods with the allowed Priority Classes

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: oil

[prometherion]

To implement this, we need several changes.

• Planning new key for upcoming CRD version
• Implement a list of priority classes using annotations for the current CRD version (v1alpha1)
• Webhook for the Pods denying/allowing the execution
• capsule-proxy endpoint to let Tenant owners list their assigned priority classes

Am I missing something else?

[bsctl]

There are two Priority Classes already defined by default

$ kubectl get priorityclass
NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            26d
system-node-critical      2000001000   false            26d

and other can be defined by cluster admin. The Priority Class is a cluster-wide resource.

The workflow is very similar to Ingress and Storage Classes we already impemented

[prometherion]

The workflow is very similar to Ingress and Storage Classes we already implemented

And this means, on the capsule-proxy side, to implement the labeling with the resource name to allow listing of the resources.