Pod Priority Class support

[prometherion]

Partially addressing #257.

We have to plan a new version to handle this feature.

[bsctl]

@prometherion
when the feature is not enabled, i.e. the annotations priorityclass.capsule.clastix.io/allowed and priorityclass.capsule.clastix.io/allowed-regex are not specified into the tenant manifest, the admission webhook should admit pods with any priorityClassName, including no priorityClassName:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: critical

when the feature is enabled, i.e. the annotations priorityclass.capsule.clastix.io/allowed and priorityclass.capsule.clastix.io/allowed-regex are specified into the tenant manifest, the admission webhook should admit pods without priorityClassName, i.e.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: 

[prometherion]

Totally agree @bsctl, just fixed with the expected behavior.

[bsctl]

@prometherion

It's better but a case is still not correct, imo. Pls, see below

1. As cluster admin, create oil tenant and priority class

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  annotations:
    priorityclass.capsule.clastix.io/allowed: oil
  name: oil
spec:
  owner: # required
    name: engineering
    kind: Group

---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: oil
value: 1000000
globalDefault: false

2. As tenant owner, create namespace

$ kubectl create ns `oil-production`

3. As tenant owner, create a pod with oil priority class

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
  namespace: oil-production
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: oil

4. apply

kubectl apply -f pods.yaml -n oil-production
pod/nginx created

Ok!

5. As tenant owner, create a pod with system-cluster-critical priority class

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
  namespace: oil-production
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  priorityClassName: system-cluster-critical

6. apply

kubectl apply -f pods.yaml -n oil-production
Error from server: error when creating "pods.yaml": admission webhook "podpriority.capsule.clastix.io" denied the request: Pod Priorioty Class nginx is forbidden for the current Tenant: use one from the following list (oil)

Ok!

7. As tenant owner, create a pod without priority class

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
  namespace: oil-production
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

8. apply

kubectl apply -f pods.yaml -n oil-production
Error from server: error when creating "pods.yaml": admission webhook "podpriority.capsule.clastix.io" denied the request: Pod Priorioty Class nginx is forbidden for the current Tenant: use one from the following list (oil)

Ko!

A null Priority Class should be allowed, i.e. null is a valid choice. If we do not consider null, then the tenant owner is forced to specify a Priority Class for all pods.

[prometherion]

A null Priority Class should be allowed, i.e. null is a valid choice. If we do not consider null, then the tenant owner is forced to specify a Priority Class for all pods.

I'm feeling awkward missing this, but yeah, going to work to address the case!

[prometherion]

Tested manually with latest force-pushed changes:

$ kubectl describe tnt oil
Name:         oil
Namespace:    
Labels:       <none>
Annotations:  priorityclass.capsule.clastix.io/allowed: oil
API Version:  capsule.clastix.io/v1alpha1
Kind:         Tenant
Metadata:
  Creation Timestamp:  2021-05-28T07:25:35Z
  Generation:          1
  Resource Version:  212798
  UID:               d5415961-ad68-4f4b-84a1-6f3bc342ddcf
Spec:
  Owner:
    Kind:  Group
    Name:  engineering
Status:
  Namespaces:
    oil-production
  Size:  1
Events:  <none>

$ cat <<EOF | kubectl apply -f -
pipe heredoc> apiVersion: v1
kind: Pod    
metadata:    
  name: nginx
  labels:    
    env: test
  namespace: oil-production 
spec:   
  containers:
  - name: nginx   
    image: nginx  
    imagePullPolicy: IfNotPresent
pipe heredoc> EOF

pod/nginx created

[bsctl]

lgtm @prometherion many thanks!