[release-1.25] Ensure that no embedded controllers are using the admin RBAC

[brandond]

• Ensure that no embedded controllers are using the admin RBAC #7212 backport

[aganesh-suse]

RHEL 9.2

k3s -v
k3s version v1.25.10+k3s-d8aac17d (d8aac17d)
go version go1.19.9

There is a new "supervisor.kubeconfig" which was not there in the v1.25.10+k3s1 version

sudo ls /var/lib/rancher/k3s/server/cred
admin.kubeconfig       cloud-controller.kubeconfig  ipsec.psk  scheduler.kubeconfig
api-server.kubeconfig  controller.kubeconfig	    passwd     supervisor.kubeconfig

Additional certs for supervisor which were not in previous version:

sudo ls /var/lib/rancher/k3s/server/tls
client-admin.crt       client-controller.crt		client-kube-apiserver.key  client-supervisor.key  server-ca.nochain.crt
client-admin.key       client-controller.key		client-kubelet.key	   dynamic-cert.json	  service.current.key
client-auth-proxy.crt  client-k3s-cloud-controller.crt	client-kube-proxy.crt	   etcd			  service.key
client-auth-proxy.key  client-k3s-cloud-controller.key	client-kube-proxy.key	   request-header-ca.crt  serving-kube-apiserver.crt
client-ca.crt	       client-k3s-controller.crt	client-scheduler.crt	   request-header-ca.key  serving-kube-apiserver.key
client-ca.key	       client-k3s-controller.key	client-scheduler.key	   server-ca.crt	  serving-kubelet.key
client-ca.nochain.crt  client-kube-apiserver.crt	client-supervisor.crt	   server-ca.key	  temporary-certs

sudo cat /var/lib/rancher/k3s/server/manifests/web-helmchart.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: web
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: apache
  namespace: kube-system
spec:
  repo: https://charts.bitnami.com/bitnami
  chart: apache
  targetNamespace: web
  valuesContent: |-
    service:
      type: ClusterIP
    ingress:
      enabled: true
      hostname: www.example.com
    metrics:
      enabled: true

k get pods -n web
NAME                     READY   STATUS    RESTARTS   AGE
apache-955d6fff5-88g85   2/2     Running   0          4m45s

[aganesh-suse]

Verify with generated audit logs:

Followed steps provided here for creating the audit logs directory and starting the k3s service with the required args:
https://docs.k3s.io/security/hardening-guide#api-server-audit-configuration

sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs

sudo cat /var/lib/rancher/k3s/server/audit.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata 

Edit etc/systemd/system/k3s.service with lines:

ExecStart=/usr/local/bin/k3s \
    server \
    '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \
    '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' \

sudo systemctl daemon-reload
sudo systemctl restart k3s.service

Sample audit log lines verifying the 'system:k3s-supervisor' was used by the helm-controller:

sudo cat /var/lib/rancher/k3s/server/logs/audit.log | grep supervisor:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"d8cf09b4-425e-4c17-b512-7a2975cc20f3","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/events","verb":"create","user":{"username":"system:k3s-supervisor","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.25.10+k3s-d8aac17d (linux/amd64) k3s/d8aac17d","objectRef":{"resource":"events","namespace":"kube-system","name":"web-helmchart.1768ea9b41e303ae","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2023-06-15T19:06:55.467110Z","stageTimestamp":"2023-06-15T19:06:55.470838Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}

In the previous version 1.25.10 the grep supervisor would return empty.