Convert default rke2.yaml over to non system:admin account for security auditing

[zlmitchell]

Is your feature request related to a problem? Please describe.

In the audit logs for rke2, actions provided from the host system, using the /etc/rancher/rke2/rke2.yaml are issued by the same user as the internal k8s cluster orchestration ['system:admin'].

Describe the solution you'd like

It would improve security and auditing if this default rke2.yaml on the nodes was registered using a seperate RBAC account to better track host run commands and changes back .

Describe alternatives you've considered

This could be done via RBAC and the manifest directory then updating the rke2.yaml on each host. But we are unsure of the consequences of manipulating this file on each node would do.

Additional context

This would also solve the issue of someone obtaining the rke2.yaml and using it remotely, So that that key could be revoked/rotated on the user/token.

[brandond]

issued by the same user as the internal k8s cluster orchestration

Can I ask which specific other operations you're seeing occur as the system:admin user? A better ask might be to have dedicated users for all the embedded controllers. We should already have this for most things, but its possible we missed something.

This could be done via RBAC and the manifest directory then updating the rke2.yaml on each host.

I don't believe that we have plans to create a different default admin account or RBAC, but we can leave this open for future consideration.

I will note that the file is rewritten on startup to ensure that the credentials are valid, so even if you change its contents, those changes will be reverted. It might be better to simply have an organization policy against using the default admin kubeconfig for cluster management.

This would also solve the issue of someone obtaining the rke2.yaml and using it remotely, So that that key could be revoked/rotated

Kubernetes does not actually support certificate revocation (it does not check CRLs), so your only option is to create unique roles and clean up the associated RBAC in order to revoke access.

This is why using service account tokens is a better idea; the tokens are bound to a SA that can be deleted, and the token is invalidated when the SA goes away.

[zlmitchell]

Can I ask which specific other operations you're seeing occur as the system:admin user? A better ask might be to have dedicated users for all the embedded controllers. We should already have this for most things, but its possible we missed something.

We see a lot of action all over the place here are a few. You can see similar audit logs in any environment.

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08bc78d4-2a10-4b4a-bf56-7003707a28be","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/configmaps/rke2","verb":"update","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1 (linux/amd64) rke2/2616002259455211fb8af2e2eb8f996593c18b86","objectRef":{"resource":"configmaps","namespace":"kube-system","name":"rke2","uid":"8781b187-576f-4982-97d3-3ee1ca27fca1","apiVersion":"v1","resourceVersion":"2883048"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:46:07.379944Z","stageTimestamp":"2023-03-27T15:46:07.383332Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"23291f3d-f32d-4581-93c9-89825a200422","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?allowWatchBookmarks=true\u0026resourceVersion=2876600\u0026timeout=9m49s\u0026timeoutSeconds=589\u0026watch=true","verb":"watch","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1 (linux/amd64) rke2/2616002259455211fb8af2e2eb8f996593c18b86","objectRef":{"resource":"clusterrolebindings","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:36:19.756862Z","stageTimestamp":"2023-03-27T15:46:08.759134Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"87a172b0-dc0d-4842-9856-922729e5f78b","stage":"ResponseStarted","requestURI":"/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?allowWatchBookmarks=true\u0026resourceVersion=2876600\u0026timeout=9m45s\u0026timeoutSeconds=585\u0026watch=true","verb":"watch","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1 (linux/amd64) rke2/2616002259455211fb8af2e2eb8f996593c18b86","objectRef":{"resource":"clusterrolebindings","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:46:08.760311Z","stageTimestamp":"2023-03-27T15:46:08.761931Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"efacc30a-eaa6-4b52-beba-8caf22d0a655","stage":"ResponseComplete","requestURI":"/apis/networking.k8s.io/v1/namespaces/kube-public/networkpolicies/rke2-flannel-host-networking","verb":"get","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1 (linux/amd64) rke2/2616002259455211fb8af2e2eb8f996593c18b86","objectRef":{"resource":"networkpolicies","namespace":"kube-public","name":"rke2-flannel-host-networking","apiGroup":"networking.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:47:04.453918Z","stageTimestamp":"2023-03-27T15:47:04.455150Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"f3146d7b-5d6c-4cad-a7cd-62007615ff29","stage":"ResponseComplete","requestURI":"/apis/networking.k8s.io/v1/namespaces/kube-public/networkpolicies/rke2-flannel-host-networking","verb":"update","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1 (linux/amd64) rke2/2616002259455211fb8af2e2eb8f996593c18b86","objectRef":{"resource":"networkpolicies","namespace":"kube-public","name":"rke2-flannel-host-networking","uid":"b3a920a8-e379-4638-99a4-20219793608e","apiGroup":"networking.k8s.io","apiVersion":"v1","resourceVersion":"2239"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"NetworkPolicy","apiVersion":"networking.k8s.io/v1","metadata":{"name":"rke2-flannel-host-networking","namespace":"kube-public","uid":"b3a920a8-e379-4638-99a4-20219793608e","resourceVersion":"2239","generation":4,"creationTimestamp":"2023-03-23T18:18:34Z","managedFields":[{"manager":"[email protected]","operation":"Update","apiVersion":"networking.k8s.io/v1","time":"2023-03-23T18:18:34Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:ingress":{},"f:policyTypes":{}}}}]},"spec":{"podSelector":{},"ingress":[{"from":[{"ipBlock":{"cidr":"10.42.0.0/32"}},{"ipBlock":{"cidr":"10.42.1.0/32"}},{"ipBlock":{"cidr":"10.42.2.0/32"}}]}],"policyTypes":["Ingress"]}},"requestReceivedTimestamp":"2023-03-27T15:47:04.455620Z","stageTimestamp":"2023-03-27T15:47:04.457520Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4296d1c9-5077-4084-9a2e-087383926957","stage":"ResponseComplete","requestURI":"/apis/k3s.cattle.io/v1/addons?allowWatchBookmarks=true\u0026resourceVersion=2883390\u0026timeout=9m38s\u0026timeoutSeconds=578\u0026watch=true","verb":"watch","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1 (linux/amd64) rke2/2616002259455211fb8af2e2eb8f996593c18b86","objectRef":{"resource":"addons","apiGroup":"k3s.cattle.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:51:59.985078Z","stageTimestamp":"2023-03-27T16:01:37.987189Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}

Here is me running commands locally on a node

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"bdca2f51-02c6-4c54-8c86-938fb7ed5b08","stage":"ResponseComplete","requestURI":"/api/v1/nodes?limit=500","verb":"list","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubectl/v1.23.10+rke2r1 (linux/amd64) kubernetes/7e54d50","objectRef":{"resource":"nodes","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:28:18.242333Z","stageTimestamp":"2023-03-27T15:28:18.247678Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"ee1cff56-f8af-4de6-8743-ff903ea56b02","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/pods?limit=500","verb":"list","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubectl/v1.23.10+rke2r1 (linux/amd64) kubernetes/7e54d50","objectRef":{"resource":"pods","namespace":"kube-system","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:29:43.143650Z","stageTimestamp":"2023-03-27T15:29:43.160277Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"9c703573-72ce-4954-a73c-b54277e3eb89","stage":"ResponseComplete","requestURI":"/api/v1/secrets?limit=500","verb":"list","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubectl/v1.23.10+rke2r1 (linux/amd64) kubernetes/7e54d50","objectRef":{"resource":"secrets","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:30:23.590596Z","stageTimestamp":"2023-03-27T15:30:23.608588Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"93b4c590-1813-4971-96cd-bf3a2c7a7b03","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/minio/secrets/minio-trusted-certs","verb":"get","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubectl/v1.23.10+rke2r1 (linux/amd64) kubernetes/7e54d50","objectRef":{"resource":"secrets","namespace":"minio","name":"minio-trusted-certs","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-03-27T15:32:06.976422Z","stageTimestamp":"2023-03-27T15:32:06.979365Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a0190888-c171-4117-a0ee-ce16b3c0dc6b","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create","verb":"create","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kubectl/v1.23.10+rke2r1 (linux/amd64) kubernetes/7e54d50","objectRef":{"resource":"secrets","namespace":"default","name":"TEST","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"Invalid","code":422},"requestReceivedTimestamp":"2023-03-27T15:43:18.110780Z","stageTimestamp":"2023-03-27T15:43:18.118692Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","mutation.webhook.admission.k8s.io/round_0_index_9":"{\"configuration\":\"rancher.cattle.io\",\"webhook\":\"rancher.cattle.io\",\"mutated\":false}"}}

I will note that the file is rewritten on startup to ensure that the credentials are valid, so even if you change its contents, those changes will be reverted. It might be better to simply have an organization policy against using the default admin kubeconfig for cluster management.

In our case our organization policy requires STIG and auditing compliance so the most effective route would be to allow the disabling of this account/file after a given period or in general. Or least disruptive would be to ensure that the use of this file is auditable back to a source/user.

This is why using service account tokens is a better idea; the tokens are bound to a SA that can be deleted, and the token is invalidated when the SA goes away.

This is exactly what I mean by this request. The rke2.yaml should be configured to use a service account per node to ensure it can be revoked/audited.

[zlmitchell]

Probably need to extend this thinking to the manifests directory as well. To audit which node a CR yaml was dropped into that folder.

[brandond]

The Rancher Federal folks regularly assist in deploying RKE2 into STIG hardened environments and this has never been raised as an issue. There needs to be some way to access the cluster for break-the-glass access locally on server nodes, you're welcome to consider the admin kubeconfig that if you like but I am not aware of any plans to eliminate it.

We can take a pass at auditing any controllers using the admin creds, there should not be any but it seems some have perhaps slipped in.

Probably need to extend this thinking to the manifests directory as well. To audit which node a CR yaml was dropped into that folder.

That is already possible, as seen in the audit logs you posted - note the node name in the user agent:
kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08bc78d4-2a10-4b4a-bf56-7003707a28be","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/configmaps/rke2","verb":"update","user":{"username":"system:admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"[email protected]/v1.23.10+rke2r1

The created resources also have the same value in the managedFields info.

[brandond]

Closing this out as we are not going to do away with the default admin user, but I have opened k3s-io/k3s#7212 to track auditing for controllers that use the admin RBAC.