Skip to content

IdP on k8s

Jump to bottom
Cesar Celis Hernandez edited this page Jun 10, 2024 · 20 revisions

Objective:

To use IdP and RBAC to grant access to k8s resources

Main links:

Secondary Links:

Steps:

RU87c899ReDjiOcuuoBOJc_9kP0NQ5EHmiRDNcvOcIFir
  • In PostMan on id token obtainer POST your code:
{"client_id":"rMVc40T7fwgbEez1svp8wmjBtSaoKIOJ","client_secret":"SlQcQAUdUjW8ZPbp5qdbQYM5P7Pkp4GtGeXKky_dThl8Uk2NWdGu13dO9ftN0umH","grant_type":"authorization_code","code":"RU87c899ReDjiOcuuoBOJc_9kP0NQ5EHmiRDNcvOcIFir","redirect_uri":"http://localhost:5005/oauth_callback"}
  • Obtain id_token:
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IklESmhKWGJSU2lzM2lqc19FQXJYOCJ9.eyJodHRwczovL21pbi5pby9wb2xpY3kiOiJjb25zb2xlQWRtaW4iLCJncm91cCI6Ims4cy1hZG1pbnMiLCJpc3MiOiJodHRwczovL2Rldi14cW01aW9xbG15N3F5anZsLnVzLmF1dGgwLmNvbS8iLCJhdWQiOiJyTVZjNDBUN2Z3Z2JFZXoxc3ZwOHdtakJ0U2FvS0lPSiIsImlhdCI6MTcxNzg2OTAzNywiZXhwIjoxNzE4NzMzMDM3LCJzdWIiOiJhdXRoMHw2NjVmNzg1Y2FkZWFhMmZiNjU1MjhlZjMiLCJzaWQiOiJiLVU2WE1ETWF4WFJucER2amVvVEJ1TnlLQWxpbHg4ciJ9.X98qFhSQZ4vRVGSMyVJ2nuYAJQD7Jqwus209P3W7eosY99qQbx650QDaWKbwnUnhGt5ddyu223u2eiAlanP3iiw2K7zQCxXVuuBZrUj3pbo41JluF79-83N8SqMro-0dW5AKHQ3ww_90tzpoBReuKSujA9i28t58JksLHmgUgN07sD78BtWkSc41vT-E4YNQ_Rt5azfQ_0mlD517mIqAwSEB9y0nNKxKvw8FTzXKt9BHgGikeT7N2fhoXLrJNJ2QzR2cUaNxF-1U2na_RxclF28-R0FsrmlxWY43PzKjlUxS7tJzP5ZgzVVwI7liQEQsrNzikjCHnesN77837-lrdg
  • Open Docker Desktop App:
Screenshot 2024-06-10 at 3 32 13 PM
  • create kind cluster from bash script: createcluster 
$ createcluster
 
####################
kind delete cluster:
####################
Deleting cluster "kind" ...
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.29.2) 🖼 
 ✓ Preparing nodes 📦 📦 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊
  • Apply RBAC:
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: restricted-user
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods", "services"]
  verbs: ["get", "watch", "list"]
EOF
kubectl apply -f - <<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-cluster-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: restricted-user
subjects:
- kind: Group
  name: k8s-restricted-users
EOF

  • With Lens, edit API Server Flags

    • Open Node control plane terminal
    • apt update
    • apt upgrade
    • apt install -y vim
    • root@kind-control-plane:/# vi /etc/kubernetes/manifests/kube-apiserver.yaml
--oidc-issuer-url=https://dev-xqm5ioqlmy7qyjvl.us.auth0.com/
--oidc-client-id=rMVc40T7fwgbEez1svp8wmjBtSaoKIOJ
--oidc-groups-claim=group

Let 5 minutes for the system to re-start with new flags.

  • Remove access to the admin user:
cd ~/.kube
subl config
  • from users, remove - name: kind-kind:
Screenshot 2024-06-10 at 4 12 54 PM
  • Test:
k get services --namespace=default --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IklESmhKWGJSU2lzM2lqc19FQXJYOCJ9.eyJodHRwczovL21pbi5pby9wb2xpY3kiOiJjb25zb2xlQWRtaW4iLCJncm91cCI6Ims4cy1yZXN0cmljdGVkLXVzZXJzIiwiaXNzIjoiaHR0cHM6Ly9kZXYteHFtNWlvcWxteTdxeWp2bC51cy5hdXRoMC5jb20vIiwiYXVkIjoick1WYzQwVDdmd2diRWV6MXN2cDh3bWpCdFNhb0tJT0oiLCJpYXQiOjE3MTgwNDcxNzYsImV4cCI6MTcxODkxMTE3Niwic3ViIjoiYXV0aDB8NjY1Zjc4NWNhZGVhYTJmYjY1NTI4ZWYzIiwic2lkIjoiWkpFbTdtY0RhcE1mUlN4TE1VRG93T2xmbGZ4WUhLQXYifQ.q3Sln-ejkN0NDw6rkMby9UWVzNlLnBsp9q0Y250TMYarZzhTllK_BHHSTelKg-1QsEFI00Sn7xMUu5tDOktdxwV8FZzU1JFr8C6mm-ss07DEYKeMFrBo8wqIhdp1UgjQefZ9J2Sc0ze6oij7HDp_iKkO2E443JzeoJWyZqeAvNBUdH_p8xmvX5XIO7yD0sM5AnaVtleYG2Jv-Tb7NtJmdqQtZe5t52_j_6LBJn9e5j-UXuT0JRykbSFkcusH3PPwuzaysaoDPrHyTR6kkWr_hv3pKFRefJYdvc4bVpnfnt5MDvCUK0S1BkNEOhaR7lpSA8BGFBCMzGdj5yfKM9IllA

Result:

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   47m

Other testing performed:

$ k get pods -A
Please enter Username: ^C
Cesars-MacBook-Pro:~ cniackz$ 
Cesars-MacBook-Pro:~ cniackz$ 
Cesars-MacBook-Pro:~ cniackz$ 
Cesars-MacBook-Pro:~ cniackz$ k get pods -A --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IklESmhKWGJSU2lzM2lqc19FQXJYOCJ9.eyJodHRwczovL21pbi5pby9wb2xpY3kiOiJjb25zb2xlQWRtaW4iLCJncm91cCI6Ims4cy1hZG1pbnMiLCJpc3MiOiJodHRwczovL2Rldi14cW01aW9xbG15N3F5anZsLnVzLmF1dGgwLmNvbS8iLCJhdWQiOiJyTVZjNDBUN2Z3Z2JFZXoxc3ZwOHdtakJ0U2FvS0lPSiIsImlhdCI6MTcxNzg2OTAzNywiZXhwIjoxNzE4NzMzMDM3LCJzdWIiOiJhdXRoMHw2NjVmNzg1Y2FkZWFhMmZiNjU1MjhlZjMiLCJzaWQiOiJiLVU2WE1ETWF4WFJucER2amVvVEJ1TnlLQWxpbHg4ciJ9.X98qFhSQZ4vRVGSMyVJ2nuYAJQD7Jqwus209P3W7eosY99qQbx650QDaWKbwnUnhGt5ddyu223u2eiAlanP3iiw2K7zQCxXVuuBZrUj3pbo41JluF79-83N8SqMro-0dW5AKHQ3ww_90tzpoBReuKSujA9i28t58JksLHmgUgN07sD78BtWkSc41vT-E4YNQ_Rt5azfQ_0mlD517mIqAwSEB9y0nNKxKvw8FTzXKt9BHgGikeT7N2fhoXLrJNJ2QzR2cUaNxF-1U2na_RxclF28-R0FsrmlxWY43PzKjlUxS7tJzP5ZgzVVwI7liQEQsrNzikjCHnesN77837-lrdg
NAMESPACE            NAME                                              READY   STATUS      RESTARTS      AGE
kube-system          coredns-76f75df574-27fxg                          1/1     Running     0             45h
kube-system          coredns-76f75df574-fpq8p                          1/1     Running     0             45h
kube-system          etcd-kind-control-plane                           1/1     Running     0             45h
kube-system          kindnet-2h2k9                                     1/1     Running     6 (23h ago)   45h
kube-system          kindnet-88hqc                                     1/1     Running     5 (23h ago)   45h
kube-system          kindnet-q58rb                                     1/1     Running     5 (23h ago)   45h
kube-system          kindnet-rgb8l                                     1/1     Running     5 (23h ago)   45h
kube-system          kindnet-zds6n                                     1/1     Running     6 (23h ago)   45h
kube-system          kube-apiserver-kind-control-plane                 1/1     Running     0             23h
kube-system          kube-controller-manager-kind-control-plane        1/1     Running     3 (23h ago)   45h
kube-system          kube-proxy-7rq88                                  1/1     Running     0             45h
kube-system          kube-proxy-b49pd                                  1/1     Running     0             45h
kube-system          kube-proxy-k7zj6                                  1/1     Running     0             45h
kube-system          kube-proxy-ktglp                                  1/1     Running     0             45h
kube-system          kube-proxy-tv2zr                                  1/1     Running     0             45h
kube-system          kube-scheduler-kind-control-plane                 1/1     Running     3 (23h ago)   45h
kube-system          node-shell-0052a0db-3440-46bc-8dda-eec5f41db4bb   0/1     Completed   0             45h
kube-system          node-shell-50c35fad-e235-44cb-a8d6-11573f255222   1/1     Running     0             169m
kube-system          node-shell-c39837bc-c7e1-479c-89ee-709bc3de032a   0/1     Completed   0             42h
kube-system          node-shell-cb54ecc5-19d2-4236-abbb-731db0daca4b   1/1     Running     0             23h
local-path-storage   local-path-provisioner-7577fdbbfb-g27sh           1/1     Running     0             45h

RBAC Used:

https://developer.okta.com/blog/2021/11/08/k8s-api-server-oidc#configure-rbac

  • Full Access:
kubectl apply -f - <<EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: Group
  name: k8s-admins
EOF
  • Restricted Access:
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: restricted-user
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods", "services"]
  verbs: ["get", "watch", "list"]
EOF
kubectl apply -f - <<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-cluster-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: restricted-user
subjects:
- kind: Group
  name: k8s-restricted-users
EOF

JS Code in Auth0 for actions:

/**
* Handler that will be called during the execution of a PostLogin flow.
*
* @param {Event} event - Details about the user and the context in which they are logging in.
* @param {PostLoginAPI} api - Interface whose methods can be used to change the behavior of the login.
*/
exports.onExecutePostLogin = async (event, api) => {
  if(event.authorization){
    // k8s-restricted-users k8s role that can only view pods and services in the default namespace.
    api.idToken.setCustomClaim("group","k8s-restricted-users");
    //api.idToken.setCustomClaim("group","k8s-admins");
    //api.idToken.setCustomClaim("group","something");
    // k8s-admins is attached via RBAC giving permissions to this user to use k8s
    // where k8s-admins works and something doesn't
    // Error from server (Forbidden): pods is forbidden: User "https://dev-xqm5ioqlmy7qyjvl.us.auth0.com/#auth0|665f785cadeaa2fb65528ef3" cannot list resource "pods" in API group "" at the cluster scope
  }
};


/**
* Handler that will be invoked when this action is resuming after an external redirect. If your
* onExecutePostLogin function does not perform a redirect, this function can be safely ignored.
*
* @param {Event} event - Details about the user and the context in which they are logging in.
* @param {PostLoginAPI} api - Interface whose methods can be used to change the behavior of the login.
*/
// exports.onContinuePostLogin = async (event, api) => {
// };

Restricted access example:

Cesars-MacBook-Pro:.kube cniackz$ k get services --namespace=default --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IklESmhKWGJSU2lzM2lqc19FQXJYOCJ9.eyJodHRwczovL21pbi5pby9wb2xpY3kiOiJjb25zb2xlQWRtaW4iLCJncm91cCI6Ims4cy1yZXN0cmljdGVkLXVzZXJzIiwiaXNzIjoiaHR0cHM6Ly9kZXYteHFtNWlvcWxteTdxeWp2bC51cy5hdXRoMC5jb20vIiwiYXVkIjoick1WYzQwVDdmd2diRWV6MXN2cDh3bWpCdFNhb0tJT0oiLCJpYXQiOjE3MTc4Njk3MzUsImV4cCI6MTcxODczMzczNSwic3ViIjoiYXV0aDB8NjY1Zjc4NWNhZGVhYTJmYjY1NTI4ZWYzIiwic2lkIjoiLTk5R1lUY3VsRHBEZF96TzZ5VGwtVnp5RFltVURQRnIifQ.sMs5-GfQfsRyuetXGnfawR1SCLOrMdVWhs-1Stb22kmAcckOdOmZeF_HfflAdWAU0KZc-hECCwC4p2s9jDTwTWpxSbB5Vt-JHcOZv_WGXWHgjw68TcFf7b6tDNwJU0TIGT1acpKuXG8cA9gcMsSuyfLco1BjNKsygc1kMgEnF6SCdK_EMiojySJ94rOdKuo7Kxs_a7PRlernFiijizIuW5C977jzcBIY0-gMfm28FRqz32ohXq3NlvnOneR2CL4nV3kk_T_GlWdhmy1KwR-NRqmSryM4Cl31UC5wTR_0-ym1UQStuAOWPyO7pkVlpRh_j5Zr9hL3c1Lrz1JTFidP0g
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   45h
Cesars-MacBook-Pro:.kube cniackz$ 
Cesars-MacBook-Pro:.kube cniackz$ 
Cesars-MacBook-Pro:.kube cniackz$ 
Cesars-MacBook-Pro:.kube cniackz$ 
Cesars-MacBook-Pro:.kube cniackz$ 
Cesars-MacBook-Pro:.kube cniackz$ 
Cesars-MacBook-Pro:.kube cniackz$ k get services --namespace=kube-system --token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IklESmhKWGJSU2lzM2lqc19FQXJYOCJ9.eyJodHRwczovL21pbi5pby9wb2xpY3kiOiJjb25zb2xlQWRtaW4iLCJncm91cCI6Ims4cy1yZXN0cmljdGVkLXVzZXJzIiwiaXNzIjoiaHR0cHM6Ly9kZXYteHFtNWlvcWxteTdxeWp2bC51cy5hdXRoMC5jb20vIiwiYXVkIjoick1WYzQwVDdmd2diRWV6MXN2cDh3bWpCdFNhb0tJT0oiLCJpYXQiOjE3MTc4Njk3MzUsImV4cCI6MTcxODczMzczNSwic3ViIjoiYXV0aDB8NjY1Zjc4NWNhZGVhYTJmYjY1NTI4ZWYzIiwic2lkIjoiLTk5R1lUY3VsRHBEZF96TzZ5VGwtVnp5RFltVURQRnIifQ.sMs5-GfQfsRyuetXGnfawR1SCLOrMdVWhs-1Stb22kmAcckOdOmZeF_HfflAdWAU0KZc-hECCwC4p2s9jDTwTWpxSbB5Vt-JHcOZv_WGXWHgjw68TcFf7b6tDNwJU0TIGT1acpKuXG8cA9gcMsSuyfLco1BjNKsygc1kMgEnF6SCdK_EMiojySJ94rOdKuo7Kxs_a7PRlernFiijizIuW5C977jzcBIY0-gMfm28FRqz32ohXq3NlvnOneR2CL4nV3kk_T_GlWdhmy1KwR-NRqmSryM4Cl31UC5wTR_0-ym1UQStuAOWPyO7pkVlpRh_j5Zr9hL3c1Lrz1JTFidP0g
Error from server (Forbidden): services is forbidden: User "https://dev-xqm5ioqlmy7qyjvl.us.auth0.com/#auth0|665f785cadeaa2fb65528ef3" cannot list resource "services" in API group "" in the namespace "kube-system"
Clone this wiki locally