kops rolling-update authorization to RBAC, kubelet groups RBAC DENY

[Parrryy]

kops version: 1.7.1
kubernetes: 1.8.0
cloud provider: AWS

kops edit cluster 
set authorizations:
    rbac: {}
kops update cluster --yes 
kops rolling-update cluster cluster.aws.learnium.com --force --yes

I'm expecting the cluster to rolling-update, create the nodes, pods, services, etc. I seem to get the following RBAC DENY errors on my master node in the kube-apiserver logs:

09:26:07.185128 5 rbac.go:116] RBAC DENY: user "kubelet" groups ["system:nodes" "system:authenticated"] cannot "list" resource "services" cluster-wide

I also cannot access with my usual client machine which does get access before the update. I get the following error message.

kubectl get pods 
Unable to connect to the server: net/http: TLS handshake timeout

To recreate: Create cluster with alwaysAllow and then switch to rbac: {}. followed by rolling-update --force --yes

EDIT:
After doing this in order to rolling-update the cluster back to alwaysAllow, I need to add the --cloudonly flag otherwise I get the following error:

error listing nodes in cluster: Get https://api.cluster.aws.learnium.com/api/v1/nodes: dial tcp 52.17.54.9:443: i/o timeout

[KashifSaadat]

Hi @Parrryy, thanks for raising the issue. I've tested a rolling update from alwaysAllow to rbac which appears to be working ok, but using the following versions:

• Kops : Built off master (including protokube & nodeup)
• Kubernetes : v1.8.0

It appears you're hitting the same issue as was raised in #3749. Can you build kops from source and test again?

[liggitt]

looks like a duplicate of #3551

@justinsb does kops support kube 1.8.0 yet, or does there need to be a release cut from master that includes #3683 ?

[chrislovecnm]

Kops is not GA with k8s 1.8 at this point. There is not a release that includes #3683 yet.

A user would need to use master.

[Parrryy]

@KashifSaadat Just to confirm on here, updating from source and my cluster is working as expected. (Given the node authorization issue and current fix)

[chrislovecnm]

@Parrryy so we can close this issue?

[Parrryy]

Yeah I will close the issue.

[naveensrinivasan]

We are running into the same issue.

• We are trying to upgrade the cluster from v1.7.7 to v1.8.6 with RBAC turned on.
• We used the kops master to upgrade kops version Version 1.8.0 (git-4876009bd)

I1227 16:17:34.682684       7 rbac.go:116] RBAC DENY: user "kubelet" groups ["system:nodes" "system:authenticated"] cannot "create" resource "pods" in namespace "kube-system"
I1227 16:17:34.682827       7 wrap.go:42] POST /api/v1/namespaces/kube-system/pods: (352.225µs) 403 [[kubelet/v1.8.6 (linux/amd64) kubernetes/6260bb0] 127.0.0.1:32806]
I1227 16:17:34.683112       7 rbac.go:116] RBAC DENY: user "kubelet" groups ["system:nodes" "system:authenticated"] cannot "create" resource "events" in namespace "default"
I1227 16:17:34.683175       7 wrap.go:42] POST /api/v1/namespaces/default/events: (204.479µs) 403 [[kubelet/v1.8.6 (linux/amd64) kubernetes/6260bb0] 127.0.0.1:32806]
I1227 16:17:34.684278       7 rbac.go:116] RBAC DENY: user "kubelet" groups ["system:nodes" "system:authenticated"] cannot "create" resource "events" in namespace "default"
I1227 16:17:34.684381       7 wrap.go:42] POST /api/v1/namespaces/default/events: (272.221µs) 403 [[kubelet/v1.8.6 (linux/amd64) kubernetes/6260bb0] 127.0.0.1:32806]

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-12-26T20:42:03Z
  name: k8s.playground.REDACTED.io
spec:
  api:
    dns: {}
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://k8s.playground.REDACTED.io/k8s.playground.REDACTED.io
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
    - instanceGroup: master-us-east-1b
      name: b
    - instanceGroup: master-us-east-1c
      name: c
    name: main
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
    - instanceGroup: master-us-east-1b
      name: b
    - instanceGroup: master-us-east-1c
      name: c
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    authorizationRbacSuperUser: admin
    storageBackend: etcd3
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.8.6
  masterInternalName: api.internal.k8s.playground.REDACTED.io
  masterPublicName: api.k8s.playground.REDACTED.io
  networkCIDR: 172.20.0.0/16
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: us-east-1a
    type: Public
    zone: us-east-1a
  - cidr: 172.20.64.0/19
    name: us-east-1b
    type: Public
    zone: us-east-1b
  - cidr: 172.20.96.0/19
    name: us-east-1c
    type: Public
    zone: us-east-1c
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

EDIT

kubectl get  clusterrolebinding system:node -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2017-12-26T20:53:27Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node
  resourceVersion: "850"
  selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/system%3Anode
  uid: d24fbe68-ea7e-11e7-a9e1-0201c744720e
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes

What is the workaround?
cc @chrislovecnm

[liggitt]

Was that an actual upgrade? What authorization modes were used with the 1.7.x deployment?

See https://kubernetes.io/docs/admin/authorization/node/#upgrades-from-previous-versions-using-rbac for a discussion of the kubelet permissions transitioning from being managed by RBAC to being managed by the Node authorizer.

[naveensrinivasan]

@liggitt We upgraded v1.7.7 to RBAC and then now we are trying to move v1.8.6. What should we be doing? Thanks

[chrislovecnm]

/open

[naveensrinivasan]

@chrislovecnm Do you want me to open another ticket?

[chrislovecnm]

@naveensrinivasan yes please

[naveensrinivasan]

@chrislovecnm is there a work around for now?

Thanks

[naveensrinivasan]

#4163