[-]poststarthook/rbac/bootstrap-roles failed: reason withheld #86715

zhangguanzhang · 2019-12-30T04:26:02Z

What happened:
the kube-apiserver logs

I1230 11:42:36.625486   51567 healthz.go:177] healthz check poststarthook/crd-informer-synced failed: not finished
I1230 11:42:36.644253   51567 healthz.go:177] healthz check poststarthook/rbac/bootstrap-roles failed: not finished
I1230 11:42:36.644262   51567 healthz.go:177] healthz check poststarthook/scheduling/bootstrap-system-priority-classes failed: not finished
I1230 11:42:36.644280   51567 healthz.go:177] healthz check poststarthook/ca-registration failed: not finished
I1230 11:42:36.644296   51567 healthz.go:191] [+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[-]poststarthook/crd-informer-synced failed: reason withheld
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld
[-]poststarthook/ca-registration failed: reason withheld
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check failed
I1230 11:42:36.633295  134293 healthz.go:193] [+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/ca-registration ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check failed

but this is ok

$ kubectl get --raw /healthz/poststarthook/rbac/bootstrap-roles
ok

the code
https://github.com/kubernetes/kubernetes/blob/v1.16.4/staging/src/k8s.io/apiserver/pkg/server/healthz/healthz.go#L162-L206

What you expected to happen:
healthz check passed
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):
v1.16.4
Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Network plugin and version (if this is a network-related bug):
Others:

The text was updated successfully, but these errors were encountered:

zhangguanzhang · 2019-12-30T04:27:26Z

/sig api-machinery

zhangguanzhang · 2019-12-30T05:25:35Z

kubeadm config

    apiServer:
      certSANs:
      - 10.96.0.1
      - 127.0.0.1
      - localhost
      - apiserver.k8s.local
      - 172.19.0.2
      - 172.19.0.3
      - 172.19.0.4
      - apiserver01.k8s.local
      - apiserver02.k8s.local
      - apiserver03.k8s.local
      - master
      - kubernetes
      - kubernetes.default
      - kubernetes.default.svc
      - kubernetes.default.svc.cluster.local
      extraArgs:
        authorization-mode: Node,RBAC
        enable-admission-plugins: NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeClaimResize,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,Priority,PodPreset
        runtime-config: api/all,settings.k8s.io/v1alpha1=true
        storage-backend: etcd3
        v: 2
      extraVolumes:
      - hostPath: /etc/localtime
        mountPath: /etc/localtime
        name: localtime
        readOnly: true
      timeoutForControlPlane: 4m0s

liggitt · 2019-12-30T14:41:03Z

It is normal for those checks to fail until they complete their startup operation. After the individual healthz get returns ok, doesn't the overall /healthz return ok as well?

tedyu · 2019-12-30T15:01:20Z

If you want to see the detailed error, you can enable the following log:

                                klog.V(4).Infof("healthz check %v failed: %v", check.Name(), err)
                                fmt.Fprintf(&verboseOut, "[-]%v failed: reason withheld\n", check.Name())

zhangguanzhang · 2019-12-31T01:13:59Z

@liggitt @tedyu I keep tracking the logs in v4 level, but there is no healthz check pass,[-]poststarthook/rbac/bootstrap-roles failed: reason withheld always failed,the reason is not finished

ialidzhikov · 2020-02-03T17:45:36Z

I observe the same behaviour with v1.17.2.
Logs of apiserver:

[-]poststarthook/rbac/bootstrap-roles failed: reason withheld

I0203 17:41:42.064129       1 healthz.go:177] healthz check poststarthook/rbac/bootstrap-roles failed: not finished

But

$ kubectl get --raw /healthz/poststarthook/rbac/bootstrap-roles
ok

ialidzhikov · 2020-02-03T22:35:33Z

I guess the health check is ok.
For the issue was that the kube-apiserver was on version v1.16.4, but the kube-controller-manager was on v1.17.2. The kube-controller-manager was not able to acquire leader election because the kube-apiserver v1.16.x does no apply the required rbac for it. The fix was also to update kueb-apiserver to v1.17.

akokshar · 2020-02-12T11:53:33Z

Have just installed fresh 1.17.2 and see the same issue:

Feb 12 12:45:06 kubernetes kube-apiserver[3171]: [-]poststarthook/rbac/bootstrap-roles failed: reason withheld
...
[root@kubernetes ~]# kube-apiserver --version
Kubernetes v1.17.2
[root@kubernetes ~]# kube-controller-manager --version
Kubernetes v1.17.2

And this is the only check which is failing.

wcollin · 2020-03-11T02:13:44Z

k8s v1.16.7 got the same error

devcui · 2020-03-23T06:29:05Z

i got the same error

Health check failed because other nodes are not configured. Please build a complete cluster and check again

devcui · 2020-03-23T06:30:00Z

and the issues can be closed

zhangguanzhang · 2020-03-23T07:22:42Z

but the log doesn't print succeed

fejta-bot · 2020-06-21T07:31:12Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

zhangguanzhang · 2020-06-21T08:25:40Z

/remove-lifecycle stale

prithviramesh · 2020-07-13T21:29:47Z

We're observing the same behavior with Kubernetes 1.16 and Kubernetes 1.14

I0710 15:26:01.457701       1 healthz.go:191] [+]ping ok
[+]log ok
[+]etcd ok
[+]kms-provider-0 ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/ca-registration ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check failed

prithviramesh · 2020-07-13T21:30:01Z

is there any idea on what is causing this?

hakuna-matatah · 2020-07-23T17:43:58Z

If healthz is succeeding from the box (Master node) -

sh-4.2$ kubectl get --raw /healthz/poststarthook/rbac/bootstrap-roles
ok

sh-4.2$ kubectl get --raw /healthz
ok
sh-4.2$

but you encounter /healthz failures in apiserver logs like this

I0723 08:47:16.694185       1 healthz.go:191] [+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/ca-registration ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok

one possibility is that your cluster might have modified CRB ClusterRoleBinding/system:public-info-viewer to not allow system:unauthenticated calls to Apiserver.

Please check that if thats the cause and if so, modify the CRB to add this

- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated

zhangguanzhang · 2020-07-24T01:28:24Z

If healthz is succeeding from the box (Master node) -

sh-4.2$ kubectl get --raw /healthz/poststarthook/rbac/bootstrap-roles
ok

sh-4.2$ kubectl get --raw /healthz
ok
sh-4.2$

but you encounter /healthz failures in apiserver logs like this

I0723 08:47:16.694185       1 healthz.go:191] [+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/ca-registration ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok

one possibility is that your cluster might have modified CRB ClusterRoleBinding/system:public-info-viewer to not allow system:unauthenticated calls to Apiserver.

Please check that if thats the cause and if so, modify the CRB to add this

- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated

I check the crb is ok, it's not been modified

[root@k8s-node1 kube-apiserver]# grep -5 -m1 'bootstrap-roles failed: reason'  kube-apiserver.INFO
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[-]poststarthook/crd-informer-synced failed: reason withheld
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld
[-]poststarthook/ca-registration failed: reason withheld
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[root@k8s-node1 kube-apiserver]# kubectl get clusterrolebinding system:public-info-viewer -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2020-04-15T14:20:12Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:public-info-viewer
  resourceVersion: "97"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system%3Apublic-info-viewer
  uid: f78c1a71-ebd0-47da-b5a0-f75cb3795232
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:public-info-viewer
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated   
[root@k8s-node1 kube-apiserver]# kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "16",
    "gitVersion": "v1.16.7",
    "gitCommit": "be3d344ed06bff7a4fc60656200a93c74f31f9a4",
    "gitTreeState": "clean",
    "buildDate": "2020-02-11T19:34:02Z",
    "goVersion": "go1.13.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "16",
    "gitVersion": "v1.16.7",
    "gitCommit": "be3d344ed06bff7a4fc60656200a93c74f31f9a4",
    "gitTreeState": "clean",
    "buildDate": "2020-02-11T19:24:46Z",
    "goVersion": "go1.13.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

fejta-bot · 2020-10-22T02:21:28Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

zhangguanzhang · 2020-10-22T03:03:10Z

/remove-lifecycle stale

fejta-bot · 2021-01-20T03:32:40Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2021-02-19T04:18:17Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

george-angel · 2021-02-19T08:50:05Z

/remove-lifecycle rotten

Sokwva · 2021-02-21T03:18:37Z

same error on v1.20.4

Dentrax · 2021-03-29T12:00:30Z

Same error on v1.20.2

$ minikube start --alsologtostderr (VirtualBox Version 6.1.16 r140961 (Qt5.6.3))
macOS 11.0.1

W0329 14:57:43.082477   36365 api_server.go:99] status: https://192.168.99.101:8443/healthz returned error 500:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check failed
| I0329 14:57:43.565964   36365 api_server.go:221] Checking apiserver healthz at https://192.168.99.101:8443/healthz ...
I0329 14:57:43.576080   36365 api_server.go:241] https://192.168.99.101:8443/healthz returned 200:
ok

liggitt · 2021-04-16T17:10:45Z

Without more information, this isn't actionable. It is possible for the startup hook to fail if it takes too long to create the bootstrap roles.

If this is encountered, please provide the output of kubectl get --raw /healthz/poststarthook/rbac/bootstrap-roles as well to get published details about the cause of the failure, and the content of the API server log to get internal details about the failure (the hook logs operations prefixed with storage_rbac.go

/close

k8s-ci-robot · 2021-04-16T17:10:54Z

@liggitt: Closing this issue.

In response to this:

Without more information, this isn't actionable. It is possible for the startup hook to fail if it takes too long to create the bootstrap roles.

If this is encountered, please provide the output of kubectl get --raw /healthz/poststarthook/rbac/bootstrap-roles as well to get published details about the cause of the failure, and the content of the API server log to get internal details about the failure (the hook logs operations prefixed with storage_rbac.go

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

1998729 · 2021-11-12T12:00:15Z

I seem to have solved this problem, the memory and cpu limited by kubelet are too small

zhangguanzhang added the kind/bug Categorizes issue or PR as related to a bug. label Dec 30, 2019

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Dec 30, 2019

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 30, 2019

zhangguanzhang changed the title ~~[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld~~ [-]poststarthook/rbac/bootstrap-roles failed: reason withheld Dec 31, 2019

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 21, 2020

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 21, 2020

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 22, 2020

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 22, 2020

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 20, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 19, 2021

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 19, 2021

Dentrax mentioned this issue Mar 29, 2021

removing minikube home fixed "the server has asked for the client to provide credentials" kubernetes/minikube#10948

Closed

k8s-ci-robot closed this as completed Apr 16, 2021

enj moved this to Closed / Done in SIG Auth Dec 5, 2022

enj added this to SIG Auth Dec 5, 2022

csviri mentioned this issue Jun 20, 2023

Jenvtest server startup error java-operator-sdk/jenvtest#105

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[-]poststarthook/rbac/bootstrap-roles failed: reason withheld #86715

[-]poststarthook/rbac/bootstrap-roles failed: reason withheld #86715

zhangguanzhang commented Dec 30, 2019 •

edited

Loading

zhangguanzhang commented Dec 30, 2019

zhangguanzhang commented Dec 30, 2019

liggitt commented Dec 30, 2019

tedyu commented Dec 30, 2019

zhangguanzhang commented Dec 31, 2019 •

edited

Loading

ialidzhikov commented Feb 3, 2020

ialidzhikov commented Feb 3, 2020

akokshar commented Feb 12, 2020 •

edited

Loading

wcollin commented Mar 11, 2020

devcui commented Mar 23, 2020

devcui commented Mar 23, 2020

zhangguanzhang commented Mar 23, 2020

fejta-bot commented Jun 21, 2020

zhangguanzhang commented Jun 21, 2020

prithviramesh commented Jul 13, 2020

prithviramesh commented Jul 13, 2020

hakuna-matatah commented Jul 23, 2020

zhangguanzhang commented Jul 24, 2020

fejta-bot commented Oct 22, 2020

zhangguanzhang commented Oct 22, 2020

fejta-bot commented Jan 20, 2021

fejta-bot commented Feb 19, 2021

george-angel commented Feb 19, 2021

Sokwva commented Feb 21, 2021

Dentrax commented Mar 29, 2021 •

edited

Loading

liggitt commented Apr 16, 2021

k8s-ci-robot commented Apr 16, 2021

1998729 commented Nov 12, 2021

[-]poststarthook/rbac/bootstrap-roles failed: reason withheld #86715

[-]poststarthook/rbac/bootstrap-roles failed: reason withheld #86715

Comments

zhangguanzhang commented Dec 30, 2019 • edited Loading

zhangguanzhang commented Dec 30, 2019

zhangguanzhang commented Dec 30, 2019

liggitt commented Dec 30, 2019

tedyu commented Dec 30, 2019

zhangguanzhang commented Dec 31, 2019 • edited Loading

ialidzhikov commented Feb 3, 2020

ialidzhikov commented Feb 3, 2020

akokshar commented Feb 12, 2020 • edited Loading

wcollin commented Mar 11, 2020

devcui commented Mar 23, 2020

devcui commented Mar 23, 2020

zhangguanzhang commented Mar 23, 2020

fejta-bot commented Jun 21, 2020

zhangguanzhang commented Jun 21, 2020

prithviramesh commented Jul 13, 2020

prithviramesh commented Jul 13, 2020

hakuna-matatah commented Jul 23, 2020

zhangguanzhang commented Jul 24, 2020

fejta-bot commented Oct 22, 2020

zhangguanzhang commented Oct 22, 2020

fejta-bot commented Jan 20, 2021

fejta-bot commented Feb 19, 2021

george-angel commented Feb 19, 2021

Sokwva commented Feb 21, 2021

Dentrax commented Mar 29, 2021 • edited Loading

liggitt commented Apr 16, 2021

k8s-ci-robot commented Apr 16, 2021

1998729 commented Nov 12, 2021

zhangguanzhang commented Dec 30, 2019 •

edited

Loading

zhangguanzhang commented Dec 31, 2019 •

edited

Loading

akokshar commented Feb 12, 2020 •

edited

Loading

Dentrax commented Mar 29, 2021 •

edited

Loading