libvirt: apiserver, controller-manager, package server pods shows CrashLoopBackOff #484
Comments
|
I did another try after destroying the cluster and waiting till all the nodes are up now only 2 pods are in CrashLoopBackOff stage. # kubectl get nodes
NAME STATUS ROLES AGE VERSION
test1-bootstrap Ready bootstrap 25m v1.11.0+d4cacc0
test1-master-0 Ready master 25m v1.11.0+d4cacc0
test1-worker-0-msddq Ready worker 13m v1.11.0+d4cacc0
# kubectl get --all-namespaces pods
NAMESPACE NAME READY STATUS RESTARTS AGE
default registry-6fcb8b7789-4x72w 1/1 Running 0 5m
kube-system kube-apiserver-krqjc 1/1 Running 0 25m
kube-system kube-controller-manager-b6985978d-zxhm2 1/1 Running 0 25m
kube-system kube-core-operator-7f4d6b8dcf-hdfgw 1/1 Running 0 17m
kube-system kube-dns-787c975867-kz8np 3/3 Running 0 25m
kube-system kube-flannel-h2kqm 2/2 Running 5 14m
kube-system kube-flannel-lt5dd 2/2 Running 0 20m
kube-system kube-flannel-t4l49 2/2 Running 0 20m
kube-system kube-proxy-5xz66 1/1 Running 0 14m
kube-system kube-proxy-kvgxr 1/1 Running 0 25m
kube-system kube-proxy-nbqvd 1/1 Running 0 25m
kube-system kube-scheduler-78d86f9754-j4pr8 1/1 Running 0 25m
kube-system metrics-server-5767bfc576-67znk 2/2 Running 0 15m
kube-system pod-checkpointer-7v2xn 1/1 Running 0 25m
kube-system pod-checkpointer-7v2xn-test1-master-0 1/1 Running 0 24m
kube-system tectonic-network-operator-76qt7 1/1 Running 0 25m
openshift-apiserver apiserver-kgcgc 1/1 Running 0 16m
openshift-cluster-api clusterapi-apiserver-6b855f7bc5-s2flm 2/2 Running 0 18m
openshift-cluster-api clusterapi-controllers-85f6bfd9d5-vrw2s 2/2 Running 0 16m
openshift-cluster-api machine-api-operator-5d85454676-m9z9d 1/1 Running 0 23m
openshift-cluster-version cluster-version-operator-fqkl6 1/1 Running 0 25m
openshift-controller-manager controller-manager-w9kdb 1/1 Running 0 15m
openshift-core-operators openshift-cluster-openshift-apiserver-operator-5fbd49d8f7-vlthq 1/1 Running 0 23m
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-7cw26h6 1/1 Running 0 23m
openshift-core-operators openshift-service-cert-signer-operator-6d6c6f55db-jspwh 1/1 Running 0 24m
openshift-image-registry cluster-image-registry-operator-58c7c9bfd6-r7l92 1/1 Running 0 14m
openshift-ingress tectonic-ingress-controller-operator-fcb9c6f4b-tlhzq 0/1 CrashLoopBackOff 7 16m
openshift-machine-config-operator machine-config-controller-6948b45dd9-n8cxd 1/1 Running 0 17m
openshift-machine-config-operator machine-config-daemon-drsmd 1/1 Running 4 14m
openshift-machine-config-operator machine-config-daemon-fsxft 1/1 Running 0 16m
openshift-machine-config-operator machine-config-operator-545fcb447d-h9pfj 1/1 Running 0 23m
openshift-machine-config-operator machine-config-server-5dwbh 1/1 Running 0 16m
openshift-monitoring cluster-monitoring-operator-c64f5b475-tlplc 1/1 Running 0 14m
openshift-monitoring prometheus-operator-5bf8644c75-xmrcm 1/1 Running 0 5m
openshift-operator-lifecycle-manager catalog-operator-5d5d8c7689-wf2ql 1/1 Running 0 23m
openshift-operator-lifecycle-manager olm-operator-76b7f57649-8zj5v 1/1 Running 0 23m
openshift-operator-lifecycle-manager package-server-f994b8699-f7hsc 0/1 CrashLoopBackOff 5 23m
openshift-service-cert-signer apiservice-cabundle-injector-d4c746869-z66ks 1/1 Running 0 17m
openshift-service-cert-signer configmap-cabundle-injector-77bd46b-bqnd8 1/1 Running 0 17m
openshift-service-cert-signer service-serving-cert-signer-55fb7cc589-l6rzs 1/1 Running 0 17m
openshift-web-console webconsole-86f4f55644-cl9r5 1/1 Running 0 5m
tectonic-system kube-addon-operator-784b4b6c7-fdkqj 1/1 Running 0 17m |
This may be new. Can you get details on the error from the pod's logs?
This may still be operator-framework/operator-lifecycle-manager#524, but without pod logs it's hard to say. |
|
@wking I will again rebase my setup to master and try it on monday to provide the logs if still occur. |
|
@wking Today after pulling up latest master I am seeing # oc get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default registry-66c8b7b9bb-wtlnx 1/1 Running 0 21m
kube-system kube-apiserver-qp96m 1/1 Running 0 34m
kube-system kube-controller-manager-f2w5w 1/1 Running 0 34m
kube-system kube-dns-787c975867-bnkb6 3/3 Running 0 34m
kube-system kube-flannel-j58tx 2/2 Running 0 29m
kube-system kube-flannel-wg96p 2/2 Running 1 21m
kube-system kube-proxy-vftmb 1/1 Running 0 21m
kube-system kube-proxy-x2bqx 1/1 Running 0 34m
kube-system kube-scheduler-78sqm 1/1 Running 0 34m
kube-system metrics-server-5767bfc576-l6bhk 2/2 Running 0 22m
kube-system pod-checkpointer-wl4pl 1/1 Running 0 34m
kube-system pod-checkpointer-wl4pl-test1-master-0 1/1 Running 0 33m
kube-system tectonic-network-operator-b4qhb 1/1 Running 0 34m
openshift-apiserver apiserver-74ldh 1/1 Running 0 26m
openshift-cluster-api clusterapi-apiserver-6b855f7bc5-b8gmh 2/2 Running 0 26m
openshift-cluster-api clusterapi-controllers-6cc744d9bb-fhnk8 2/2 Running 0 25m
openshift-cluster-api machine-api-operator-79948cdf8-nsb54 1/1 Running 0 32m
openshift-cluster-version cluster-version-operator-vscpx 1/1 Running 0 34m
openshift-controller-manager controller-manager-v5qxs 1/1 Running 0 27m
openshift-core-operators openshift-cluster-kube-apiserver-operator-6dd9878746-dmfjs 1/1 Running 0 32m
openshift-core-operators openshift-cluster-kube-controller-manager-operator-c977687xcpp7 1/1 Running 0 32m
openshift-core-operators openshift-cluster-kube-scheduler-operator-86b94ddbc9-2x5vh 1/1 Running 0 32m
openshift-core-operators openshift-cluster-openshift-apiserver-operator-77c4ddb555-4dwb8 1/1 Running 0 32m
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-6c4xw7m 1/1 Running 0 32m
openshift-core-operators openshift-service-cert-signer-operator-5959bd9c86-nh69h 1/1 Running 0 32m
openshift-image-registry cluster-image-registry-operator-679d6cfb55-xm7mb 1/1 Running 0 24m
openshift-ingress default-http-backend-6985d557bb-tpzn5 1/1 Running 0 22m
openshift-ingress router-b5bd684d-dm9gx 1/1 Running 0 22m
openshift-ingress tectonic-ingress-controller-operator-fcb9c6f4b-rkgj2 1/1 Running 0 24m
openshift-kube-apiserver apiserver-75ddf46bd9-gkz2q 1/1 Running 0 26m
openshift-kube-controller-manager controller-manager-69c959cb5-rmm9f 1/1 Running 0 26m
openshift-kube-scheduler scheduler-5df7b4d4df-4rphm 1/1 Running 0 26m
openshift-machine-config-operator machine-config-controller-f8b479fc6-zmg75 1/1 Running 0 25m
openshift-machine-config-operator machine-config-daemon-c6dh4 1/1 Running 0 24m
openshift-machine-config-operator machine-config-daemon-ppz9d 1/1 Running 1 21m
openshift-machine-config-operator machine-config-operator-7dcb5b4746-8ptf2 1/1 Running 0 32m
openshift-machine-config-operator machine-config-server-2nnp6 1/1 Running 0 25m
openshift-monitoring cluster-monitoring-operator-7c6598dc88-thkwq 1/1 Running 0 24m
openshift-monitoring grafana-57f595895d-xs62x 2/2 Running 0 16m
openshift-monitoring prometheus-k8s-0 3/4 CrashLoopBackOff 7 14m
openshift-monitoring prometheus-k8s-1 3/4 CrashLoopBackOff 6 7m
openshift-monitoring prometheus-operator-5bf8644c75-m6tvc 1/1 Running 0 17m
openshift-operator-lifecycle-manager catalog-operator-b6c5c6dff-d4gcd 1/1 Running 0 32m
openshift-operator-lifecycle-manager olm-operator-865b4d6f65-86fqq 1/1 Running 0 32m
openshift-operator-lifecycle-manager package-server-5b9cb6948c-8z4hk 1/1 Running 0 32m
openshift-service-cert-signer apiservice-cabundle-injector-79695dcbf-w5rvk 1/1 Running 0 26m
openshift-service-cert-signer configmap-cabundle-injector-754df4fd98-b98ck 1/1 Running 0 26m
openshift-service-cert-signer service-serving-cert-signer-5d7d4fbd49-t9pcd 1/1 Running 0 26m
openshift-web-console webconsole-86f4f55644-cjkrp 1/1 Running 0 22m
tectonic-system kube-addon-operator-784b4b6c7-5j7rp 1/1 Running 0 26m
[root@test1-bootstrap core]# oc logs prometheus-k8s-1 -n openshift-monitoring
Error from server (BadRequest): a container name must be specified for pod prometheus-k8s-1, choose one of: [prometheus prometheus-config-reloader prometheus-proxy rules-configmap-reloader]
[root@test1-bootstrap core]# oc logs prometheus-k8s-1 -c prometheus-proxy -n openshift-monitoring
2018/10/23 06:03:46 provider.go:98: Defaulting client-id to system:serviceaccount:openshift-monitoring:prometheus-k8s
2018/10/23 06:03:46 provider.go:103: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2018/10/23 06:03:46 provider.go:526: Performing OAuth discovery against https://10.3.0.1/.well-known/oauth-authorization-server
2018/10/23 06:03:46 provider.go:572: 200 GET https://10.3.0.1/.well-known/oauth-authorization-server {
"issuer": "",
"authorization_endpoint": "/oauth/authorize",
"token_endpoint": "/oauth/token",
"scopes_supported": [
"user:check-access",
"user:full",
"user:info",
"user:list-projects",
"user:list-scoped-projects"
],
"response_types_supported": [
"code",
"token"
],
"grant_types_supported": [
"authorization_code",
"implicit"
],
"code_challenge_methods_supported": [
"plain",
"S256"
]
}
2018/10/23 06:03:46 provider.go:300: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2018/10/23 06:03:46 oauthproxy.go:201: mapping path "/" => upstream "http://localhost:9090/"
2018/10/23 06:03:46 oauthproxy.go:222: compiled skip-auth-regex => "^/metrics"
2018/10/23 06:03:46 oauthproxy.go:228: OAuthProxy configured for Client ID: system:serviceaccount:openshift-monitoring:prometheus-k8s
2018/10/23 06:03:46 oauthproxy.go:238: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> refresh:disabled
2018/10/23 06:03:46 main.go:150: using htpasswd file /etc/proxy/htpasswd/auth
2018/10/23 06:03:46 http.go:96: HTTPS: listening on [::]:9091
[root@test1-bootstrap core]# oc logs prometheus-k8s-1 -c rules-configmap-reloader -n openshift-monitoring
[root@test1-bootstrap core]# oc logs prometheus-k8s-1 -c prometheus -n openshift-monitoring
level=info ts=2018-10-23T06:14:39.878123439Z caller=main.go:238 msg="Starting Prometheus" version="(version=2.4.2, branch=HEAD, revision=c305ffaa092e94e9d2dbbddf8226c4813b1190a0)"
level=info ts=2018-10-23T06:14:39.881033034Z caller=main.go:239 build_context="(go=go1.10.3, user=root@prometheus-binary-7-build, date=20181002-12:31:36)"
level=info ts=2018-10-23T06:14:39.881146084Z caller=main.go:240 host_details="(Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 prometheus-k8s-1 (none))"
level=info ts=2018-10-23T06:14:39.881227377Z caller=main.go:241 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-10-23T06:14:39.881295755Z caller=main.go:242 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2018-10-23T06:14:39.884319568Z caller=main.go:554 msg="Starting TSDB ..."
level=info ts=2018-10-23T06:14:39.884410934Z caller=web.go:397 component=web msg="Start listening for connections" address=127.0.0.1:9090
level=info ts=2018-10-23T06:14:39.896942298Z caller=main.go:564 msg="TSDB started"
level=info ts=2018-10-23T06:14:39.897006958Z caller=main.go:624 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=error ts=2018-10-23T06:14:39.899689905Z caller=main.go:643 msg="Failed to apply configuration" err="unable to use specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt: no such file or directory"
level=info ts=2018-10-23T06:14:39.900074887Z caller=kubernetes.go:187 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-10-23T06:14:39.90227583Z caller=kubernetes.go:187 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-10-23T06:14:39.905001056Z caller=kubernetes.go:187 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-10-23T06:14:39.907965629Z caller=kubernetes.go:187 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-10-23T06:14:39.956327797Z caller=main.go:423 msg="Stopping scrape discovery manager..."
level=info ts=2018-10-23T06:14:39.956739876Z caller=main.go:437 msg="Stopping notify discovery manager..."
level=info ts=2018-10-23T06:14:39.956872512Z caller=main.go:459 msg="Stopping scrape manager..."
level=info ts=2018-10-23T06:14:39.956968977Z caller=manager.go:638 component="rule manager" msg="Stopping rule manager..."
level=info ts=2018-10-23T06:14:39.959579908Z caller=manager.go:644 component="rule manager" msg="Rule manager stopped"
level=info ts=2018-10-23T06:14:39.956770339Z caller=main.go:419 msg="Scrape discovery manager stopped"
level=error ts=2018-10-23T06:14:39.957368079Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2018-10-23T06:14:39.957407807Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2018-10-23T06:14:39.957515268Z caller=endpoints.go:130 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=info ts=2018-10-23T06:14:39.957552286Z caller=main.go:433 msg="Notify discovery manager stopped"
level=error ts=2018-10-23T06:14:39.957611062Z caller=endpoints.go:130 component="discovery manager notify" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=info ts=2018-10-23T06:14:39.95763689Z caller=main.go:453 msg="Scrape manager stopped"
level=info ts=2018-10-23T06:14:39.962747958Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."
level=info ts=2018-10-23T06:14:39.962922428Z caller=main.go:608 msg="Notifier manager stopped"
level=error ts=2018-10-23T06:14:39.965707931Z caller=main.go:617 err="error loading config from \"/etc/prometheus/config_out/prometheus.env.yaml\": one or more errors occurred while applying the new configuration (--config.file=\"/etc/prometheus/config_out/prometheus.env.yaml\")"
[root@test1-bootstrap core]# oc logs prometheus-k8s-1 -c prometheus-config-reloader -n openshift-monitoring
[...]
level=error ts=2018-10-23T06:17:34.064320481Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:17:38.964201123Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:17:44.164029301Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:17:49.064394938Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:17:54.063771972Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:17:59.16391977Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:18:03.963718407Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:18:08.966018418Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused"
level=error ts=2018-10-23T06:18:14.263761354Z caller=runutil.go:43 msg="function failed. Retrying" err="trigger reload: reload request failed: Post http://localhost:9090/-/reload: dial tcp 127.0.0.1:9090: connect: connection refused" |
|
After today rerun on the master below is the logs from # kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default registry-6979444448-2qm2c 1/1 Running 0 46m
kube-system kube-apiserver-bdgtd 1/1 Running 0 54m
kube-system kube-controller-manager-2h2pm 1/1 Running 0 54m
kube-system kube-dns-787c975867-7qjpj 3/3 Running 0 54m
kube-system kube-flannel-d2xwd 2/2 Running 0 51m
kube-system kube-flannel-dvfmf 2/2 Running 4 45m
kube-system kube-proxy-4jm8g 1/1 Running 0 54m
kube-system kube-proxy-ds6q9 1/1 Running 0 45m
kube-system kube-scheduler-nx4tq 1/1 Running 0 54m
kube-system metrics-server-5767bfc576-vn5dw 2/2 Running 0 47m
kube-system pod-checkpointer-pbd4f 1/1 Running 0 54m
kube-system pod-checkpointer-pbd4f-test1-master-0 1/1 Running 0 54m
kube-system tectonic-network-operator-cdgrk 1/1 Running 0 54m
openshift-apiserver apiserver-xc2v2 1/1 Running 1 49m
openshift-cluster-api clusterapi-manager-controllers-6898dcd5d9-qmb5m 2/2 Running 0 49m
openshift-cluster-api machine-api-operator-6876cb8c7c-jpn2r 1/1 Running 0 51m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-798dc7b787-d2glh 1/1 Running 0 46m
openshift-cluster-node-tuning tuned-l9mbp 1/1 Running 0 37m
openshift-cluster-samples-operator cluster-samples-operator-d789dbd9b-66z6g 1/1 Running 0 46m
openshift-cluster-version cluster-version-operator-27zqx 1/1 Running 0 54m
openshift-console console-operator-6c879b4655-r47qt 1/1 Running 0 46m
openshift-console openshift-console-67b8f48b9d-zxbmv 1/1 Running 0 37m
openshift-controller-manager controller-manager-gw2lt 1/1 Running 0 49m
openshift-core-operators openshift-cluster-kube-apiserver-operator-77867dd47f-5j4rd 1/1 Running 0 51m
openshift-core-operators openshift-cluster-kube-controller-manager-operator-7bbf776lt7ng 1/1 Running 0 51m
openshift-core-operators openshift-cluster-kube-scheduler-operator-7bb466d6c7-g5vzw 1/1 Running 0 51m
openshift-core-operators openshift-cluster-openshift-apiserver-operator-7598b98cbc-gzbrn 1/1 Running 0 51m
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-598w7k2 1/1 Running 0 51m
openshift-core-operators openshift-service-cert-signer-operator-69995ffb-fsg8m 1/1 Running 0 51m
openshift-csi-operator csi-operator-6c65999974-tl7fl 1/1 Running 0 47m
openshift-image-registry cluster-image-registry-operator-675b9b9d56-nqbsj 1/1 Running 0 46m
openshift-ingress tectonic-ingress-controller-operator-fcb9c6f4b-54vsb 0/1 CrashLoopBackOff 12 48m
openshift-kube-apiserver apiserver-5449bbf5c4-c2bkl 1/1 Running 2 49m
openshift-kube-controller-manager controller-manager-79b6c98b79-nztkx 1/1 Running 0 49m
openshift-kube-scheduler scheduler-7d4d8c4fff-9c9hl 1/1 Running 0 50m
openshift-machine-config-operator machine-config-controller-cff76dc7d-nnjbb 1/1 Running 0 49m
openshift-machine-config-operator machine-config-daemon-d6hng 1/1 Running 3 45m
openshift-machine-config-operator machine-config-daemon-thm6k 1/1 Running 0 47m
openshift-machine-config-operator machine-config-operator-65987dd664-rthjk 1/1 Running 0 51m
openshift-machine-config-operator machine-config-server-zpl67 1/1 Running 0 48m
openshift-monitoring cluster-monitoring-operator-c5cd98654-kl8q2 1/1 Running 0 46m
openshift-monitoring prometheus-operator-5bf8644c75-c9q85 1/1 Running 0 37m
openshift-operator-lifecycle-manager catalog-operator-5c4889c5fc-zfsm5 1/1 Running 0 51m
openshift-operator-lifecycle-manager olm-operator-657b94fc4c-ptqs2 1/1 Running 0 51m
openshift-operator-lifecycle-manager package-server-85fd7c95c-p9ztm 1/1 Running 0 51m
openshift-service-cert-signer apiservice-cabundle-injector-cdc9699d7-qsddt 1/1 Running 0 49m
openshift-service-cert-signer configmap-cabundle-injector-586b98ddb8-pb2w2 1/1 Running 0 49m
openshift-service-cert-signer service-serving-cert-signer-578f88cccf-jdq56 1/1 Running 0 49m
openshift-web-console webconsole-86f4f55644-vrf5b 1/1 Running 0 47m
tectonic-system kube-addon-operator-784b4b6c7-msrjh 1/1 Running 0 48m
# oc logs tectonic-ingress-controller-operator-fcb9c6f4b-54vsb -n openshift-ingress
I1025 06:32:48.745254 1 run.go:60] tectonic-ingress-controller-operator starting
I1025 06:32:48.845515 1 leaderelection.go:174] attempting to acquire leader lease...
E1025 06:32:49.454221 1 event.go:260] Could not construct reference to: '&v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"tectonic-ingress", GenerateName:"", Namespace:"tectonic-system", SelfLink:"/api/v1/namespaces/tectonic-system/configmaps/tectonic-ingress", UID:"b6261eb9-d819-11e8-8c14-0a580a02000f", ResourceVersion:"51108", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63676043357, loc:(*time.Location)(0x193f080)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"tectonic-ingress-controller-operator-fcb9c6f4b-54vsb\",\"leaseDurationSeconds\":90,\"acquireTime\":\"2018-10-25T05:49:17Z\",\"renewTime\":\"2018-10-25T06:32:48Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Data:map[string]string(nil)}' due to: 'no kind is registered for the type v1.ConfigMap'. Will not report event: 'Normal' 'LeaderElection' 'tectonic-ingress-controller-operator-fcb9c6f4b-54vsb became leader'
I1025 06:32:49.455330 1 leaderelection.go:184] successfully acquired lease tectonic-system/tectonic-ingress
I1025 06:32:49.455442 1 run.go:99] started leading: running tectonic-ingress-controller-operator
I1025 06:32:50.465343 1 update.go:53] Upgrade triggered, req: 1, comp: 0
F1025 06:33:19.246974 1 operator.go:28] Failed to process manifests: unable to create upgrade spec from manifest /manifests/ingress/haproxy-router/svc.yaml: unable to decode service manifest: v1.Service: ObjectMeta: v1.ObjectMeta: Annotations: ReadString: expects " or n, parsing 157 ...assword":2... at {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"prometheus.io/port":"1936","prometheus.io/scrape":"true","prometheus.openshift.io/password":250218,"prometheus.openshift.io/username":"admin","service.alpha.openshift.io/serving-cert-secret-name":"router-metrics-tls","tectonic-operators.coreos.com/upgrade-behaviour":"CreateOrUpgrade","tectonic-operators.coreos.com/upgrade-strategy":"Patch"},"labels":{"component":"ingress-controller","k8s-app":"openshift-router","tectonic-operators.coreos.com/managed-by":"tectonic-ingress-controller-operator"},"name":"router","namespace":"openshift-ingress"},"spec":{"ports":[{"name":"80-tcp","nodePort":32001,"port":80,"targetPort":8080},{"name":"443-tcp","nodePort":32000,"port":443,"targetPort":8443},{"name":"1936-tcp","nodePort":32002,"port":1936,"protocol":"TCP","targetPort":1936}],"selector":{"k8s-app":"openshift-router"},"type":"NodePort"}} |
wking
changed the title
Tectonic-ingress is going away with #600. Maybe try again after that lands? |
And now that's landed. Can you test again? Otherwise I'll probably close this as stale on Friday or so. |
|
I'm using I show the following pods CrashLooping... Could this problem be related to directories ["/var/lib/containers/storage", "/var/lib/kubelet/pods"] landing on / filesystem with xfs filesystem with noquota set? |
Latest pull is even worse... |
|
Can you get versions and logs for the crashing pods and file issues with those repos? Docs for finding repo/commit in flight with #664. |
|
Closing due to inactivity. |
|
@crawford: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
Platform (aws|libvirt|openshift):
libvirt
What happened?
What you expected to happen?
All pods should be in running state.
How to reproduce it (as minimally and precisely as possible)?
Follow the libvirt howto guide and start the cluster.
$ bin/openshift-install clusterAnything else we need to know?
Host system details.
The text was updated successfully, but these errors were encountered: