Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes API is not available after install using latest dev-scripts #771

Closed
chunfuwen opened this issue Aug 27, 2019 · 4 comments
Closed

Kubernetes API is not available after install using latest dev-scripts #771

chunfuwen opened this issue Aug 27, 2019 · 4 comments

Comments

@chunfuwen
Copy link

install console log

level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Fetching \"Install Config\"..."
level=debug msg="Loading \"Install Config\"..."
level=debug msg="  Loading \"SSH Key\"..."
level=debug msg="  Loading \"Base Domain\"..."
level=debug msg="    Loading \"Platform\"..."
level=debug msg="  Loading \"Cluster Name\"..."
level=debug msg="    Loading \"Base Domain\"..."
level=debug msg="  Loading \"Pull Secret\"..."
level=debug msg="  Loading \"Platform\"..."
level=debug msg="Using \"Install Config\" loaded from state file"
level=debug msg="Reusing previously-fetched \"Install Config\""
level=error msg="Cannot fetch the bootstrap and control plane host addresses from state file for baremetal platform"
level=error msg="bootstrap host address and at least one control plane host address must be provided"
level=fatal msg="waiting for Kubernetes API: context deadline exceeded"
make: *** [ocp_run] Error 1

master node log for etcd

[core@master-0 ~]$ sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet)
+ source /run/etcd/environment
++ export ETCD_DISCOVERY_SRV=ostest.test.metalkube.org
++ ETCD_DISCOVERY_SRV=ostest.test.metalkube.org
++ ETCD_IPV4_ADDRESS=192.168.111.20
++ ETCD_DNS_NAME=etcd-0.ostest.test.metalkube.org
++ ETCD_WILDCARD_DNS_NAME='*.ostest.test.metalkube.org'
+ '[' -e /etc/ssl/etcd/system:etcd-server:etcd-0.ostest.test.metalkube.org.crt -a -e /etc/ssl/etcd/system:etcd-server:etcd-0.ostest.test.metalkube.org.key ']'
+ kube-client-agent request --kubeconfig=/etc/kubernetes/kubeconfig --orgname=system:etcd-servers --assetsdir=/etc/ssl/etcd '--dnsnames=localhost,etcd.kube-system.svc,etcd.kube-system.svc.cluster.local,etcd.openshift-etcd.svc,etcd.openshift-etcd.svc.cluster.local,*.ostest.test.metalkube.org' --commonname=system:etcd-server:etcd-0.ostest.test.metalkube.org --ipaddrs=192.168.111.20,127.0.0.1
ERROR: logging before flag.Parse: E0827 08:29:15.987360       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:29:25.986507       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:29:45.988025       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0827 08:29:56.037335       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: read tcp 192.168.111.20:49506->192.168.111.5:6443: read: connection reset by peer
ERROR: logging before flag.Parse: E0827 08:30:05.986309       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:15.986797       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:25.986735       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:35.986845       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:30:55.987163       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0827 08:31:05.987512       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:15.986275       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:25.986536       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:35.986467       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:45.986553       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:31:55.986145       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:05.986902       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:15.986221       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:25.986357       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:32:45.986733       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
ERROR: logging before flag.Parse: E0827 08:32:55.986656       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:05.985978       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:15.986121       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:25.986330       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:35.986483       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:45.986863       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:33:55.986413       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:05.986601       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:15.986423       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:25.986429       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:35.986807       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:45.986458       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:34:55.986482       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:05.986371       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:15.987388       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:25.986737       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 192.168.111.5:6443: connect: connection refused
ERROR: logging before flag.Parse: E0827 08:35:45.986822       7 agent.go:116] error sending CSR to signer: Post https://api-int.ostest.test.metalkube.org:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: net/http: TLS handshake timeout
@russellb
Copy link
Member

Can you get the bootkube log from the bootstrap VM?

@chunfuwen
Copy link
Author

boostrap vm log

[chwen@dell-r730-058 dev-scripts]$ ./bootstrap_openshift_log.sh 
Attempting to follow openshift.service on 192.168.111.56 ...
Warning: Permanently added '192.168.111.56' (ECDSA) to the list of known hosts.
-- Logs begin at Tue 2019-08-27 08:14:05 UTC. --
Aug 28 00:54:12 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:12 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:17 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:17 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:22 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:22 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:27 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:27 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:32 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:32 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:37 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:37 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:43 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:43 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:48 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:48 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:53 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:53 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds...
Aug 28 00:54:58 localhost openshift.sh[1675]: error: unable to recognize "./90_metal3_baremetalhost_crd.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Aug 28 00:54:58 localhost openshift.sh[1675]: kubectl create --filename ./90_metal3_baremetalhost_crd.yaml failed. Retrying in 5 seconds..

@stbenjam
Copy link
Member

You're most likely hitting the issue we found with podman on the current verison of rhcos. podman isn't returning exit codes correctly, so when etcd fails to start the first time, bootkube doesn't retry. We need openshift/installer to update their rhcos release, and then merge #766

@hardys
Copy link

hardys commented Aug 29, 2019

Closed via #766

@hardys hardys closed this as completed Aug 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@russellb @stbenjam @hardys @chunfuwen