bootstrap error kubelet.service Failed to locate executable /usr/bin/hyperkube No such file or directory

[bobby0724]

Describe the bug

Trying to perform UPI installation on bare metal I cannot pass the bootstrap initialization
I can connect to the bootstrap server and get the following

This is the bootstrap node; it will be destroyed when the master is fully up.

The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.

journalctl -b -f -u release-image.service -u bootkube.service
Fedora CoreOS 34.20210518.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Last login: Sat Jun 12 16:39:59 2021 from 10.10.41.19
[core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Journal begins at Sat 2021-06-12 16:39:32 UTC. --
Jun 12 16:43:13 localhost systemd[1]: release-image.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 16:43:13 localhost systemd[1]: release-image.service: Failed with result 'exit-code'.
Jun 12 16:43:13 localhost systemd[1]: Failed to start Download the OpenShift Release Image.
Jun 12 16:43:13 localhost systemd[1]: Dependency failed for Bootstrap a Kubernetes cluster.
Jun 12 16:43:13 localhost systemd[1]: bootkube.service: Job bootkube.service/start failed with result 'dependency'.
Jun 12 16:43:23 localhost systemd[1]: Starting Download the OpenShift Release Image...
Jun 12 16:43:23 localhost release-image-download.sh[3488]: Pulling quay.io/openshift/okd@sha256:78ebb2404a0f3a5622fa141a6bcc923284d773ed0ccf65239a47ede1a3ad07d2...
Jun 12 16:43:25 localhost podman[3489]: 2021-06-12 16:43:25.004940616 +0000 UTC m=+1.446180099 image pull
Jun 12 16:43:25 localhost release-image-download.sh[3489]: bff465f7ce0411ee83bc0054ff5199a8952f7d58d8cb8d47b623595df2532f9b
Jun 12 16:43:25 localhost systemd[1]: Finished Download the OpenShift Release Image.

Jun 12 17:14:55 localhost systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 181.
Jun 12 17:14:55 localhost systemd[1]: Stopped Kubernetes Kubelet.
Jun 12 17:14:55 localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib>
Jun 12 17:14:55 localhost systemd[1]: Starting Kubernetes Kubelet...
Jun 12 17:14:55 localhost audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/>
Jun 12 17:14:55 localhost kernel: audit: type=1130 audit(1623518095.044:947): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet >
Jun 12 17:14:55 localhost kernel: audit: type=1131 audit(1623518095.044:948): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet >
Jun 12 17:14:55 localhost podman[26753]: 2021-06-12 17:14:55.209168739 +0000 UTC m=+0.078696186 container create 3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca53>
Jun 12 17:14:55 localhost systemd[1]: Started libcrun container.
Jun 12 17:14:55 localhost systemd-resolved[1100]: Failed to determine the local hostname and LLMNR/mDNS names, ignoring: No such device or address
Jun 12 17:14:55 localhost podman[26753]: 2021-06-12 17:14:55.25276033 +0000 UTC m=+0.122287755 container init 3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca535c3>
Jun 12 17:14:55 localhost podman[26753]: 2021-06-12 17:14:55.258194541 +0000 UTC m=+0.127721965 container start 3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca535>
Jun 12 17:14:55 localhost podman[26753]: 2021-06-12 17:14:55.258269492 +0000 UTC m=+0.127796907 container attach 3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca53>
Jun 12 17:14:55 localhost systemd[1]: libpod-3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca535c337.scope: Deactivated successfully.
Jun 12 17:14:55 localhost podman[26753]: 2021-06-12 17:14:55.307022608 +0000 UTC m=+0.176550033 container died 3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca535c>
Jun 12 17:14:55 localhost podman[26753]: 2021-06-12 17:14:55.358613354 +0000 UTC m=+0.228140778 container remove 3787ac6f71a9780887ebefa8cda60b4307ba1d2535b76a8e9eda090ca53>
Jun 12 17:14:55 localhost systemd[1]: kubelet.service: Found left-over process 26790 (conmon) in control group while starting unit. Ignoring.
Jun 12 17:14:55 localhost systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 12 17:14:55 localhost systemd[26838]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Jun 12 17:14:55 localhost systemd[26838]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Jun 12 17:14:55 localhost systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jun 12 17:14:55 localhost systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jun 12 17:14:55 localhost systemd[1]: Failed to start Kubernetes Kubelet.
Jun 12 17:14:55 localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib>
Jun 12 17:14:55 localhost kernel: audit: type=1130 audit(1623518095.365:949): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet

Version

UPI installation version 4.7.0-0.okd-2021-06-04-191031
How reproducible
I have try with vmware bootstrap vm and also with a bare metal server and I always get the same result

here is my install-config

apiVersion: v1
baseDomain: domain.com
compute:

• hyperthreading: Enabled
  name: worker
  replicas: 0
  controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
  metadata:
  name: okd4lab
  networking:
  clusterNetworks:
  • cidr: 172.16.0.0/16
    hostPrefix: 23
    networkType: OpenShiftSDN
    serviceNetwork:
  • 172.30.0.0/16
    platform:
    none: {}
    pullSecret:
    sshKey:

Log bundle

[vrutkovs]

Please attach log bundle

[bobby0724]

hello vrutkovs

I do a research in the issues and find interesting point to use version 33.20210217.3.0 of fcos
and now the bootstrap server was build correctly

I was using fedora-coreos-34.20210518.3.0-metal.x86_64.raw.xz and the bootstrap build never ends