Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Error] crc start : Error starting machine: Error in driver during machine start: Unable to detemine VM's IP address, did it fail to boot? #3560

Open
jasonwcc opened this issue Mar 26, 2023 · 10 comments
Open

[Error] crc start : Error starting machine: Error in driver during machine start: Unable to detemine VM's IP address, did it fail to boot? #3560

jasonwcc opened this issue Mar 26, 2023 · 10 comments
Labels
nested-virt To identify issues that are related to nested virtualization resolution/unsupported Unsupported setup; hardware and/or software, nested virtualization status/need more information Issue needs more information before it will be looked at

Comments

@jasonwcc
Copy link

jasonwcc commented Mar 26, 2023 •
edited
Loading

General information

OS: Windows
Hypervisor: Vmware Workstation 16 Pro
Did you run crc setup before starting it (Yes) But it fails
Running CRC on: Dell Desktop

CRC version

CRC version: 2.15.0+cc05160
OpenShift version: 4.12.15
Podman version: 4.3.1

CRC status

CRC VM:          Running
OpenShift:       Unreachable (v4.12.5)
RAM Usage:       511.5MB of 9.399GB
Disk Usage:      12.65GB of 32.74GB (Inside the CRC VM)
Cache Usage:     16.06GB
Cache Directory: /home/student/.crc/cache

CRC config

consent-telemetry                     : no

Host Operating System

NAME="Red Hat Enterprise Linux"
VERSION="9.1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
Red Hat Enterprise Linux release 9.1 (Plow)
Red Hat Enterprise Linux release 9.1 (Plow)
processor       : 0-7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 167
model name      : 11th Gen Intel(R) Core(TM) i9-11900KF @ 3.50GHz
stepping        : 1
microcode       : 0x57
cpu MHz         : 3503.999
cache size      : 16384 KB
physical id     : 1
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 27
wp              : yes

Steps to reproduce

# crc delete -f
Deleted the instance

# crc cleanup
INFO Removing vsock configuration
INFO Removing 'crc' network from libvirt
INFO Removing /etc/NetworkManager/dnsmasq.d/crc.conf file
INFO Using root access: Removing NetworkManager configuration file in /etc/NetworkManager/dnsmasq.d/crc.conf
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Removing /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf file
INFO Using root access: Removing NetworkManager configuration file in /etc/NetworkManager/conf.d/crc-nm-dnsmasq.con
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Removing crc daemon systemd socket units
INFO Removing crc daemon systemd service
INFO Removing crc's virtual machine
INFO Removing crc libvirt storage pool
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
Cleanup finished

# crc setup
INFO Using bundle path /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Creating symlink for crc executable
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd service
INFO Setting up crc daemon systemd service
INFO Checking crc daemon systemd socket units
INFO Setting up crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Writing Network Manager config for crc
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf
INFO Using root access: Changing permissions for /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf to 644
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Writing dnsmasq config for crc
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/dnsmasq.d/crc.conf
INFO Using root access: Changing permissions for /etc/NetworkManager/dnsmasq.d/crc.conf to 644
INFO Using root access: Executing systemctl daemon-reload command
INFO Using root access: Executing systemctl reload NetworkManager
INFO Checking if libvirt 'crc' network is available
INFO Setting up libvirt 'crc' network
INFO Checking if libvirt 'crc' network is active
INFO Starting libvirt 'crc' network
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

# crc start
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_libvirt_4.12.5_amd64...
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret *************************************************************************************

INFO Creating CRC VM for openshift 4.12.5...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.12.5...
**** wait for good old 20 minutes. Then this ***
Error starting machine: Error in driver during machine start: Unable to determine VM's IP address, did it fail to boot?

View logs

# crc setup --log-level debug
DEBU CRC version: 2.15.0+cc05160
DEBU OpenShift version: 4.12.5
DEBU Podman version: 4.3.1
DEBU Running 'crc setup'
INFO Using bundle path /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-resolved.service could not be found.
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
DEBU Running '/home/student/.crc/bin/crc-admin-helper-linux --version'
DEBU Found crc-admin-helper-linux version 0.0.12
DEBU crc-admin-helper executable already cached
INFO Checking if running on a supported CPU architecture
DEBU GOARCH is amd64 GOOS is linux
INFO Checking minimum RAM requirements
DEBU Total memory of system is 12240977920 bytes
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo
DEBU CPU virtualization flags are good
INFO Checking if KVM is enabled
DEBU Checking if /dev/kvm exists
DEBU /dev/kvm was found
INFO Checking if libvirt is installed
DEBU Checking if 'virsh' is available
DEBU 'virsh' was found in /usr/bin/virsh
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability
DEBU Running 'virsh --readonly --connect qemu:///system capabilities'
DEBU Found x86_64 hypervisor with 'hvm' capabilities
INFO Checking if user is part of libvirt group
DEBU Checking if current user is part of the libvirt group
DEBU Current user is already in the libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
DEBU Checking if libvirtd service is running
DEBU Running 'systemctl status virtqemud.socket'
DEBU Command failed: exit status 3
DEBU stdout: * virtqemud.socket - Libvirt qemu local socket
     Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; enabled; vendor preset: disabled)
     Active: inactive (dead)
   Triggers: * virtqemud.service
     Listen: /run/libvirt/virtqemud-sock (Stream)
DEBU stderr:
DEBU virtqemud.socket is neither running nor listening
DEBU Running 'systemctl status libvirtd.socket'
DEBU libvirtd.socket is running
INFO Checking if a supported libvirt version is installed
DEBU Checking if libvirt version is >=3.4.0
DEBU Running 'virsh -v'
INFO Checking if crc-driver-libvirt is installed
DEBU Checking if crc-driver-libvirt is installed
DEBU Running '/home/student/.crc/bin/crc-driver-libvirt version'
DEBU Found crc-driver-libvirt version 0.13.5
DEBU crc-driver-libvirt is already installed
INFO Checking crc daemon systemd service
DEBU Checking crc daemon systemd service
DEBU Checking if crc-daemon.service is running
DEBU Running 'systemctl --user status crc-daemon.service'
DEBU Command failed: exit status 3
DEBU stdout: * crc-daemon.service - CRC daemon
     Loaded: loaded (/home/student/.config/systemd/user/crc-daemon.service; static)
     Active: inactive (dead)
TriggeredBy: * crc-vsock.socket
             * crc-http.socket

Mar 26 09:56:49 crc2.example.com systemd[2233]: Starting CRC daemon...
Mar 26 09:56:49 crc2.example.com crc[8841]: level=info msg="using socket provided by crc-vsock.socket"
Mar 26 09:56:49 crc2.example.com crc[8841]: level=info msg="using socket provided by crc-http.socket"
Mar 26 09:56:49 crc2.example.com systemd[2233]: Started CRC daemon.
Mar 26 09:56:50 crc2.example.com crc[8841]: @ - - [26/Mar/2023:09:56:49 +0800] "GET /api/status HTTP/1.1" 200 189
Mar 26 09:57:10 crc2.example.com crc[8841]: @ - - [26/Mar/2023:09:57:09 +0800] "GET /api/status HTTP/1.1" 200 189
Mar 26 10:07:42 crc2.example.com systemd[2233]: Stopping CRC daemon...
Mar 26 10:07:42 crc2.example.com systemd[2233]: Stopped CRC daemon.
DEBU stderr:
DEBU crc-daemon.service is neither running nor listening
DEBU Checking if crc-daemon.service has the expected content
INFO Checking crc daemon systemd socket units
DEBU Checking crc daemon systemd socket units
DEBU Checking if crc-http.socket is running
DEBU Running 'systemctl --user status crc-http.socket'
DEBU crc-http.socket is listening
DEBU Checking if crc-http.socket has the expected content
DEBU Checking if crc-vsock.socket is running
DEBU Running 'systemctl --user status crc-vsock.socket'
DEBU crc-vsock.socket is listening
DEBU Checking if crc-vsock.socket has the expected content
INFO Checking if systemd-networkd is running
DEBU Checking if systemd-networkd.service is running
DEBU Running 'systemctl status systemd-networkd.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-networkd.service could not be found.
DEBU systemd-networkd.service is not running
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU Running 'systemctl status NetworkManager.service'
DEBU NetworkManager.service is already running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
DEBU Checking NetworkManager configuration
DEBU NetworkManager configuration is good
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
DEBU Checking dnsmasq configuration
DEBU dnsmasq configuration is good
INFO Checking if libvirt 'crc' network is available
DEBU Checking if libvirt 'crc' network exists
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU Checking if libvirt 'crc' definition is up to date
DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc'
DEBU libvirt 'crc' network has the expected value
INFO Checking if libvirt 'crc' network is active
DEBU Checking if libvirt 'crc' network is active
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU libvirt 'crc' network is already active
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists
DEBU /home/student/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
@praveenkumar
Copy link
Member

This looks like a nested virtualization setup, @jasonwcc any reason why don't you start crc directly on the windows using hyperV instead nested virtualization setup? We have close to no testing around nested virtualization side.

@jasonwcc
Copy link
Author

jasonwcc commented Mar 29, 2023 via email

@praveenkumar
Copy link
Member

@jasonwcc what was the last crc version you tried and worked as expected?

@jasonwcc
Copy link
Author

jasonwcc commented Apr 4, 2023 via email

@praveenkumar
Copy link
Member

@jasonwcc there is lot of things changes from each openshift release on OS level also (like kernel package updates) we are not sure what changed and not working for you. We also don't have any CI in place to do nested virtualization test for these kind of setup and mentioned in the doc.

@praveenkumar praveenkumar added the nested-virt To identify issues that are related to nested virtualization label Apr 11, 2023
@gbraad
Copy link
Contributor

gbraad commented Apr 19, 2023 •
edited
Loading

Error starting machine: Error in driver during machine start: Unable to determine VM's IP address, did it fail to boot?

Most likely a VMware related issue. Without further info it is hard to determine the cause. Please try to add more info, such as console output. You could try to disable any VPN you run and IPv6.

Unsupported.

@gbraad gbraad added status/need more information Issue needs more information before it will be looked at resolution/unsupported Unsupported setup; hardware and/or software, nested virtualization labels Apr 19, 2023
@wuchuxiong
Copy link

I face exact the same issue with crc 2.13.1+3b466f8 on RHEL 9.1.
Similar issue found at #2397 which is not really fixed. If I try multiple times and reboot my VM, it may succeed occasionally. But this is still an issue to me.

[myuser@myhost crc-linux-2.13.1-amd64]$ ./crc start -m 24000 -c 8 -d 100
WARN A new version (2.20.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.20.0/crc-linux-amd64.tar.xz
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_libvirt_4.12.0_amd64...
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret ****************************************************************************************************************************************************************
INFO Creating CRC VM for openshift 4.12.0...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.12.0...
Error starting machine: Error in driver during machine start: Unable to determine VM's IP address, did it fail to boot?

@cfergeau
Copy link
Contributor

At this point 4.12.0 is quite old, any chance you could try a newer version? Either the latest release with a 4.12.x bundle, or the latest release which ships with OpenShift 4.13?
And is this with nested virtualization?

@valkiriaaquatica
Copy link

valkiriaaquatica commented Jul 25, 2024 •
edited
Loading

Hey,
same issue here but with a different problem approach.
What I'm running :

  • A latest fedora 40VM vm in a ESXII7 (the vm has nested virtualziaicon applied in ESXII config)
  • Crc version by the moment: 2.39.0+0849cb
  • Openshift version: 4.16.0
  • libvirtd version: 10.1.0
  • dnsmasq version 2.90
  • hostname of the vm: localhost

I have found the problem is related with the service libvirtd as is shown in the following exmaple:

  1. I run crc start and I get this:
crc start --log-level debug
DEBU CRC version: 2.39.0+0849cb
DEBU OpenShift version: 4.16.0
DEBU Running 'crc start'
DEBU Total memory of system is 15705567232 bytes
DEBU No new version available. The latest version is 2.39.0
DEBU Checking file: /home/user/.crc/machines/crc/.crc-exist
DEBU Found binary path at /home/user/.crc/bin/crc-driver-libvirt-amd64
DEBU Launching plugin server for driver libvirt
DEBU Plugin server listening at address 127.0.0.1:40199
DEBU () Calling .GetVersion
DEBU Using API Version 1
DEBU () Calling .SetConfigRaw
DEBU () Calling .GetMachineName
DEBU (crc) Calling .GetBundleName
DEBU (crc) Calling .GetState
DEBU (crc) DBG | time="2024-07-25T19:12:15+02:00" level=debug msg="Getting current state..."
DEBU (crc) DBG | time="2024-07-25T19:12:15+02:00" level=debug msg="Fetching VM..."
DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU (crc) DBG | time="2024-07-25T19:12:15+02:00" level=debug msg="Closing plugin on server side"
DEBU Failed to make call to close driver server: unexpected EOF
DEBU Making call to close connection to plugin binary
DEBU Checking file: /home/user/.crc/machines/crc/.crc-exist
INFO Loading bundle: crc_libvirt_4.16.0_amd64...
DEBU Checking file: /home/user/.crc/machines/crc/.crc-exist
DEBU Found binary path at /home/user/.crc/bin/crc-driver-libvirt-amd64
DEBU Launching plugin server for driver libvirt
DEBU Plugin server listening at address 127.0.0.1:32901
DEBU () Calling .GetVersion
DEBU Using API Version 1
DEBU () Calling .SetConfigRaw
DEBU () Calling .GetMachineName
DEBU (crc) Calling .GetBundleName
DEBU (crc) Calling .GetState
DEBU (crc) DBG | time="2024-07-25T19:12:16+02:00" level=debug msg="Getting current state..."
DEBU (crc) DBG | time="2024-07-25T19:12:16+02:00" level=debug msg="Fetching VM..."
INFO A CRC VM for OpenShift 4.16.0 is already running
DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU (crc) DBG | time="2024-07-25T19:12:16+02:00" level=debug msg="Closing plugin on server side"
DEBU Failed to make call to close driver server: unexpected EOF
DEBU Making call to close connection to plugin binary
Started the OpenShift cluster.
INFO Loading bundle: crc_libvirt_4.16.0_amd64...
INFO A CRC VM for OpenShift 4.16.0 is already running
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: 123456789

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
  1. In other terminal check for libvirtd service logs (localhost is the name of the fedora40 machine)
 sudo journalctl -u libvirtd -f
Jul 25 19:10:44 localhost systemd[1]: Starting libvirtd.service - libvirt legacy monolithic daemon...
Jul 25 19:10:45 localhost systemd[1]: Started libvirtd.service - libvirt legacy monolithic daemon.
Jul 25 19:11:03 localhost libvirtd[1781]: libvirt version: 10.1.0, package: 3.fc40 (Fedora Project, 2024-07-09-11:57:17, )
Jul 25 19:11:03 localhost libvirtd[1781]: hostname: localhost
Jul 25 19:11:03 localhost libvirtd[1781]: End of file while reading data: Input/output error
Jul 25 19:11:03 localhost libvirtd[1781]: End of file while reading data: Input/output error
  1. There is a relation between everytime I run crc start with a new errror of: Jul 25 18:53:20 localhost libvirtd[4632]: End of file while reading data: Input/output error

I have trid to restart the libvirtd service and the dnsmasq service, but still not workking.

@praveenkumar
Copy link
Member

@valkiriaaquatica nested virtualization is something not tested by us also as per logs I can see that cluster is already running so I am not sure what is the issue you are facing can you delete the cluster and provide the output of following commands (may be in a gist)

$ crc delete
$ crc setup --log-level debug
$ crc start --log-level debug
$ sudo cat /var/log/libvirt/qemu/crc.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
nested-virt To identify issues that are related to nested virtualization resolution/unsupported Unsupported setup; hardware and/or software, nested virtualization status/need more information Issue needs more information before it will be looked at
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@gbraad @praveenkumar @cfergeau @wuchuxiong @jasonwcc @valkiriaaquatica