Nested virtualization on VMware Fusion 12.1.0 -> CentOS (KVM) -> CRC 1.123 not working as expected (slow performance)

[dominikbaeribm]

General information

• OS: Linux /
• Hypervisor: KVM
• Did you run crc setup before starting it -- Yes
• Running CRC on: Laptop (VM: Centos 8)

CRC version 1.23

OpenShift version: 4.7.0 (embedded in executable)
  
## CRC status
```[db2oltp@localhost ~]$ crc status
CRC VM:          Running
OpenShift:       Unreachable (v4.7.0)
Disk Usage:      9.105GB of 32.74GB (Inside the CRC VM)
Cache Usage:     13.01GB
Cache Directory: /home/db2oltp/.crc/cache

# Put `crc status --log-level debug` output here
```crc status --log-level debug
DEBU CodeReady Containers version: 1.23.1+be17b141 
DEBU OpenShift version: 4.7.0 (embedded in executable) 
DEBU Running 'crc status'                         
DEBU Checking file: /home/db2oltp/.crc/machines/crc/.crc-exist 
DEBU Checking file: /home/db2oltp/.crc/machines/crc/.crc-exist 
DEBU Found binary path at /home/db2oltp/.crc/bin/crc-driver-libvirt 
DEBU Launching plugin server for driver libvirt   
DEBU Plugin server listening at address 127.0.0.1:35337 
DEBU () Calling .GetVersion                       
DEBU Using API Version 1                          
DEBU () Calling .SetConfigRaw                     
DEBU () Calling .GetMachineName                   
DEBU (crc) Calling .GetState                      
DEBU (crc) DBG | time="2021-03-09T13:45:34-08:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2021-03-09T13:45:34-08:00" level=debug msg="Fetching VM..." 
DEBU (crc) Calling .GetBundleName                 
DEBU (crc) Calling .GetIP                         
DEBU (crc) DBG | time="2021-03-09T13:45:34-08:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2021-03-09T13:45:34-08:00" level=debug msg="Getting current state..." 
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1 
DEBU (crc) DBG | time="2021-03-09T13:45:34-08:00" level=debug msg="IP address: 192.168.130.11" 
DEBU Using ssh private keys: [/home/db2oltp/.crc/machines/crc/id_ecdsa] 
DEBU SSH command results: err: <nil>, output: 32737570816 12141580288 /sysroot 
DEBU Running private SSH command                  
DEBU SSH command failed                           
DEBU cannot get OpenShift status: ssh command error:
command : timeout 5s oc get co -ojson --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1\n 
DEBU Making call to close driver server           
DEBU (crc) Calling .Close                         
DEBU (crc) DBG | time="2021-03-09T13:45:37-08:00" level=debug msg="Closing plugin on server side" 
DEBU Successfully made call to close driver server 
DEBU Making call to close connection to plugin binary 
CRC VM:          Running
OpenShift:       Unreachable (v4.7.0)
Disk Usage:      12.14GB of 32.74GB (Inside the CRC VM)
Cache Usage:     13.01GB
Cache Directory: /home/db2oltp/.crc/cache
[db2oltp@localhost ~]$ 


## CRC config
```crc config view
- consent-telemetry                     : yes

## Host Operating System
```Mac 11.2.2


# Put the output of `cat /etc/os-release` in case of Linux
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
[db2oltp@localhost ~]$ 

Steps to reproduce

1. crc delete
2. crc cleanup
3. crc setup
4. crc start -c 8 -m 28000 -p /home/db2oltp/pull-secret -d 80

Expected

a stable running cluster - I have enough resources on the machine.
It looks like to me, that this version is not really good. I tried with fedora 33 and ubuntu 20.1 and both have issues as well.
The only linux system I can see the OCP console is this one here, but this one is not stable at all.

Actual

The cluster takes 12min!!!! to start and comes up with warnings. The UI is sometimes reachable, sometimes not.
The major error message on the UI says that admin is only temporarily and that authentication should be adapted. As soon as I click on the link, the link is broken.
"You are logged in as a temporary administrative user. Update the cluster OAuth configuration to allow others to log in."
Looks now after the 3rd time I restarted the cluster it seems to work...???

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

I did that several times

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

[gbraad]

crc 1.23 on centos 8 ( 8 core, 28GB RAM , 100 GB Disk) in VM (Vmware Fusion 12.1.0) Mac 11.2.2 -- i9, 8cores, 64GB RAM

We do not support a nested virtualization setup; especially when a hypervisor is used other than Hyperkit, Hyper-V or KVM.

The reasons are plenty, mostly due to unexplained issues or just slowness/performance issues.

[gbraad]

The cluster takes 12min!!!! to start and comes up with warnings.

This is a recorded known issue
https://access.redhat.com/documentation/en-us/red_hat_codeready_containers/1.23/html/release_notes_and_known_issues/general_issues#increased_startup_time_for_the_openshift_container_platform_4_7_cluster