Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] x509 ingress controller error when connecting from macOS #3375

Closed
KevinMGranger opened this issue Oct 4, 2022 · 7 comments
Closed

[BUG] x509 ingress controller error when connecting from macOS #3375

KevinMGranger opened this issue Oct 4, 2022 · 7 comments
Labels
kind/bug Something isn't working status/need triage

Comments

@KevinMGranger
Copy link

Note: crc-related information is on the linux machine, and oc-related information is on the macOS machine.

I also embedded various files and log outputs for convenience. I can move them to a gist if you wish.

I actually wrote this up for the first time on September 12th but never posted it. Now that the M1 is finally supported, I tried running it locally on my mac-- and I'm getting the same error!

General information

  • OS: Linux, but also macOS
    • The CRC instance is running on Linux, but I'm using oc from macOS.
  • Hypervisor: KVM
  • Did you run crc setup before starting it (Yes/No)? Yes
  • Running CRC on: Baremetal-Server (desktop)

CRC version

linux$ crc version
CRC version: 2.8.0+217b3bd
OpenShift version: 4.11.1
Podman version: 4.1.1

OC version

mac$ oc version
Client Version: 4.11.1
Kustomize Version: v4.5.4
error: You must be logged in to the server (Unauthorized)

CRC status

linux$ crc status --log-level debug
level=debug msg="CRC version: 2.8.0+217b3bd\n"
level=debug msg="OpenShift version: 4.11.1\n"
level=debug msg="Podman version: 4.1.1\n"
level=debug msg="Running 'crc status'"
level=debug msg="Checking file: /home/kevin/.crc/machines/crc/.crc-exist"
level=debug msg="Checking file: /home/kevin/.crc/machines/crc/.crc-exist"
level=debug msg="Found binary path at /home/kevin/.crc/bin/crc-driver-libvirt"
level=debug msg="Launching plugin server for driver libvirt"
level=debug msg="Plugin server listening at address 127.0.0.1:41339"
level=debug msg="() Calling .GetVersion"
level=debug msg="Using API Version 1"
level=debug msg="() Calling .SetConfigRaw"
level=debug msg="() Calling .GetMachineName"
level=debug msg="(crc) Calling .GetBundleName"
level=debug msg="(crc) Calling .GetState"
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Getting current state...\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Fetching VM...\""
level=debug msg="(crc) Calling .GetIP"
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"GetIP called for crc\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Getting current state...\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"IP address: 192.168.130.11\""
level=debug msg="(crc) Calling .GetIP"
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"GetIP called for crc\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Getting current state...\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"IP address: 192.168.130.11\""
level=debug msg="Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1"
level=debug msg="Using ssh private keys: [/home/kevin/.crc/machines/crc/id_ecdsa /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64/id_ecdsa_crc]"
level=debug msg="SSH command results: err: <nil>, output: 68171051008 15629176832 /sysroot\n"
level=debug msg="Making call to close driver server"
level=debug msg="(crc) Calling .Close"
level=debug msg="Successfully made call to close driver server"
level=debug msg="Making call to close connection to plugin binary"
CRC VM:          Running
OpenShift:       Running (v4.11.1)
Podman:          
Disk Usage:      15.63GB of 68.17GB (Inside the CRC VM)
Cache Usage:     17.04GB
Cache Directory: /home/kevin/.crc/cache

CRC config

linux$ crc config view
- consent-telemetry                     : yes
- disk-size                             : 64
- memory                                : 20480
- pull-secret-file                      : /home/kevin/crc_pull_secret

Host Operating System

Linux

linux$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="35 (Server Edition)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server

macOS

mac$ sw_vers
ProductName:    macOS
ProductVersion: 12.5.1
BuildVersion:   21G83

topology details

There are three machines, each with different purposes:

  1. a macOS machine, used for day-to-day work. I run oc from it. Let's call this one mac
  2. a powerful Linux desktop, used to run crc. Let's call this one linux.
  3. a low-power Linux computer, used to run DNS (dnsmasq). Let's call this one dns-linux.

All three machines use the third for DNS configuration.

They are connected over tailscale, although that shouldn't matter much.

Other relevant config files

kubeconfig on linux

~/.crc/machines/crc/kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXekNDQWtPZ0F3SUJBZ0lJVEU2NHBCWlQzVzR3RFFZSktvWklodmNOQVFFTEJRQXdKakVrTUNJR0ExVUUKQXd3YmFXNW5jbVZ6Y3kxdmNHVnlZWFJ2Y2tBeE5qWXhNekUxTURnek1CNFhEVEl5TURneU5EQTBNalEwTkZvWApEVEkwTURneU16QTBNalEwTlZvd0hURWJNQmtHQTFVRUF3d1NLaTVoY0hCekxXTnlZeTUwWlhOMGFXNW5NSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6MW5kK2tWL090VmRITzVQbkxubkNGLzcKczBkMzkxY0J4TGkxQzcyS0lIVEtxWFJ1eXNYQjBYbFdiT1JRZ0o3b3dieTIxbGR6bVBFdWxoejZlZFdOcXZFMwp6SEloNDViSFpBSUdCaDN1UHB4SFhwa0VING5xY2NLd3hna3EyZEdwZVJhN3NVUUI1T1Z5NVVUellTYkxIU1JoCmduMjhlcDVtUE81MzBhK0FnRVVVMy84NTRlbVZyMm5XL29YZVJMZmpYZVUvSzBrNkJpQ2Y3dFpEcUtub0tHUHQKRC8xbjBBZmlHclNUK0NUNllWS281S0VSUTFPa2V0Mm56Mnd3Y1krWkZBTCt0L2trMW52WXI5WVlLUlJJRngxRQp6M0t0TUEzTk9ldnEwZ3VGeTVKZkdHN3dKZ2lMRWNtZDZZVjNzbjY1RllqNjJ5dTZ6ajdDMkNjOUFIR1NJd0lECkFRQUJvNEdWTUlHU01BNEdBMVVkRHdFQi93UUVBd0lGb0RBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREFUQU0KQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUTNBL24zQk5nRXMrWm9pdEZFc0d6cVFCSExYekFmQmdOVgpIU01FR0RBV2dCUWllaTRIZG5BVzgzVVh2Um1UdllsYkJ4ZmlqVEFkQmdOVkhSRUVGakFVZ2hJcUxtRndjSE10ClkzSmpMblJsYzNScGJtY3dEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ2k2aDgyYVRiQ1RpNU9UV09TWTk3bTIKb1FUckJGQ3BLWHdBYWMvdEtjc1V4ZUZKZHBHaFhHa2ZnMklPWXMvRldORzdGNU4wZTJkQ2lROXQvL2x2UlVhTAp3NkRnYWZYZ1BldFdvb2svMGpJakJKWi9oK3RMNjlhOE5qcWcxb2ZWem5JcmM5eE5kRldCSzdpVklraUpRV0k0Cmwyc2xBWElnQUVQUnJLYmtyWXN4UkRXcXlkL3l6eDNOZTJQaldOZUFpV1h1MS9XNTRyWC9QRVg1cmxpSVR4WkcKWU1tSG4vWnp3VXk1UlFwTjZhMEhkTTR3akNGdTFxUFFJdjd3S21nVXJRQnVURFp4ZSt1VmpabVlGYXdtSkQ0VApPdVZLK2h1NjdJTnZsdzJhT2I4dDJROU5JVmRzRWJjTStuRVQxNjh3ZEZpL3BvSFMyOFk5WGV0LzIrSlcvdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURERENDQWZTZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0cGJtZHkKWlhOekxXOXdaWEpoZEc5eVFERTJOakV6TVRVd09ETXdIaGNOTWpJd09ESTBNRFF5TkRReVdoY05NalF3T0RJegpNRFF5TkRReldqQW1NU1F3SWdZRFZRUUREQnRwYm1keVpYTnpMVzl3WlhKaGRHOXlRREUyTmpFek1UVXdPRE13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURmUVRMQWNFdExNYXdnaVQyU1FOLzcKdEttbm9rN1gwMll0S0NVU3ZRVjBuUWhMZGZ0c3RocGZVdTdqa3lCRXRhSTI4ajlJbXU1L1Q0QU1yUE5vOEhpcgpwNkhLbHB3cDVubjM3REpwUFJ0ZHc3dmd5UmdqRmpxcmR0Vk5VWXJidWFTdEJ3WTl2c2ppNFhvNTR1OHMxWkpGCjNOOWJ2c20rMm15SnZEd0VyNG5OZXZwZDV6WDlqR1g1UU1oS1RhdHFsMTJoaXNVYWNJRTcvaFgzdUoyZXZxcU8KcGtqVU93Ri8yVlNVbkd4Qmg5NHpXRVNUNlI2SVcrUFo5Zlo0d25kRFVzK3laWVZCdmIxRGVaQ3NLQW9EU0l1Ugpnc2pvbzYyKzZqR0Q2bDl1QklQVlE5bnozRFgzeW1FTEdjbzR5TFh0eGlkS282Y0sraUwyZTRMblB2QWFZRVRUCkFnTUJBQUdqUlRCRE1BNEdBMVVkRHdFQi93UUVBd0lDcERBU0JnTlZIUk1CQWY4RUNEQUdBUUgvQWdFQU1CMEcKQTFVZERnUVdCQlFpZWk0SGRuQVc4M1VYdlJtVHZZbGJCeGZpalRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQQpGTjhZRDhFeTNUMVpROGdrS0Y4VFZSVEdwZG5aTmlLUjhtVjZaOEt5WUtyNVpETUJHd29tTXVWZUw3di85NlBOCmtRSUhGUzZ4QzhnSkxrUjFpdkZJZFlHdmxqN3dvbXcxS0p4V0k5SlBGZnVWTzhIbU9NaDNzNCtJUXRyV1QvYlgKSDZMWVlLLzByWXNiMDIzTGtYUTJGdU1iRzNxeWJEcnJCc3crNDlFVmkya3lRbHF2ZExBSUEwNEtKdExuL0RaagpOZEJyc3k4a0tzaEhyOGJpdnpsT2E4V0xDTDBlcTBUb3NrT1hOUUlYMmNXVVRGMlNvcUV3QmF4a2ZzdDUxTUduCjZHM1FrZmhYRDdXYk8rTm5OTmJjbTFxeHhQd2EzdUFBUXo3N3BiMUNJTG91MHVqdWtLM3h5MTYwWlVaVzNGSWIKQWV2ejFxYzJqckJTU1NLYjFSdTNVUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXekNDQWtPZ0F3SUJBZ0lJVEU2NHBCWlQzVzR3RFFZSktvWklodmNOQVFFTEJRQXdKakVrTUNJR0ExVUUKQXd3YmFXNW5jbVZ6Y3kxdmNHVnlZWFJ2Y2tBeE5qWXhNekUxTURnek1CNFhEVEl5TURneU5EQTBNalEwTkZvWApEVEkwTURneU16QTBNalEwTlZvd0hURWJNQmtHQTFVRUF3d1NLaTVoY0hCekxXTnlZeTUwWlhOMGFXNW5NSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6MW5kK2tWL090VmRITzVQbkxubkNGLzcKczBkMzkxY0J4TGkxQzcyS0lIVEtxWFJ1eXNYQjBYbFdiT1JRZ0o3b3dieTIxbGR6bVBFdWxoejZlZFdOcXZFMwp6SEloNDViSFpBSUdCaDN1UHB4SFhwa0VING5xY2NLd3hna3EyZEdwZVJhN3NVUUI1T1Z5NVVUellTYkxIU1JoCmduMjhlcDVtUE81MzBhK0FnRVVVMy84NTRlbVZyMm5XL29YZVJMZmpYZVUvSzBrNkJpQ2Y3dFpEcUtub0tHUHQKRC8xbjBBZmlHclNUK0NUNllWS281S0VSUTFPa2V0Mm56Mnd3Y1krWkZBTCt0L2trMW52WXI5WVlLUlJJRngxRQp6M0t0TUEzTk9ldnEwZ3VGeTVKZkdHN3dKZ2lMRWNtZDZZVjNzbjY1RllqNjJ5dTZ6ajdDMkNjOUFIR1NJd0lECkFRQUJvNEdWTUlHU01BNEdBMVVkRHdFQi93UUVBd0lGb0RBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREFUQU0KQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUTNBL24zQk5nRXMrWm9pdEZFc0d6cVFCSExYekFmQmdOVgpIU01FR0RBV2dCUWllaTRIZG5BVzgzVVh2Um1UdllsYkJ4ZmlqVEFkQmdOVkhSRUVGakFVZ2hJcUxtRndjSE10ClkzSmpMblJsYzNScGJtY3dEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ2k2aDgyYVRiQ1RpNU9UV09TWTk3bTIKb1FUckJGQ3BLWHdBYWMvdEtjc1V4ZUZKZHBHaFhHa2ZnMklPWXMvRldORzdGNU4wZTJkQ2lROXQvL2x2UlVhTAp3NkRnYWZYZ1BldFdvb2svMGpJakJKWi9oK3RMNjlhOE5qcWcxb2ZWem5JcmM5eE5kRldCSzdpVklraUpRV0k0Cmwyc2xBWElnQUVQUnJLYmtyWXN4UkRXcXlkL3l6eDNOZTJQaldOZUFpV1h1MS9XNTRyWC9QRVg1cmxpSVR4WkcKWU1tSG4vWnp3VXk1UlFwTjZhMEhkTTR3akNGdTFxUFFJdjd3S21nVXJRQnVURFp4ZSt1VmpabVlGYXdtSkQ0VApPdVZLK2h1NjdJTnZsdzJhT2I4dDJROU5JVmRzRWJjTStuRVQxNjh3ZEZpL3BvSFMyOFk5WGV0LzIrSlcvdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURERENDQWZTZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0cGJtZHkKWlhOekxXOXdaWEpoZEc5eVFERTJOakV6TVRVd09ETXdIaGNOTWpJd09ESTBNRFF5TkRReVdoY05NalF3T0RJegpNRFF5TkRReldqQW1NU1F3SWdZRFZRUUREQnRwYm1keVpYTnpMVzl3WlhKaGRHOXlRREUyTmpFek1UVXdPRE13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURmUVRMQWNFdExNYXdnaVQyU1FOLzcKdEttbm9rN1gwMll0S0NVU3ZRVjBuUWhMZGZ0c3RocGZVdTdqa3lCRXRhSTI4ajlJbXU1L1Q0QU1yUE5vOEhpcgpwNkhLbHB3cDVubjM3REpwUFJ0ZHc3dmd5UmdqRmpxcmR0Vk5VWXJidWFTdEJ3WTl2c2ppNFhvNTR1OHMxWkpGCjNOOWJ2c20rMm15SnZEd0VyNG5OZXZwZDV6WDlqR1g1UU1oS1RhdHFsMTJoaXNVYWNJRTcvaFgzdUoyZXZxcU8KcGtqVU93Ri8yVlNVbkd4Qmg5NHpXRVNUNlI2SVcrUFo5Zlo0d25kRFVzK3laWVZCdmIxRGVaQ3NLQW9EU0l1Ugpnc2pvbzYyKzZqR0Q2bDl1QklQVlE5bnozRFgzeW1FTEdjbzR5TFh0eGlkS282Y0sraUwyZTRMblB2QWFZRVRUCkFnTUJBQUdqUlRCRE1BNEdBMVVkRHdFQi93UUVBd0lDcERBU0JnTlZIUk1CQWY4RUNEQUdBUUgvQWdFQU1CMEcKQTFVZERnUVdCQlFpZWk0SGRuQVc4M1VYdlJtVHZZbGJCeGZpalRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQQpGTjhZRDhFeTNUMVpROGdrS0Y4VFZSVEdwZG5aTmlLUjhtVjZaOEt5WUtyNVpETUJHd29tTXVWZUw3di85NlBOCmtRSUhGUzZ4QzhnSkxrUjFpdkZJZFlHdmxqN3dvbXcxS0p4V0k5SlBGZnVWTzhIbU9NaDNzNCtJUXRyV1QvYlgKSDZMWVlLLzByWXNiMDIzTGtYUTJGdU1iRzNxeWJEcnJCc3crNDlFVmkya3lRbHF2ZExBSUEwNEtKdExuL0RaagpOZEJyc3k4a0tzaEhyOGJpdnpsT2E4V0xDTDBlcTBUb3NrT1hOUUlYMmNXVVRGMlNvcUV3QmF4a2ZzdDUxTUduCjZHM1FrZmhYRDdXYk8rTm5OTmJjbTFxeHhQd2EzdUFBUXo3N3BiMUNJTG91MHVqdWtLM3h5MTYwWlVaVzNGSWIKQWV2ejFxYzJqckJTU1NLYjFSdTNVUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lJSFFXRC9nRU9YWkl3RFFZSktvWklodmNOQVFFTEJRQXdQakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TZ3dKZ1lEVlFRREV4OXJkV0psTFdGd2FYTmxjblpsY2kxc2IyTmhiR2h2YzNRdApjMmxuYm1WeU1CNFhEVEl5TURneU5EQTBNVEF5TVZvWERUTXlNRGd5TVRBME1UQXlNVm93UGpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU2d3SmdZRFZRUURFeDlyZFdKbExXRndhWE5sY25abGNpMXNiMk5oYkdodmMzUXQKYzJsbmJtVnlNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXBpTnNaa3pWYUxTMgpFMjNmVDl4bHFRVGlQdUxnUkU0MDFjNFVsVE9jTXI2UjhqNVo2cnYwbjlvYTEzdzRocFFUZjJNeVl1cmg4WCs5CnliL3diL2wvRVBENzByS09hUHhqbUhJWVpQZXkrZHJaT3BwTm5DMUwzUHZlcVlzYVNYL2NQTm10cFRBbGZEZUkKazNoZE4vaUlreDhSemNFMU1qbWxhWjgxTUN3M1BieWV2d1hHZUV1aEtvRmE5NVhlcnU1UnR1WlAvS2V1UWZCUApoR2h2T1B5d3Q2SWd0OHlrK09LdXd2blU3SEd2dWpjelJEZDVYT1ExbHZ6eUttNXJIMTRXRjM5Y0N6cjFpNnFsCjBlQisrVWdyTXYwaWp1WjQ4Y2dZalE3R0daT2VJMzZhelJqUVFiamtLbU5xRHlvWGJCdkxKZzdqRUFwdnYvVEkKRnMyWjJzMjBBUUlEQVFBQm8wSXdRREFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVNlFkc3ZYbzdDZlhOejEvaW8rWG80TyszRVl3d0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBRUpYamlIMjNJMTlFOGpmcmlvV3hJbWZLTFpaRFFiYVRpSVJaTXZPb3ltWHcyNnhZL2RMTm1JK3doZmsKYjNZWWpDdjFJZWc5S3lFaDNXTlVraytZQ3hQMCtzUkRwaHNmcUNhSWI0RG9NTnR6VTl6RmRoMmYwOWpxR2FDaApCVDM0T1pGQVQ4Um5MTXVrTTYzc1NpV0VVRlhrMWtucXVCSEpBU0NkbzBvM2ZGOUt1VzcwTXhtVlRodFJzaTBpCnAvMC8vVVNFZDhLVXQ4M2YvMzhQTWI5TW13T1RQUjFlR0ZOOW5uek95NjlSUm1aSW5tc3E3cE9qSGZ3eVVVZXQKbzhXMmhoYU9qVnMxWDNrdHA4YTdnaCtRYTZkQ09GQWdPR01xTXREbXAyNTIxbVhzTXdlWm9iQzU2K0NjLzhocgo5MVptN3dyMFYyQ3NWd0dIc2I0NTl0S1ZQRzA9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURURENDQWpTZ0F3SUJBZ0lJSm5nWmVhcENKTGN3RFFZSktvWklodmNOQVFFTEJRQXdSREVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TNHdMQVlEVlFRREV5VnJkV0psTFdGd2FYTmxjblpsY2kxelpYSjJhV05sTFc1bApkSGR2Y21zdGMybG5ibVZ5TUI0WERUSXlNRGd5TkRBME1UQXlNbG9YRFRNeU1EZ3lNVEEwTVRBeU1sb3dSREVTCk1CQUdBMVVFQ3hNSmIzQmxibk5vYVdaME1TNHdMQVlEVlFRREV5VnJkV0psTFdGd2FYTmxjblpsY2kxelpYSjIKYVdObExXNWxkSGR2Y21zdGMybG5ibVZ5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQwpBUUVBM2NnTlF1ajVRbVhXRkxUMFFmTkF0blJxTks5RUlQWFBiSGFQdUJzWjRXV2YrWFdJK3dUUEhLNWlIYTRpCjZCM2xqWnlBUFd4UDhzSmNQbjl3ODRtZGJpd3VzdzIyVXMxdlo2Umw1dVZ0ME1DUGR4YUhqWGo3UThJSEc3UG8KSEJhWTRwOGdhMVFDS2docmpRMHJ2clEwZ2lZejR0cU81cGpFdSt0RmVQR3hnK01DeE03YmFRVUNNYUtQdnRldgpONDBiTVVNYy9raTNTRVV3QnRQelRxTlV0Y08wa0xDeEkyVGdBQ3NpU0daZlVoZ1kxMUxxMTlieEd3MnBQMHFkCnhrMjM1ZFcwTFBTYVZlY3FGelVEbGJqd3VWQXRweHhmRS9uc0ZzTVMvOWdwdjUvM21XdkpIUGc0WUc3M3Q0RWQKUGxhOGJSc3g4WFh2Nk4yN0cveGZMUy9kcHdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRApWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVXpTb3BnMURmOFZacjdXeGZ4Y1JiV1NjWjZIWXdEUVlKCktvWklodmNOQVFFTEJRQURnZ0VCQUZhdzJsV3ZqNTlrc3FTb2xMN2xCYWU5TXkrWUF4S0NOWjhQTGg4SUx5dWoKY3c4a0lDNUt0KzQ4cjNZWWZVcUlNd3B6QTV4cStVNnJNa3FRSnk0aSswbmsrdEtCZkplV3hPTlQ0Ui9xVVNEOQpKVFN5R1dwK2hTYkF2alRieHNKZnRrMjFkTFVSOXdCNGluZWdsbXkrQ1FyeFFzaGJMb1VCVVVtSmhjYTd3VmFHClZjK2FRdmZPZ3lGc2VQVFQrbVNFWWYwTEJEWnUzR2xMQnMvRElCTVAyNHYzV2hEZDRXeW4yd0RzSWdEd0c4UVMKZlFmaTFZdkE2T3c0NWsyb2dFODluM3kyUW5LMHVDNDlDeGt5Mnhmc3RTdmpoWWdEOGdwZkxhMU1wL2d2VVpRQgpGclg3T2NVTWMyTzFqeDlGam51NllxSEFzeUZpc2h6Um05K3pjRXVWaG4wPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlETWpDQ0FocWdBd0lCQWdJSVhCY0tZcytqbFdFd0RRWUpLb1pJaHZjTkFRRUxCUUF3TnpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU0V3SHdZRFZRUURFeGhyZFdKbExXRndhWE5sY25abGNpMXNZaTF6YVdkdVpYSXcKSGhjTk1qSXdPREkwTURReE1ESXlXaGNOTXpJd09ESXhNRFF4TURJeVdqQTNNUkl3RUFZRFZRUUxFd2x2Y0dWdQpjMmhwWm5ReElUQWZCZ05WQkFNVEdHdDFZbVV0WVhCcGMyVnlkbVZ5TFd4aUxYTnBaMjVsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1PSGROaUZ3SnR6LzNXeVFvcUM2d1JjR3FnazRpTHQKSDNOYm15LzE1L1AvVVA5cWZvMDlYOTBGRUF4K043cUR2a29nd3BTTEZPWjVRMFdQYzBjVS9LVENkbWdhNW1YWgpsdFR6L1pDTWNGODN2OVdIMHdGcGtXUXBVRzE1OVZQdmNXTHpjTDJjeVJJb21SZld1R21Ca3FIUXNKdGNtV0s4Cmc2VjIzYncyc2FyR093ejgrVU9zNm44bDBReFJqSWovRmJJa3RUclk5VU1LQldPZlE5ODAweXdBd1pJU3FmbU8KVWZBa0tGUUwxR29aRXZaMkQ1Q1pjM0hGY1FrYmYwLzNidmMxZUZNYXI1MUNXRGNXNkZIbCtXYVhsTnlLUU1INQpnYmxFT3p4TkpXM2doNHhPdEhzczF6eC9qOFNFU3pxcldrbGR5YkZQT0diQTdQd2gzQTU0QTRVQ0F3RUFBYU5DCk1FQXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKejQKekVMNkJMTTRXb001UDFSSDBBOWtWM210TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFDQXQ3ZUJRNXVuY21rTwp0SThPV251aGFxclpFbmtIcGdOelRYL1BWM0JXS2xaUlY1VTdYZmhjUkNXcUw1QitrVGJ1d0d2cytWV3lQKzZGCndRWWgrZndLTVcwYjA0V2pJa3QvOVFyV3V2dXU3YUdKZG5VNW0wbzJoWW9oU0JFUzVwUU9vbXJUR09MVXVDSWcKWmZEOHRGbXJZaGc2QWM3WWFJcUZWRldGcFgycCs3SlBZT1FLNnlaaWhMRnFVQTRYRU5hQ0taekFGcFVBdUQ3MgpxSUlTM2F2cGp1dStZdy9zSzFpZE5FZ1dLdHlYR2ZkTFZ5WldTZmJ5Vk5lcklIYWNBVVRzeldyTHQ4RmNiNGROClB4b0ErVmhycUZ3MFY4RHpiM1Y2bjJ6VWpYSmJmUjRlWW9SUjVKMlVrU0IzRC8xdTVGMllVRytROUhYRkxYQTMKOHhsV0lOdisKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://api.crc.testing:6443
  name: crc
contexts:
- context:
    cluster: crc
    user: admin
  name: admin
current-context: admin
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpRENDQW5DZ0F3SUJBZ0lJR1VhVllvMUw0b1F3RFFZSktvWklodmNOQVFFTEJRQXdQVEVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TY3dKUVlEVlFRREV4NWhaRzFwYmkxcmRXSmxZMjl1Wm1sbkxYTnBaMjVsY2kxagpkWE4wYjIwd0hoY05Nakl3T1RFeU1qQXdOVE0xV2hjTk16SXdPVEE1TWpBd05UTTFXakF3TVJjd0ZRWURWUVFMCkV3NXplWE4wWlcwNmJXRnpkR1Z5Y3pFVk1CTUdBMVVFQXhNTWMzbHpkR1Z0T21Ga2JXbHVNSUlCSWpBTkJna3EKaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF5VG54bHNHci9POExIWk1DUFEvcUd0YW8wMEtrZmRFbgoydC9wa0JsSVhSVEhoL2IxV2ptWTZ5VzZIdGRZTFFaRlNmczdRU0MvY0pqN2c2YTdHc2htbUtjWHh4ZGEwa3lZCkhKVG5vZ3lPRGVLVzVXVWYwYy9CL0pGSjhYZGJuQ3crZXd3NjBzMmxESUxDYTVOZjN3M2JKSm1EM2dQS3l0MlkKQ2g3dHNueWlsV0RrRFZoMjRRY09BcDlzeC9pMHF6M2x5b2tlcktZTXRFeThKaW1EaGxNMjlrWkZ6SDNSMy93NApkRHlLbXBUNUVIajZ1ajUzSEhwNnVZMk0rbUtjT01HU3liMjJwRFB6SXZueFNZZmZiNWcxQ3RzR2I5bXRJc2RJCmFIdVpUZ28rWjFRTUlTUzVRMSswV2NxbjJIZDErRzZ2T04vU3lpZjV1UDV1VzhQeGxxTzhGUUlEQVFBQm80R1kKTUlHVk1BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSApBd0l3REFZRFZSMFRBUUgvQkFJd0FEQXBCZ05WSFE0RUlnUWc4bEZrNHhHd1JkYStsdDBzV1FGOEJueHVvcEdmClhEamU2OE5UQTVhazdCZ3dLd1lEVlIwakJDUXdJb0FnQnQzN21xZEE1bkZIMUxrWkFiMG0wanIxQXo5eUVmYTgKQU8yL3ZTMnVSdUl3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJrRjNydHR6Mmg4RElUd2JzL0luQmVsNkN2dApBK2tRSzkwbWJFTkY4OWp4SnM4dksxczZGSlhpVXdKVk9ZbU5meTczY2dTYmp5TnljcmFrVUhXS05mMDNkSW1yCmpXdVZHL0U4R0xkUEhZTDY0KzhwcW1uU2NvTG0rRFFSU3FROTBuQllRZUs4YnJsVGY0SG9KQkpUeXVGZGFjaUkKUzRXK3F5SEZsb3JZWGNBWEFxWVZQOHdOVmtXbXBqK0w5bEtaZGxmaFMyK2wwSzU2NjVNcUU1WUQzUm4xckNxUQpQY2JkNHhLL2J3KzJpRFFsMXdOK1N3dm9oS2JBcE1aOG1NdmpuMy9kTDBzMTQyMVgvc2crT3hoY2xXeWpRMC96CmFxN3orQk14ekh5RnlBQ0o1SVlqYlFJVTJvbEp1bmtEOHRyRDdCRTIvaXY0emdJdnM1UlU5eXRQTlFnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: REDACTED

resolv.conf on linux machine (crc host)

linux /etc/resolv.conf 
# resolv.conf(5) file generated by tailscale
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
search TAILSCALE_USERNAME_REDACTED.beta.tailscale.net apps-crc.testing api.crc.testing

macos dns config

mac /etc/resolv.conf ``` # # macOS Notice # # This file is not consulted for DNS hostname resolution, address # resolution, or the DNS query routing mechanism used by most # processes on this system. # # To view the DNS configuration used by this system, use: # scutil --dns # # SEE ALSO # dns-sd(1), scutil(8) # # This file is automatically generated. # search TAILSCALE_USERNAME_REDACTED.beta.tailscale.net apps-crc.testing api.crc.testing ISP_DNS_DOMAIN_REDACTED nameserver 100.100.100.100 ```
scutil --dns 
mac$ scutil --dns
DNS configuration

resolver #1
  search domain[0] : TAILSCALE_USERNAME_REDACTED.beta.tailscale.net
  search domain[1] : apps-crc.testing
  search domain[2] : api.crc.testing
  search domain[3] : ISP_DNS_DOMAIN_REDACTED
  nameserver[0] : 100.100.100.100
  if_index : 23 (utun5)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102600

resolver #2
  nameserver[0] : 192.168.1.1
  if_index : 13 (en0)
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 200000

resolver #3
  domain   : TAILSCALE_USERNAME_REDACTED.beta.tailscale.net.
  nameserver[0] : 100.100.100.100
  if_index : 23 (utun5)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102601

resolver #4
  domain   : apps-crc.testing.
  nameserver[0] : 100.100.100.100
  if_index : 23 (utun5)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102602

resolver #5
  domain   : api.crc.testing.
  nameserver[0] : 100.100.100.100
  if_index : 23 (utun5)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102603

resolver #6
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #7
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #8
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #9
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #10
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #11
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : ISP_DNS_DOMAIN_REDACTED
  nameserver[0] : 192.168.1.1
  if_index : 13 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : TAILSCALE_USERNAME_REDACTED.beta.tailscale.net
  search domain[1] : apps-crc.testing
  search domain[2] : api.crc.testing
  nameserver[0] : 100.100.100.100
  if_index : 23 (utun5)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)

dnsmasq on linux-dns

/etc/dnsmasq.conf 
# If you don't want dnsmasq to read /etc/resolv.conf or any other
# file, getting its servers from this file instead (see below), then
# uncomment this.
no-resolv

server=192.168.1.1

# If you want dnsmasq to change uid and gid to something other
# than the default, edit the following lines.
user=dnsmasq
group=dnsmasq

# If you want dnsmasq to listen for DHCP and DNS requests only on
# specified interfaces (and the loopback) give the name of the
# interface (eg eth0) here.
# Repeat the line for more than one interface.
#interface=
# Listen only on localhost by default
#interface=lo
#interface=tailscale0
# Or you can specify which interface _not_ to listen on
#except-interface=
# Or which to listen on by address (remember to include 127.0.0.1 if
# you use this.)
listen-address=127.0.0.1
listen-address=192.168.1.63
listen-address=192.168.1.220
listen-address=TAILSCALE_IP_REDACTED

# Include all files in /etc/dnsmasq.d except RPM backup files
conf-dir=/etc/dnsmasq.d,.rpmnew,.rpmsave,.rpmorig
/etc/dnsmasq.d/crc.conf 
# tailscale
address=/apps-crc.testing/TAILSCALE_IP_REDACTED
address=/api.crc.testing/TAILSCALE_IP_REDACTED

haproxy on linux

/etc/haproxy/haproxy.cfg 
global
    log /dev/log local0

defaults
    balance roundrobin
    log global
    maxconn 100
    mode tcp
    timeout connect 5s
    timeout client 500s
    timeout server 500s

listen apps
    bind 0.0.0.0:80
    server crcvm 192.168.130.11:80 check

listen apps_ssl
    bind 0.0.0.0:443
    server crcvm 192.168.130.11:443 check

listen api
    bind 0.0.0.0:6443
    server crcvm 192.168.130.11:6443 check

Steps to reproduce

Note: these steps are simplified since it started happening locally on macOS too.
I can rewrite this issue to reflect that, but it's a lot of work to expose the same issue.

  1. Set up with crc setup
  2. Log in with the login command provided by crc console --credentials

Expected behavior: I am successfully logged in.
Actual behavior: x509 errors.

oc login on linux

linux$ oc login -u kubeadmin -p PASSWORD_REDACTED https://api.crc.testing:6443
Login successful.

You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
oc login on mac 
mac$ rm ~/.kube/config
mac$ oc login -u kubeadmin  https://api.crc.testing:6443 -p PASSWORD_REDACTED --loglevel=9 --insecure-skip-tls-verify
I0912 16:57:48.582512   23120 round_trippers.go:466] curl -v -XHEAD  'https://api.crc.testing:6443/'
I0912 16:57:53.592736   23120 round_trippers.go:495] HTTP Trace: DNS Lookup for api.crc.testing resolved to [{TAILSCALE_IP_REDACTED }]
I0912 16:57:53.597938   23120 round_trippers.go:510] HTTP Trace: Dial to tcp:TAILSCALE_IP_REDACTED:6443 succeed
I0912 16:57:53.619555   23120 round_trippers.go:553] HEAD https://api.crc.testing:6443/  in 5036 milliseconds
I0912 16:57:53.619593   23120 round_trippers.go:570] HTTP Statistics: DNSLookup 5009 ms Dial 5 ms TLSHandshake 20 ms Duration 5036 ms
I0912 16:57:53.619603   23120 round_trippers.go:577] Response Headers:
error: x509: “kube-apiserver-lb-signer” certificate is not trusted
mac$ scp linux:~/.crc/machines/crc/kubeconfig ~/.kube/config # attempt to use certificate information from here
kubeconfig
mac$ oc login -u kubeadmin  https://api.crc.testing:6443 -p PASSWORD_REDACTED --loglevel=9 --insecure-skip-tls-verify
I0912 16:59:35.036954   23283 loader.go:372] Config loaded from file:  /Users/kevin/.kube/config
I0912 16:59:35.037341   23283 round_trippers.go:466] curl -v -XHEAD  'https://api.crc.testing:6443/'
I0912 16:59:35.047021   23283 round_trippers.go:495] HTTP Trace: DNS Lookup for api.crc.testing resolved to [{TAILSCALE_IP_REDACTED }]
I0912 16:59:35.050609   23283 round_trippers.go:510] HTTP Trace: Dial to tcp:TAILSCALE_IP_REDACTED:6443 succeed
I0912 16:59:35.067353   23283 round_trippers.go:553] HEAD https://api.crc.testing:6443/ 403 Forbidden in 29 milliseconds
I0912 16:59:35.067380   23283 round_trippers.go:570] HTTP Statistics: DNSLookup 9 ms Dial 3 ms TLSHandshake 7 ms ServerProcessing 8 ms Duration 29 ms
I0912 16:59:35.067386   23283 round_trippers.go:577] Response Headers:
I0912 16:59:35.067429   23283 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0912 16:59:35.067437   23283 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 8e4390e2-7947-48f1-b7a5-aca69ea99dbc
I0912 16:59:35.067442   23283 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: bfdd957e-32ce-4b55-84aa-209ed9a9ef83
I0912 16:59:35.067446   23283 round_trippers.go:580]     Content-Length: 186
I0912 16:59:35.067450   23283 round_trippers.go:580]     Date: Mon, 12 Sep 2022 20:59:35 GMT
I0912 16:59:35.067454   23283 round_trippers.go:580]     Audit-Id: 066a7292-eaf6-480d-ba77-5f8f360bdb4a
I0912 16:59:35.067458   23283 round_trippers.go:580]     Cache-Control: no-cache, private
I0912 16:59:35.067462   23283 round_trippers.go:580]     Content-Type: application/json
I0912 16:59:35.067725   23283 round_trippers.go:466] curl -v -XGET  -H "X-Csrf-Token: 1" 'https://api.crc.testing:6443/.well-known/oauth-authorization-server'
I0912 16:59:35.070964   23283 round_trippers.go:553] GET https://api.crc.testing:6443/.well-known/oauth-authorization-server 200 OK in 3 milliseconds
I0912 16:59:35.070987   23283 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 3 ms
I0912 16:59:35.070992   23283 round_trippers.go:577] Response Headers:
I0912 16:59:35.070998   23283 round_trippers.go:580]     Content-Type: application/json
I0912 16:59:35.071003   23283 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 8e4390e2-7947-48f1-b7a5-aca69ea99dbc
I0912 16:59:35.071007   23283 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: bfdd957e-32ce-4b55-84aa-209ed9a9ef83
I0912 16:59:35.071011   23283 round_trippers.go:580]     Content-Length: 573
I0912 16:59:35.071015   23283 round_trippers.go:580]     Date: Mon, 12 Sep 2022 20:59:35 GMT
I0912 16:59:35.071019   23283 round_trippers.go:580]     Audit-Id: bce7f502-78f2-4747-a3d1-bf137b79daa8
I0912 16:59:35.071023   23283 round_trippers.go:580]     Cache-Control: no-cache, private
I0912 16:59:35.108001   23283 request_token.go:477] unexpected error during system roots probe: x509: “ingress-operator@1661315083” certificate is not trusted
I0912 16:59:35.108355   23283 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.11.0 (darwin/amd64) kubernetes/fcf512e" 'https://api.crc.testing:6443/api/v1/namespaces/openshift/configmaps/motd'
I0912 16:59:35.112550   23283 round_trippers.go:553] GET https://api.crc.testing:6443/api/v1/namespaces/openshift/configmaps/motd 403 Forbidden in 4 milliseconds
I0912 16:59:35.112575   23283 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 4 ms Duration 4 ms
I0912 16:59:35.112580   23283 round_trippers.go:577] Response Headers:
I0912 16:59:35.112587   23283 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0912 16:59:35.112591   23283 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 8e4390e2-7947-48f1-b7a5-aca69ea99dbc
I0912 16:59:35.112595   23283 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: bfdd957e-32ce-4b55-84aa-209ed9a9ef83
I0912 16:59:35.112599   23283 round_trippers.go:580]     Content-Length: 303
I0912 16:59:35.112603   23283 round_trippers.go:580]     Date: Mon, 12 Sep 2022 20:59:35 GMT
I0912 16:59:35.112607   23283 round_trippers.go:580]     Audit-Id: df94edd2-1cf5-411e-8195-b16505313f30
I0912 16:59:35.112611   23283 round_trippers.go:580]     Cache-Control: no-cache, private
I0912 16:59:35.112990   23283 round_trippers.go:580]     Content-Type: application/json
I0912 16:59:35.113146   23283 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403}
error: x509: “ingress-operator@1661315083” certificate is not trusted

Logs

Before beginning, I ran crc delete -f; rm -rf ~/.crc/, and downloaded a fresh version of crc.

crc setup

linux$ crc setup --log-level 9 
Successfully configured consent-telemetry to yes
Successfully configured pull-secret-file to /home/kevin/crc_pull_secret
Changes to configuration property 'memory' are only applied when the CRC instance is started.
If you already have a running CRC instance, then for this configuration change to take effect, stop the CRC instance with 'crc stop' and restart it with 'crc start'.
Changes to configuration property 'disk-size' are only applied when the CRC instance is started.
If you already have a running CRC instance, then for this configuration change to take effect, stop the CRC instance with 'crc stop' and restart it with 'crc start'.
�[36mINFO�[0m Using bundle path /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64.crcbundle 
�[36mINFO�[0m Checking if running as non-root              
�[36mINFO�[0m Checking if running inside WSL2              
�[36mINFO�[0m Checking if crc-admin-helper executable is cached 
�[36mINFO�[0m Caching crc-admin-helper executable          
�[36mINFO�[0m Using root access: Changing ownership of /home/kevin/.crc/bin/crc-admin-helper-linux 
[sudo] password for kevin: 
�[36mINFO�[0m Using root access: Setting suid for /home/kevin/.crc/bin/crc-admin-helper-linux 
�[36mINFO�[0m Checking for obsolete admin-helper executable 
�[36mINFO�[0m Checking if running on a supported CPU architecture 
�[36mINFO�[0m Checking minimum RAM requirements            
�[36mINFO�[0m Checking if crc executable symlink exists    
�[36mINFO�[0m Creating symlink for crc executable          
�[36mINFO�[0m Checking if Virtualization is enabled        
�[36mINFO�[0m Checking if KVM is enabled                   
�[36mINFO�[0m Checking if libvirt is installed             
�[36mINFO�[0m Checking if user is part of libvirt group    
�[36mINFO�[0m Checking if active user/process is currently part of the libvirt group 
�[36mINFO�[0m Checking if libvirt daemon is running        
�[36mINFO�[0m Checking if a supported libvirt version is installed 
�[36mINFO�[0m Checking if crc-driver-libvirt is installed  
�[36mINFO�[0m Installing crc-driver-libvirt                
�[36mINFO�[0m Checking crc daemon systemd service          
�[36mINFO�[0m Checking crc daemon systemd socket units     
�[36mINFO�[0m Checking if systemd-networkd is running      
�[36mINFO�[0m Checking if NetworkManager is installed      
�[36mINFO�[0m Checking if NetworkManager service is running 
�[36mINFO�[0m Checking if dnsmasq configurations file exist for NetworkManager 
�[36mINFO�[0m Checking if the systemd-resolved service is running 
�[36mINFO�[0m Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
�[36mINFO�[0m Checking if libvirt 'crc' network is available 
�[36mINFO�[0m Checking if libvirt 'crc' network is active  
�[36mINFO�[0m Checking if CRC bundle is extracted in '$HOME/.crc' 
�[36mINFO�[0m Checking if /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64.crcbundle exists 
�[36mINFO�[0m Getting bundle for the CRC executable        
�[36mINFO�[0m Downloading crc_libvirt_4.11.1_amd64.crcbundle 

(progress bar redacted)

�[36mINFO�[0m Uncompressing /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64.crcbundle 

(progress bar redacted)

Your system is correctly setup for using CRC. Use 'crc start' to start the instance

crc start

linux$ crc start --log-level=9 
level=info msg="Checking if running as non-root"
level=info msg="Checking if running inside WSL2"
level=info msg="Checking if crc-admin-helper executable is cached"
level=info msg="Checking for obsolete admin-helper executable"
level=info msg="Checking if running on a supported CPU architecture"
level=info msg="Checking minimum RAM requirements"
level=info msg="Checking if crc executable symlink exists"
level=info msg="Checking if Virtualization is enabled"
level=info msg="Checking if KVM is enabled"
level=info msg="Checking if libvirt is installed"
level=info msg="Checking if user is part of libvirt group"
level=info msg="Checking if active user/process is currently part of the libvirt group"
level=info msg="Checking if libvirt daemon is running"
level=info msg="Checking if a supported libvirt version is installed"
level=info msg="Checking if crc-driver-libvirt is installed"
level=info msg="Checking crc daemon systemd socket units"
level=info msg="Checking if systemd-networkd is running"
level=info msg="Checking if NetworkManager is installed"
level=info msg="Checking if NetworkManager service is running"
level=info msg="Checking if dnsmasq configurations file exist for NetworkManager"
level=info msg="Checking if the systemd-resolved service is running"
level=info msg="Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists"
level=info msg="Checking if libvirt 'crc' network is available"
level=info msg="Checking if libvirt 'crc' network is active"
level=info msg="Loading bundle: crc_libvirt_4.11.1_amd64..."
level=info msg="Creating CRC VM for openshift 4.11.1..."
level=info msg="Generating new SSH key pair..."
level=info msg="Generating new password for the kubeadmin user"
level=info msg="Starting CRC VM for openshift 4.11.1..."
level=info msg="CRC instance is running with IP 192.168.130.11"
level=info msg="CRC VM is running"
level=info msg="Updating authorized keys..."
level=info msg="Resizing /dev/vda4 filesystem"
level=info msg="Configuring shared directories"
level=info msg="Check internal and public DNS query..."
level=info msg="Check DNS query from host..."
level=info msg="Verifying validity of the kubelet certificates..."
level=info msg="Starting kubelet service"
level=info msg="Waiting for kube-apiserver availability... [takes around 2min]"
level=info msg="Adding user's pull secret to the cluster..."
level=info msg="Updating SSH key to machine config resource..."
level=info msg="Waiting for user's pull secret part of instance disk..."
level=info msg="Changing the password for the kubeadmin user"
level=info msg="Updating cluster ID..."
level=info msg="Updating root CA cert to admin-kubeconfig-client-ca configmap..."
level=info msg="Starting openshift instance... [waiting for the cluster to stabilize]"
level=info msg="3 operators are progressing: image-registry, network, openshift-controller-manager"
level=info msg="2 operators are progressing: image-registry, openshift-controller-manager"
level=info msg="All operators are available. Ensuring stability..."
level=info msg="2 operators are progressing: kube-apiserver, openshift-controller-manager"
level=info msg="Operator kube-apiserver is progressing"
level=info msg="Operator kube-apiserver is progressing"
level=info msg="Operator authentication is not yet available"
level=info msg="Operator authentication is not yet available"
level=info msg="Operator authentication is not yet available"
level=info msg="Operator authentication is not yet available"
level=error msg="Cluster is not ready: cluster operators are still not stable after 10m1.49983431s"
level=info msg="Adding crc-admin and crc-developer contexts to kubeconfig..."
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: REDACTED

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval (crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
@KevinMGranger KevinMGranger added kind/bug Something isn't working status/need triage labels Oct 4, 2022
@KevinMGranger
Copy link
Author

I'm happy to talk in Google Chat if that's easier. I'd love to help solve this for others if they're experiencing it too.

@cfergeau
Copy link
Contributor

cfergeau commented Oct 4, 2022 •
edited
Loading

This is fixed by crc-org/snc#578
The fix was too late for the 2.8.0 release, but should be in the 2.9.0 one.

@KevinMGranger
Copy link
Author

KevinMGranger commented Oct 4, 2022 •
edited
Loading

I just tried it with 2.9.0 directly on macOS and the issue is still present. I'll try to change my writeup, it was just a lot of work collecting that the first time. I wish I had automated it 😅

Interesting that the issue was with the client though. I guess that's a good workaround.

I know it's not strictly relevant to CRC, but if you know what certs I'd need to export from the cluster, I'd be happy to manually import and trust them until there's a fix.

@KevinMGranger
Copy link
Author

Oh, it also looks like the fix wasn't in the release, or isn't working?

$ uname -a; crc version
Darwin m1a1 21.6.0 Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000 arm64
CRC version: 2.9.0+589ab2cd
OpenShift version: 4.11.3
Podman version: 4.2.0

@cfergeau
Copy link
Contributor

cfergeau commented Oct 4, 2022

Can you check ~/.crc/bin/oc/oc version?

@KevinMGranger
Copy link
Author

Ah! I thought crc-org/snc#578 was talking about downgrading the cluster, not the client.

The dev env setup we have for our project automatically downloads the latest. I knew it was 4.11, and was about to manually downgrade. I'll use the version that came with CRC now. Maybe I'll even add a "I'm using CRC" option to our setup script.

Thank you for helping with this, this has been bothering me for a long time.

This should definitely be in the "Known Issues" section for CRC, since many folks won't read the openshift release notes in addition to them. But it looks like the release notes / docs haven't been updated for a few releases?

@KevinMGranger
Copy link
Author

Looks like anyone using crc properly won't hit this, so it's already fixed :)

@cfergeau cfergeau mentioned this issue Dec 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working status/need triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cfergeau @KevinMGranger