[BUG] unable to establish ssh connection - command results: handshake failed: read tcp 127.0.0.1:62297->127.0.0.1:2222: read: connection reset by peer #3410
Labels
kind/bug
Something isn't working
os/macos
status/need more information
Issue needs more information before it will be looked at
Comments
ahaffar
changed the title
|
@ahaffar did it exit after retries automatically because I can see you did |
praveenkumar
changed the title
|
@praveenkumar i tried multiple times, i did
|
@ahaffar Now we don't add Are you using |
|
I stopped docker by running haffar@Ahmeds-MacBook-Pro ~ % crc start --log-level debug
DEBU CRC version: 2.10.2+065f0741
DEBU OpenShift version: 4.11.7
DEBU Podman version: 4.2.0
DEBU Running 'crc start'
DEBU Total memory of system is 17179869184 bytes
DEBU No new version available. The latest version is 2.10.2
DEBU Checking file: /Users/haffar/.crc/machines/crc/.crc-exist
DEBU Checking file: /Users/haffar/.crc/machines/crc/.crc-exist
INFO Loading bundle: crc_vfkit_4.11.7_arm64...
DEBU Checking file: /Users/haffar/.crc/machines/crc/.crc-exist
INFO A CRC VM for OpenShift 4.11.7 is already running
Started the OpenShift cluster.
The server is accessible via web console at:
https://console-openshift-console.apps-crc.testing
Log in as administrator:
Username: kubeadmin
Password: wieNw-aRjR6-CwHKJ-KuHRe
Log in as user:
Username: developer
Password: developer
Use the 'oc' command line interface:
$ eval $(crc oc-env)
$ oc login -u developer https://api.crc.testing:6443
DEBU Running 'sw_vers -productVersion'
haffar@Ahmeds-MacBook-Pro ~ % crc status --log-level debug
DEBU CRC version: 2.10.2+065f0741
DEBU OpenShift version: 4.11.7
DEBU Podman version: 4.2.0
DEBU Running 'crc status'
CRC VM: Running
OpenShift: Unreachable (v4.11.7)
RAM Usage: 0B of 0B
Disk Usage: 0B of 0B (Inside the CRC VM)
Cache Usage: 36.71GB
Cache Directory: /Users/haffar/.crc/cache
DEBU Running 'sw_vers -productVersion'
haffar@Ahmeds-MacBook-Pro ~ % oc login -u developer https://api.crc.testing:6443
error: dial tcp: lookup api.crc.testing on 192.168.8.1:53: no such host - verify you have provided the correct host and port and that the server is currently running.
haffar@Ahmeds-MacBook-Pro ~ % |
Run
|
|
|
|
thank you @cfergeau haffar@Ahmeds-MacBook-Pro ~ % crc delete
Do you want to delete the instance? [y/N]: y
Deleted the instance
haffar@Ahmeds-MacBook-Pro ~ % crc start --log-level debug
DEBU CRC version: 2.10.2+065f0741
DEBU OpenShift version: 4.11.7
DEBU Podman version: 4.2.0
DEBU Running 'crc start'
DEBU Total memory of system is 17179869184 bytes
DEBU No new version available. The latest version is 2.10.2
DEBU Checking file: /Users/haffar/.crc/machines/crc/.crc-exist
INFO Checking if running as non-root
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
DEBU Checking if an older admin-helper executable is installed
DEBU No older admin-helper executable found
INFO Checking if running on a supported CPU architecture
DEBU GOARCH is arm64 GOOS is darwin
INFO Checking minimum RAM requirements
DEBU Total memory of system is 17179869184 bytes
INFO Checking if crc executable symlink exists
INFO Checking if running emulated on a M1 CPU
INFO Checking if vfkit is installed
INFO Checking if old launchd config for tray and/or daemon exists
INFO Checking if crc daemon plist file is present and loaded
DEBU Running 'bash -c launchctl list | grep com.redhat.crc.daemon | awk '{print $1}''
DEBU Checking file: /Users/haffar/.crc/machines/crc/.crc-exist
INFO Loading bundle: crc_vfkit_4.11.7_arm64...
DEBU Cannot load secret from configuration: empty path
DEBU Using secret from keyring
INFO Creating CRC VM for openshift 4.11.7...
DEBU Running pre-create checks...
DEBU Creating machine...
DEBU /Users/haffar/.crc/machines/crc/crc.img is already 33285996544 bytes
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
DEBU Created /Users/haffar/.crc/machines/crc/.crc-exist
DEBU Machine successfully created
DEBU Checking file: /Users/haffar/.crc/machines/crc/.crc-exist
INFO Starting CRC VM for openshift 4.11.7...
DEBU Updating CRC VM configuration
DEBU Running 'sw_vers -productVersion'
DEBU Running /Applications/Red Hat OpenShift Local.app/Contents/Resources/vfkit --cpus 4 --memory 13000 --kernel /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/vmlinuz-5.14.0-70.26.1.el9_0.aarch64 --initrd /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/initramfs-5.14.0-70.26.1.el9_0.aarch64.img --kernel-cmdline console=hvc0 BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-caefb4ee863008487ed96ac7f0ea85354ea877e6f9636f87ad1b10c20102aea8/vmlinuz-5.14.0-70.26.1.el9_0.aarch64 random.trust_cpu=on ignition.platform.id=qemu ostree=/ostree/boot.0/rhcos/caefb4ee863008487ed96ac7f0ea85354ea877e6f9636f87ad1b10c20102aea8/0 root=UUID=3873ab67-a779-4b5a-9a47-6d12eea58372 rw rootflags=prjquota boot=UUID=49471370-5a0a-4a7f-ab51-c3b1a17380fa --device virtio-serial,logFilePath=/Users/haffar/.crc/machines/crc/vfkit.log --device virtio-rng --device virtio-blk,path=/Users/haffar/.crc/machines/crc/crc.img --device virtio-vsock,port=1024,socketURL=/Users/haffar/.crc/tap.sock
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="&{4 13000 /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/vmlinuz-5.14.0-70.26.1.el9_0.aarch64 console=hvc0 BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-caefb4ee863008487ed96ac7f0ea85354ea877e6f9636f87ad1b10c20102aea8/vmlinuz-5.14.0-70.26.1.el9_0.aarch64 random.trust_cpu=on ignition.platform.id=qemu ostree=/ostree/boot.0/rhcos/caefb4ee863008487ed96ac7f0ea85354ea877e6f9636f87ad1b10c20102aea8/0 root=UUID=3873ab67-a779-4b5a-9a47-6d12eea58372 rw rootflags=prjquota boot=UUID=49471370-5a0a-4a7f-ab51-c3b1a17380fa /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/initramfs-5.14.0-70.26.1.el9_0.aarch64.img [virtio-serial,logFilePath=/Users/haffar/.crc/machines/crc/vfkit.log virtio-rng virtio-blk,path=/Users/haffar/.crc/machines/crc/crc.img virtio-vsock,port=1024,socketURL=/Users/haffar/.crc/tap.sock]}"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="boot parameters:"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="\tkernel: /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/vmlinuz-5.14.0-70.26.1.el9_0.aarch64"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="\tkernel command line:console=hvc0 BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-caefb4ee863008487ed96ac7f0ea85354ea877e6f9636f87ad1b10c20102aea8/vmlinuz-5.14.0-70.26.1.el9_0.aarch64 random.trust_cpu=on ignition.platform.id=qemu ostree=/ostree/boot.0/rhcos/caefb4ee863008487ed96ac7f0ea85354ea877e6f9636f87ad1b10c20102aea8/0 root=UUID=3873ab67-a779-4b5a-9a47-6d12eea58372 rw rootflags=prjquota boot=UUID=49471370-5a0a-4a7f-ab51-c3b1a17380fa"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="\tinitrd: /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/initramfs-5.14.0-70.26.1.el9_0.aarch64.img"
DEBU time="2022-11-11T14:28:29+03:00" level=info
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="virtual machine parameters:"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="\tvCPUs: 4"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="\tmemory: 13000 MiB"
DEBU time="2022-11-11T14:28:29+03:00" level=info
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="Adding virtio-serial device (logFile: /Users/haffar/.crc/machines/crc/vfkit.log)"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="Adding virtio-rng device"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="Adding virtio-blk device (imagePath: /Users/haffar/.crc/machines/crc/crc.img)"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="Adding virtio-vsock device"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="virtual machine is running"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="Exposing vsock port 1024 on /Users/haffar/.crc/tap.sock"
DEBU time="2022-11-11T14:28:29+03:00" level=info msg="waiting for VM to stop"
DEBU Waiting for machine to be running, this may take a few minutes...
DEBU retry loop: attempt 0
DEBU Machine is up and running!
INFO CRC instance is running with IP 127.0.0.1
DEBU Waiting until ssh is available
DEBU retry loop: attempt 0
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/haffar/.crc/machines/crc/id_ecdsa /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:64466->127.0.0.1:2222: read: connection reset by peer, output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err : ssh: handshake failed: read tcp 127.0.0.1:64466->127.0.0.1:2222: read: connection reset by peer
- sleeping 1s
DEBU retry loop: attempt 1
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/haffar/.crc/machines/crc/id_ecdsa /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:64467->127.0.0.1:2222: read: connection reset by peer, output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err : ssh: handshake failed: read tcp 127.0.0.1:64467->127.0.0.1:2222: read: connection reset by peer
- sleeping 1s
DEBU retry loop: attempt 2
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/haffar/.crc/machines/crc/id_ecdsa /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:64468->127.0.0.1:2222: read: connection reset by peer, output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err : ssh: handshake failed: read tcp 127.0.0.1:64468->127.0.0.1:2222: read: connection reset by peer
- sleeping 1s
DEBU retry loop: attempt 3
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/haffar/.crc/machines/crc/id_ecdsa /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:64469->127.0.0.1:2222: read: connection reset by peer, output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err : ssh: handshake failed: read tcp 127.0.0.1:64469->127.0.0.1:2222: read: connection reset by peer
- sleeping 1s
DEBU retry loop: attempt 4
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [/Users/haffar/.crc/machines/crc/id_ecdsa /Users/haffar/.crc/cache/crc_vfkit_4.11.7_arm64/id_ecdsa_crc]
^C
haffar@Ahmeds-MacBook-Pro ~ % crc config set network-mode system
Configuration property 'network-mode' does not exist
haffar@Ahmeds-MacBook-Pro ~ %
|
|
I faced with similiar issue too: |
|
$ crc config view
|
|
@AndrienkoAleksandr do you have M1 chip |
@ahaffar No, I'm using macbook 2019 with intel cpu, so amd64 arc I think. |
|
From time to time I recreate my crc instance due this bug. New instance works fine some time, for example few days. I can stop crc instance and start it again and I can't reproduce this bug. But seems after some long laptop sleep, when I closed laptop, it reproduces. So I can't provide stable reproduction steps... Because sometimes after laptop sleep crc instance auto-restarts fine, sometimes doesn't. And when I see this error: I don't know how to recovery my instance, so I remove it and create new one... Stop and start again doesn't help. Restart laptop doesn't help too. |
|
Hi, has anybody came up with the solution? I'm facing the same issue on mac with M1 chip. Results of Results of I added |
|
I think killing the crc daemon would help when this happens. Kill all |
|
still get this error on mac m1. |
|
The same error on M1 with the latest version: killing crc daemon or restarting laptop didn't help |
|
Does this always happen? Does this happen after using crc for a while as stated in this report? |
|
crc config view crc setup --log-level debug crc start --log-level debug ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null [email protected] -p 2222 ~/.crc/machines/crc/vfkit.log is empty |
|
Is the crc daemon running? I think it is from the |
|
ps aux | grep crc |
|
The vfkit command line matches what I get on my system, however it's odd that you don't have anything in in case this goes further? The objective is to get something in |
It's stuck here. |
|
It's not printing to stdout/stderr by default, but to |
|
I used exactly the command you gave. What do you mean by " |
|
|
no errors from console. |
|
Same issue here |
|
Can you describe more related to your network connection, such as VPN or tools like snitch? |
gbraad
changed the title
gbraad
changed the title
gbraad
changed the title
gbraad
added
os/macos
status/need more information
|
Hi @gbraad log for |
|
Maybe help: I can not open any connection to the server: |
|
Something blocks connectivity. This is a corporate image on your system? Can you access a small website server on localhost like with Python? |
No
Yes, I can, I have tested it with As the virtual environment closes the connection, I think maybe the virtual environment has a problem! |
|
I'd start with |
|
@cfergeau I am also encountering this issue.
I've checked and run it, and the issue goes away temporarily, but soon the same issue occurs. In my case, It is occurring when creating an image with a Dockerfile with a high build load via the Please let me know if you have any information that I should provide. I like this tool and would be happy to provide more information. crc version crc status crc config view sw_vers |
Can you share the dockerfile and steps how your run those which result to this failure? It will help use to reproduce and find out exact issue. |
|
@praveenkumar Although I cannot disclose all of the files, I was able to cause similar behavior with the following configuration. The $ crc cleanup
$ crc setup
$ crc start
$ eval $(crc podman-env)
$ env | grep DOCKER
DOCKER_BUILDKIT=0
DOCKER_HOST=unix:///Users/nao/.crc/machines/crc/docker.sock
$ tree
.
├── Dockerfile
└── compose.yaml
0 directories, 2 files
$ docker compose build
# ... after a while of build progress, the build process looks like it is in error or hangsDockerfilecompose.yamlservices:
web-1:
build:
context: .
args:
APP: web-1
web-2:
build:
context: .
args:
APP: web-2
web-3:
build:
context: .
args:
APP: web-3
web-4:
build:
context: .
args:
APP: web-4
web-5:
build:
context: .
args:
APP: web-5
web-6:
build:
context: .
args:
APP: web-6
web-7:
build:
context: .
args:
APP: web-7
web-8:
build:
context: .
args:
APP: web-8
web-9:
build:
context: .
args:
APP: web-9
web-10:
build:
context: .
args:
APP: web-10Once hung, commands such as Sometimes executing Builds also rarely work, so I have wondered if the behavior depends on the CPU and memory utilization of the host, but I have not found a way to reproduce the event reliably. |
|
@na0x2c6 Is your use case around crc is just consuming it for podman/docker client or you also want to use openshift cluster? If earlier then can you try |
|
@praveenkumar $ crc config get preset
preset : podman |
|
Sorry. Is this issue for the |
|
When experiencing a hang during ssh connection, I noticed that the kernel panic log was outputted upon checking the vfkit.log. It seems to be similar to the issue in #3349. I share the log below. ~/.crc/machines/crc/vfkit.log |
@na0x2c6 I think it is the issue with heavy workload to be run on the cluster ( openshift preset) or podman (podman preset) but for @cgruver it is fixed with latest version. I am wondering if we do need to test the heavy workload (creating parallel builds ..etc.) |
|
@praveenkumar I will refrain from commenting further for now, as the following log suggests that this issue may not be directly related to crc. Thank you for your kind assistance. |
|
Yes, this looks very similar to #3349 Admittedly, I don't use the |
|
@na0x2c6 Try giving If I read this thread correctly, it looks like you have only allocated 8GB to crc config set memory 12288 |
|
Thank you for your comment. Strangely enough, I noticed that the same issue occurs with another M1 Macbook Air (unfortunately with only 8GB Memory) using Kernel panic log when running `docker compose stop`:I found reports of issues that seem to be related to Virtualization.Framework on Apple's Developer Forums: Running Linux with GUI under Virtualization framework unexpectedly stuck after several minutes Although the title of the issue above includes "GUI", looking at other users' replies, it seems that it does not matter whether it is a GUI or not; rather, the issue appears to occur with specific (literally "specific", rather than old or new) kernel versions. |
|
I also had this issue on my Mac (2019 - Intel), but those commands worked: crc delete
crc cleanup
crc setup --log-level debug
crc start --cpus 6 --memory 16000 --disk-size 64 --log-level debugI think |
|
|
|
Operators needed for Service Mesh, Service Mesh, Kubeflow. |
|
I think this issue would be resolved by crc-org/vfkit#76 because I met a similarly issue at Lima and it had fixed by lima-vm/lima@488c95c. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
General information
crc setupbefore starting it (Yes/No)? YesCRC version
CRC status
# Put `crc status --log-level debug` output hereCRC config
Host Operating System
Steps to reproduce
Expected
Vm is running
Actual
VM is not running
Logs
Before gather the logs try following if that fix your issue -- No
The text was updated successfully, but these errors were encountered: