podman generated remote connections may fail from possible mismatch between "localhost" and 127.0.0.1 listener #16470

uaru · 2022-11-10T07:19:00Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman cannot create connect to the machine

Steps to reproduce the issue:

brew install podman
podman machine init

~ podman machine init
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

	podman machine start

podman machine init

~ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users/tomanik:/Users/tomanik

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful

API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully

podman machine info

~ podman machine info
Host:
  Arch: arm64
  CurrentMachine: podman-machine-default
  DefaultMachine: podman-machine-default
  EventsDir: /var/folders/l3/0fn2jt1d62b7s3kmv4q7m0yh0000gv/T/podman-run--1/podman
  MachineConfigDir: /Users/tomanik/.config/containers/podman/machine/qemu
  MachineImageDir: /Users/tomanik/.local/share/containers/podman/machine/qemu
  MachineState: Running
  NumberOfMachines: 1
  OS: darwin
  VMType: qemu
Version:
  APIVersion: 4.3.0
  Built: 1666115757
  BuiltTime: Wed Oct 19 02:55:57 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: darwin
  OsArch: darwin/arm64
  Version: 4.3.0

Describe the results you received:

~ podman --log-level DEBUG run quay.io/podman/hello
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level DEBUG run quay.io/podman/hello)
DEBU[0000] SSH Ident Key "/Users/tomanik/.ssh/podman-machine-default" SHA256:u......uSJjM ssh-ed25519
Error: failed to connect: dial tcp [::1]:51112: connect: connection refused

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

There is almost identical issue with previous version: #12728

But is still does not work.

I did try recreate the machine and unset SSH_AUTH_SOCK - it did not work.

Ironically, I was able to run docker-compose up -d, only with the issue that it did not accepted environment variables.

Output of podman version:

➜  ~ podman version
Error: failed to connect: dial tcp [::1]:51112: connect: connection refused

Output of podman info:

➜  ~ podman info
Error: failed to connect: dial tcp [::1]:51112: connect: connection refused

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

➜  ~ brew info podman
==> podman: stable 4.3.0 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.3.0 (185 files, 47.6MB) *
  Poured from bottle on 2022-11-10 at 15:44:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0 and GPL-3.0-or-later
==> Dependencies
Build: go-md2man ✘, [email protected] ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions

To restart podman after an upgrade:
  brew services restart podman
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/podman/bin/podman system service --time=0
==> Analytics
install: 28,592 (30 days), 75,743 (90 days), 226,171 (365 days)
install-on-request: 27,263 (30 days), 72,937 (90 days), 222,845 (365 days)
build-error: 23 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

MacBook M1 macOS 12.6.1

The text was updated successfully, but these errors were encountered:

Luap99 · 2022-11-10T17:52:07Z

Do you have a gvproxy process running and does it listen on the mentioned port?

uaru · 2022-11-11T00:23:14Z

It seems so.

~ ps -ef |grep gvproxy
507 59659     1   0  9:18AM ttys006    0:00.11 /opt/homebrew/Cellar/podman/4.3.0/libexec/podman/gvproxy -listen-qemu unix:///var/folders/l3/0fn2jt1d62b7s3kmv4q7m0yh0000gv/T/podman/qmp_podman-machine-default.sock -pid-file /var/folders/l3/0fn2jt1d62b7s3kmv4q7m0yh0000gv/T/podman/podman-machine-default_proxy.pid -ssh-port 51112 -forward-sock /Users/tomanik/.local/share/containers/podman/machine/podman-machine-default/podman.sock -forward-dest /run/user/507/podman/podman.sock -forward-user core -forward-identity /Users/tomanik/.ssh/podman-machine-default

benfay88 · 2022-11-13T10:37:27Z

Had the same problem, I had to create a new machine and then

podman system connection default my-machine

Solved the problem.

uaru · 2022-11-14T00:16:13Z

@benfay88 Thank you for reply. Unfortunately, in my case it did not change anything.

Luap99 · 2022-11-14T11:48:50Z

@uaru Can you manually change the ssh host url in ~/.config/containers/containers.conf from localhost to 127.0.0.1 to see if this changes anything.

EvilJohn · 2022-11-18T21:07:26Z

@Luap99 I had this problem, and your suggested change to the config from localhost to 127.0.0.1 worked. I did delete the machine and reinitialized it via the desktop app and it worked.

MacOS Ventura 13.0.1 running on M1 Max Ultra

uaru · 2022-11-25T01:11:52Z

@Luap99 Thank you. Your hint worked.

➜  ~ podman run quay.io/podman/hello
Trying to pull quay.io/podman/hello:latest...
Getting image source signatures
Copying blob sha256:5fb05dbb69d7111df80948bebe9015f75950bb8bc527ef543bee75f65f7980fc
Copying config sha256:77ba9de027738149c2e1f12c749de13d188e828092032b41ae9eb80e5b746139
Writing manifest to image destination
Storing signatures
!... Hello Podman World ...!

         .--"--.
       / -     - \
      / (O)   (O) \
   ~~~| -=(,Y,)=- |
    .---. /`  \   |~~
 ~/  o  o \~~~~.----. ~~
  | =(X)= |~  / (O (O) \
   ~~~~~~~  ~| =(Y_)=-  |
  ~~~~    ~~~|   U      |~~

Project:   https://github.com/containers/podman
Website:   https://podman.io
Documents: https://docs.podman.io
Twitter:   @Podman_io
➜  ~

OscarVanL · 2022-11-27T06:26:41Z

Thanks @Luap99, one annoying thing I noticed is that if I destroy the machine and recreate it, then this will be created again with localhost instead of 127.0.0.1. I had to trawl through my browser history to find this thread again 😅

Btijmen · 2022-12-07T13:51:13Z

For me, changing localhost to 127.0.0.1 did not help unfortunately. I do not have any gvproxy process running.
Mac OS Ventura 13.0.1 running on Apple M1 Pro chip.

% podman version
Error: failed to connect: dial tcp 127.0.0.1:50284: connect: connection refused

% brew info podman
==> podman: stable 4.3.1 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.3.1 (185 files, 47.6MB) *
  Poured from bottle on 2022-11-16 at 15:13:30
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0 and GPL-3.0-or-later
==> Dependencies
Build: go-md2man ✘, [email protected] ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions

To restart podman after an upgrade:
  brew services restart podman
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/podman/bin/podman system service --time=0
==> Analytics
install: 29,067 (30 days), 79,649 (90 days), 241,349 (365 days)
install-on-request: 26,537 (30 days), 75,187 (90 days), 235,714 (365 days)
build-error: 15 (30 days)

Update
installing the mac helper resolved the issue
sudo /opt/homebrew/Cellar/podman/4.3.1/bin/podman-mac-helper install

github-actions · 2023-01-07T00:06:25Z

A friendly reminder that this issue had no activity for 30 days.

anselm94 · 2023-01-11T08:36:34Z

In my case, I used Podman Desktop which created a new machine. I listed the machines (podman machine list) and stopped & deleted the machine created by the Podman Desktop (podman machine rm new-machine-1) and then starting the default machine (podman machine start podman-machine-default) worked

cajual · 2023-01-14T16:29:24Z

This does not resolve OPs issue, but may resolve others.

Folks, remember to follow the machine start prompts, if applicable:

This is Macbook M1 Max
Darwin my-mac 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the
following commands:

	sudo /opt/homebrew/Cellar/podman/4.3.1/bin/podman-mac-helper install
	podman machine stop; podman machine start

You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

	export DOCKER_HOST='unix:///Users/user1/.local/share/containers/podman/machine/podman-machine-default/podman.sock'

Once I added the system helper service I have had zero issues with pulling/running containers.

amoscatelli · 2023-01-20T09:55:29Z

On Windows I reproduce the same error.
From time to time I have to do the following :

podman machine stop
podman machine rm
podman machine init

otherwise podman won't start again.

I have to repeat this steps once every 2-3 days ...

amoscatelli · 2023-01-20T09:56:30Z

And there is no ./.config/containers/containers.conf in my user folder

heathfx · 2023-02-09T04:03:50Z

On Windows I reproduce the same error. From time to time I have to do the following :

podman machine stop podman machine rm podman machine init

otherwise podman won't start again.

I have to repeat this steps once every 2-3 days ...

And there is no ./.config/containers/containers.conf in my user folder

I have this exact same issue as well.

rhatdan · 2023-02-09T18:02:31Z

@baude @ashley-cui @Luap99 @n1hility WDYT?

schiruma · 2023-02-15T22:33:16Z

sudo /opt/homebrew/Cellar/podman/4.3.1/bin/podman-mac-helper install

This command doesn't seem to be working. Could you tell how did you install it ?

n1hility · 2023-02-15T23:53:01Z

I think folks might be reporting multiple different issues against this issue. The mac helper pertains only to Docker API clients, so anything using the REST interface. The podman CLI talks directly to podman machine over ssh. The OP is describing a failure on the latter case. If this latter case with the podman cli is happening it implies something is wrong with the connection configuration. You can access this with “podman system connection ls” The key and port of the default marked connection value should match the values returned from podman machine info / ls.

uaru · 2023-02-16T02:19:48Z

In my case manually replacing localhost with 127.0.0.1 in ~/.config/containers/containers.conf helped.

However, recently I could not restart the podman machine (just hanged indefinitely), and in the end, I was forced to recreate the machine (delete and create).

Doing it I faced the same problem again, having to manually correct the config file again. So clearly something is wrong with the way how default configuration is generated.

The solution in #16579 is not a real solution, but a workaround.

It is good that it works and I would agree it is not a high priority, but I think it cannot be called a final solution if out-of-the-box installation has to be tweaked by removing something manually and then adding something.

vyasgun · 2023-03-02T14:08:21Z

/assign

punisher1 · 2023-03-08T13:47:45Z

The port may be used. You need to kill the process that occupies the port
Ports may be reserved;

View reserved ports:

netsh interface ipv4 show excludedportrange protocol=tcp

Add excluded reserved port:

net stop winnat

netsh int ipv4 add excludedportrange  protocol=tcp startport=5402 numberofports=1

net start winnat

n1hility · 2023-03-08T23:57:17Z

For those on mac that are seeing the change of localhost -> 127.0.0.1 working? Could you paste the following:

grep localhost /etc/hosts
host localhost

@rhatdan @Luap99 I took a look and gvproxy is hardcoded to bind 127.0.0.1 yet we generate our client configs to use localhost. I suspect some mac configs (perhaps most recent installs) are now preferring ::1 in the host db, so that leads to a mismatch. Possible solutions would be generating config to match or dual binding a second listener on ::1

deboer-tim · 2023-03-09T08:09:45Z

I haven't hit this problem yet, but FWIW here is what's returned from my 13.2.1 M1:

> ~ % grep localhost /etc/hosts
127.0.0.1 localhost podman.crc.testing
::1 localhost
> ~ % host localhost
Host localhost not found: 3(NXDOMAIN)

Luap99 · 2023-03-09T13:20:37Z

@rhatdan @Luap99 I took a look and gvproxy is hardcoded to bind 127.0.0.1 yet we generate our client configs to use localhost. I suspect some mac configs (perhaps most recent installs) are now preferring ::1 in the host db, so that leads to a mismatch. Possible solutions would be generating config to match or dual binding a second listener on ::1

I would just code the config to use 127.0.0.1, seems simpler than having to deal with two listeners.

yosiasz · 2023-04-15T18:31:18Z

in my case on windows using v4.4.4

failed to connect: dial tcp [::1]:49858

just stopped default machine
deleted default machine
initialized default machine
started default machine

the above worked in my case. maybe default machine in a funky state?

kayhayen · 2023-04-16T08:59:57Z

This is on Windows after a recent OS upgrade. I was using a 4.x which worked at some point. Returning to it today, it did not. I tried to stop/start the machine, remove the folders as advised, upgrade to 4.5, and lots of other things. Even rebooting, assuming somehow WSL networking is messed up.

I am getting this:

podman.exe run localhost/nuitka-ci python --version
Trying to pull localhost/nuitka-ci:latest...
Error: initializing source docker://localhost/nuitka-ci:latest: pinging container registry localhost: Get "https://localhost/v2/": dial 
tcp 127.0.0.1:443: connect: connection refused

Looking at this:

podman.exe version
Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.20.2
Git Commit:   75e3c12579d391b81d871fd1cded6cf0d043550a
Built:        Fri Apr 14 17:21:09 2023
OS/Arch:      windows/amd64

Server:       Podman Engine
Version:      4.4.4
API Version:  4.4.4
Go Version:   go1.19.7
Built:        Mon Apr  3 13:31:25 2023
OS/Arch:      linux/amd64

I am not sure, if I should not have 4.5 for the engine as well. I had tried to delete it in WSL with --unregister, but maybe that wasn't hard enough. I find it strange, that there is no upgrade for the machine. Interestingly I am using 127.0.0.1 already, but it seems C:\Users\kayha\.local\share\containers\podman\machine\wsl\podman-machine-default_fedora-podman-amd64-v37.0.5.tar is used.

Can anybody confirm or deny that these versions ought to be identical?

Luap99 · 2023-04-17T11:34:20Z

Can anybody confirm or deny that these versions ought to be identical?

We have currently no way of making sure that the versions are equal, the VM image just uses standard fedora versions so it is expected that there is some drift. This shouldn't cause any issues here.

kayhayen · 2023-04-17T13:01:40Z

Ok, just for completeness sake. Downgrading to 4.4.4 on the outside didn't change it. I had installed it also on a Windows 11 laptop, and there is also stopped working. I am pretty sure, Windows updates are breaking it somehow for me. I cannot be the only one, can I.

n1hility · 2023-04-17T13:54:22Z

@kayhayen I think there is an issue with the run command you are passing, localhost/nuitka-ci specifies localhost as a remote image registry to pull from, and there isn't one running. If this is an image you created locally, you could just say podman run nuitka-ci. Since your podman version command is working it looks like you are getting a connection to the machine, so you could try another image as well (e.g. podman run -it ubi8-micro)

kayhayen · 2023-04-18T08:17:11Z

@n1hility you are spot on, that is not how it works indeed. Shame on me and sorry for the noise.

jeremycaine · 2023-04-27T17:53:37Z

I have been trying to get crc and podman to co-exist nicely on my Mac. Then many of these issues ^^ started appearing. I installed crc, and then I brew install podman.

I am getting the original problem

podman version
Error: failed to connect: dial tcp 127.0.0.1:2222: connect: connection refused

and one thing I notice after cleaning things up like remove and init podman is this:

Name                         URI                                                         Identity                                        Default
crc                          ssh://[email protected]:2222/run/user/1000/podman/podman.sock  /Users/jeremycaine/.crc/machines/crc/id_ecdsa   true
crc-root                     ssh://[email protected]:2222/run/podman/podman.sock            /Users/jeremycaine/.crc/machines/crc/id_ecdsa   false
podman-machine-default       ssh://core@localhost:49710/run/user/501/podman/podman.sock  /Users/jeremycaine/.ssh/podman-machine-default  false
podman-machine-default-root  ssh://root@localhost:49710/run/podman/podman.sock           /Users/jeremycaine/.ssh/podman-machine-default  false

so I did the following to change which VM to issue podman commands against

podman system connection default podman-machine-default

and then

podman version
Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.20.3
Git Commit:   75e3c12579d391b81d871fd1cded6cf0d043550a
Built:        Fri Apr 14 14:28:20 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.4.4
API Version:  4.4.4
Go Version:   go1.20.1
Built:        Mon Apr  3 12:28:56 2023
OS/Arch:      linux/arm64

n1hility · 2023-04-27T20:48:31Z

@jeremycaine just a small tip, you can also do -c after podman if you switch frequently and don't want to change the default (e.g. podman -c crc run -it ubi8)

devops-corgi · 2023-05-13T03:17:33Z

Just want to add my 5 cents. Want to switch to podman from docker.

M1 Macbook Pro
Uninstalled Docker via official uninstall scripts, installed Podman Desktop
Running into this exact error, IE
- Error: failed to connect: ssh: handshake failed: read tcp 127.0.0.1:63693->127.0.0.1:52406: read: connection reset by peer
Stopping and restarting the VM helps for 1-2 days

From the looks of it, I don't think it's anything networking related. My working theory is, something in the machine itself breaks/freezes if left running for a couple of days, which is why a restart fixes it. Maybe when the computer goes to sleep.

Luap99 · 2023-05-15T09:56:23Z

First of all everyone keeps commenting with different kind of issues. The symptom may be similar but there are definitely different kind of problems reported here. Pleas look at the error messages carefully.

Based on my #16470 (comment) this seems to be one problem which is easily fixed. The error message for this is dial tcp [::1]:<random port>: connect: connection refused Based on the comments this works and is what the original issue is about, I will try to fix it later.

And to everyone please stop hijacking existing issues with your problems, unless the error is the same or you are actually sure this is the same issue please just create a new separate issue. I triage almost all issues here. It is much simpler for me to close them as duplicates when they are actually are the same issue and link them the the correct ones. This allows users and developers to properly debug issues and not getting side tracked all the time by other unrelated comments.
Thank you for your understanding.

n1hility · 2023-05-15T15:16:44Z

@Luap99 took a shot at a more specific subject line hoping it might help encourage new issue opens for other problems

gvproxy listens on 127.0.0.1, using localhost as hostname can result in the client trying to connect to the ipv6 localhost (`::1`). This will fail as shown in the issue. This switches the hostname in the system connection to 127.0.0.1 to fix this problem. I switched the qemu, hyperV and WSL backend. I haven't touched the applehv code because it uses two different ips and I am not sure what is the correct thing there. I leave this to Brent to figure out. [NO NEW TESTS NEEDED] [1] https://github.com/containers/gvisor-tap-vsock/blob/main/cmd/gvproxy/main.go#L197-L199 Fixes containers#16470 Signed-off-by: Paul Holzinger <[email protected]>

tppalani · 2023-05-28T17:38:57Z

podman system connection default podman-machine-default

Hi Team everyone is facing same issue can you please someone provide the resolution for this.

JDUNNIN · 2023-07-11T14:35:21Z

Had the same problem, I had to create a new machine and then
podman system connection default my-machine
Solved the problem.

Little gotcha, that if you run podman machine in rootful mode then you should name the root connection in the command like podman system connection default podman-machine-default-root to avoid accidentally switching back to non-root 501 user.

podman system connection list should then return with the default being the one with ssh://root in the URI.

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 10, 2022

github-actions bot added the macos MacOS (OSX) related label Nov 10, 2022

Luap99 added the machine label Nov 10, 2022

Luap99 mentioned this issue Nov 22, 2022

Podman ssh connection reset by peer M1 Mac #16579

Closed

github-actions bot added the stale-issue label Jan 7, 2023

github-actions bot removed the stale-issue label Jan 12, 2023

openshift-ci bot assigned vyasgun Mar 2, 2023

dr1nf3rn0 mentioned this issue Apr 20, 2023

No connection could be made because the target machine actively refused it #19554

Closed

vyasgun removed their assignment May 2, 2023

Luap99 mentioned this issue May 15, 2023

machine: fix default connection URL to use 127.0.0.1 #18565

Merged

Luap99 self-assigned this May 15, 2023

n1hility changed the title ~~podman failed to connect: connection refused~~ podman generated remote connections may fail from possible mismatch between "localhost" and 127.0.0.1 listener May 15, 2023

openshift-merge-robot closed this as completed in #18565 May 16, 2023

openshift-merge-robot closed this as completed in ce07860 May 16, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Oct 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 10, 2023

podman generated remote connections may fail from possible mismatch between "localhost" and 127.0.0.1 listener #16470

podman generated remote connections may fail from possible mismatch between "localhost" and 127.0.0.1 listener #16470

Comments

uaru commented Nov 10, 2022 • edited Loading

Luap99 commented Nov 10, 2022

uaru commented Nov 11, 2022

benfay88 commented Nov 13, 2022 • edited Loading

uaru commented Nov 14, 2022

Luap99 commented Nov 14, 2022

EvilJohn commented Nov 18, 2022

uaru commented Nov 25, 2022

OscarVanL commented Nov 27, 2022

Btijmen commented Dec 7, 2022 • edited Loading

github-actions bot commented Jan 7, 2023

anselm94 commented Jan 11, 2023

cajual commented Jan 14, 2023

amoscatelli commented Jan 20, 2023

amoscatelli commented Jan 20, 2023 • edited Loading

heathfx commented Feb 9, 2023

rhatdan commented Feb 9, 2023

schiruma commented Feb 15, 2023

n1hility commented Feb 15, 2023 • edited Loading

uaru commented Feb 16, 2023

vyasgun commented Mar 2, 2023

punisher1 commented Mar 8, 2023 • edited Loading

n1hility commented Mar 8, 2023

deboer-tim commented Mar 9, 2023

Luap99 commented Mar 9, 2023

yosiasz commented Apr 15, 2023 • edited Loading

kayhayen commented Apr 16, 2023

Luap99 commented Apr 17, 2023

kayhayen commented Apr 17, 2023

n1hility commented Apr 17, 2023 • edited Loading

kayhayen commented Apr 18, 2023

jeremycaine commented Apr 27, 2023

n1hility commented Apr 27, 2023

devops-corgi commented May 13, 2023

Luap99 commented May 15, 2023

n1hility commented May 15, 2023

tppalani commented May 28, 2023

JDUNNIN commented Jul 11, 2023

uaru commented Nov 10, 2022 •

edited

Loading

benfay88 commented Nov 13, 2022 •

edited

Loading

Btijmen commented Dec 7, 2022 •

edited

Loading

amoscatelli commented Jan 20, 2023 •

edited

Loading

n1hility commented Feb 15, 2023 •

edited

Loading

punisher1 commented Mar 8, 2023 •

edited

Loading

yosiasz commented Apr 15, 2023 •

edited

Loading

n1hility commented Apr 17, 2023 •

edited

Loading