On a Mac, the extension keeps timing out when using Podman instead of Docker

[benatshippabo]

Steps to reproduce

# based on this blog post https://www.redhat.com/sysadmin/podman-mac-machine-architecture
brew install podman
pip3 install podman-compose
podman machine init
podman machine start

# to get the ssh path
podman system connection ls

Then set the vscode config to use Podman:

{
  "docker.dockerPath": "podman",
  "docker.host": "ssh://core@localhost:50261"
}

Afterwards, go to the Docker extension and see the following:

Expected outcome

After configuring the Docker extension, the user should see the list of created containers.

Context

cross posted here as well containers/podman#12745 (comment)

[bwateratmsft]

@benatshippabo do you have an SSH agent set up? Our explorer uses the dockerode NPM package, which in turn uses ssh2, which unfortunately requires an SSH agent--it is not able to pick up anything from ~/.ssh, unlike the Docker CLI (and presumably the Podman CLI too). I've noticed in the past that ssh2 throws a timeout error when it tries and fails to connect to an SSH agent.

More info about setting up an SSH agent here: https://code.visualstudio.com/docs/containers/ssh

[benatshippabo]

@benatshippabo do you have an SSH agent set up? Our explorer uses the dockerode NPM package, which in turn uses ssh2, which unfortunately requires an SSH agent--it is not able to pick up anything from ~/.ssh, unlike the Docker CLI (and presumably the Podman CLI too). I've noticed in the past that ssh2 throws a timeout error when it tries and fails to connect to an SSH agent.

More info about setting up an SSH agent here: https://code.visualstudio.com/docs/containers/ssh

@bwateratmsft Thanks for the swift reply, I think we are getting somewhere. After adding the ssh private key using:

ssh-add ~/.ssh/podman-machine-default

The error message has now progressed to:

[bwateratmsft]

That looks promising! Is a value present in VSCode for environment variable SSH_AUTH_SOCK?

[benatshippabo]

That looks promising! Is a value present in VSCode for environment variable SSH_AUTH_SOCK?

@bwateratmsft yep, it is set to:

❯ echo $SSH_AUTH_SOCK
/private/tmp/com.apple.launchd.KLlERFdN3t/Listeners

---

edit: I just tried setting the docker.dockerodeOptions instead of docker.host:

{
  "docker.dockerodeOptions": {
    "socketPath": "ssh://core@localhost:50685/run/user/1000/podman/podman.sock"
  }
}

And now it looks like it is connecting now 🎉

But there is an error, I don't think Podman supports the context command:

❯ podman context ls --format="{{json .}}"
Error: unknown flag: --format
❯ podman context ls
Error: unrecognized command `podman context`
Try 'podman --help' for more information.
❯ podman context
Error: unrecognized command `podman context`
Try 'podman --help' for more information.

[bwateratmsft]

Can you try that value of dockerodeOptions' socketPath as docker.host? I wonder if that works.

[benatshippabo]

Can you try that value of dockerodeOptions' socketPath as docker.host? I wonder if that works.

It leads to the same socket hang up error, maybe it's how the socket path is concatenated?

[bwateratmsft]

Hmm...I wonder if the SSH is actually necessary. Does unix:///run/user/1000/podman/podman.sock work for docker.host?

[benatshippabo]

unix:///run/user/1000/podman/podman.sock

That would be a negative. My understanding is that on a Mac, Podman creates a linux vm to run the containers on and their cli currently interacts through ssh only. Although it seems like they are working on it.

Btw, I added a comment here so we can figure out how to resolve the context command not being available.

[bwateratmsft]

Rats. Seems like this is yet one more thing that would be solved by #3263. I'll mark this as investigate but it may be pretty similar/the same as #3241, and also maybe just solved by #3263.

[bwateratmsft]

I had another thought, what happens if you set all of the following settings?

  "docker.dockerPath": "podman",
  "docker.host": "ssh://core@localhost:50261"
  "docker.dockerodeOptions": {
    "socketPath": "ssh://core@localhost:50685/run/user/1000/podman/podman.sock"
  }

I don't know for sure without trying it, but I think it might cause the logic that is running podman context ls ... to be skipped.

[benatshippabo]

I had another thought, what happens if you set all of the following settings?

I don't know for sure without trying it, but I think it might cause the logic that is running podman context ls ... to be skipped.

Hmm, that gave a message we haven't seen yet:

[bwateratmsft]

Is that port of 50685 still accurate? Do both docker.host and docker.dockerodeOptions have the current port?

[benatshippabo]

Is that port of 50685 still accurate? Do both docker.host and docker.dockerodeOptions have the current port?

Yup they are still accurate:

{
  "docker.dockerPath": "podman",
  "docker.host": "ssh://core@localhost:50685",
  "docker.explorerRefreshInterval": 5000,
  "docker.dockerodeOptions": {
    "socketPath": "ssh://core@localhost:50685/run/user/1000/podman/podman.sock"
  }
}

❯ podman system connection ls
Name                         Identity                                 URI
podman-machine-default*      /Users/btea/.ssh/podman-machine-default  ssh://core@localhost:50685/run/user/1000/podman/podman.sock
podman-machine-default-root  /Users/btea/.ssh/podman-machine-default  ssh://root@localhost:50685/run/podman/podman.sock

[bwateratmsft]

Hm. I am stumped. Setting docker.host in addition to docker.dockerodeOptions did indeed skip that podman context ls ... logic but I am not sure what would cause the ENOENT error. I think that suggests a successful connection to ssh://core@localhost:50685 but /run/user/1000/podman/podman.sock was not found or not accessible. It's not ideal but can you try using root, with ssh://root@localhost:50685/run/podman/podman.sock, to see if that works?

It may also require a different SSH key with ssh-add.

[benatshippabo]

Yeah, same error even when trying to connect as root user. 😞

[bwateratmsft]

@benatshippabo to make sure I set up a realistic repro, does your Mac have an Intel processor or the newer M1 (aka Apple Silicon)?

[benatshippabo]

@benatshippabo to make sure I set up a realistic repro, does your Mac have an Intel processor or the newer M1 (aka Apple Silicon)?

It is on the M1 processor. Thanks @bwateratmsft

[bwateratmsft]

@philliphoff do you mind trying out the scenario on your M1? I'll try it out on my Intel Macbook and see how it goes.

[philliphoff]

This is as far as I get with my M1 (the ENOENT error), and is the same whether I specifically set docker.host or not.

[bwateratmsft]

I was able to get the same results on my Intel Mac.

I've spent some time today looking into this. It seems things are going sideways in docker-modem. It successfully establishes a connection to the SSH host, whereupon it runs docker system dial-stdio in order to communicate with Docker on the host over ordinary HTTP tunneled through the SSH connection. Naturally, this is not going to work for Podman, since it's literally running a docker command.

dockerode / docker-modem accept a custom agent object which could probably be coerced into making it work, but it wouldn't be possible to use this with the Docker extension, because only JSON objects can be fed to docker.dockerodeOptions.

@benatshippabo bad news and good news. The bad news is that I don't think there's currently any way to make this work. The good news is that #3263 ought to solve this and we're hoping to get it in our next release. For now, I'll resolve this one as a dupe of that.

[benatshippabo]

I was able to get the same results on my Intel Mac.

I've spent some time today looking into this. It seems things are going sideways in docker-modem. It successfully establishes a connection to the SSH host, whereupon it runs docker system dial-stdio in order to communicate with Docker on the host over ordinary HTTP tunneled through the SSH connection. Naturally, this is not going to work for Podman, since it's literally running a docker command.

dockerode / docker-modem accept a custom agent object which could probably be coerced into making it work, but it wouldn't be possible to use this with the Docker extension, because only JSON objects can be fed to docker.dockerodeOptions.

@benatshippabo bad news and good news. The bad news is that I don't think there's currently any way to make this work. The good news is that #3263 ought to solve this and we're hoping to get it in our next release. For now, I'll resolve this one as a dupe of that.

Yeah that makes sense to me. Thanks for keeping me posted @bwateratmsft