Remote ssh docker fails with "Error: Request timed out."

[markedwards]

VSCode version: 1.58.2
Local OS: OS X 11.5
Remote OS: AWS Linux
Local Docker version: 20.10.7
Remote Docker version: 20.10.4

I can no longer connect to the remote docker using VSCode, no matter what method I try. I've tried docker.host, docker.context, etc. Always results in a timeout. This used to work fine, seems like it broke in the latest VSCode update for me.

I can connect to the remote just fine using docker from the command line.

[bwateratmsft]

@markedwards do you have an SSH agent set up? More info here: https://code.visualstudio.com/docs/containers/ssh

[markedwards]

Yes, I confirm with ssh-add -l, and I've also run ssh-add again to no avail.

[bwateratmsft]

Ok. If you run this in a VSCode terminal, does it give a result? echo $SSH_AUTH_SOCK

[markedwards]

Yes, and the path it sends back exists.

[bwateratmsft]

Hm... I assume you've also done a restart at some point since the problem came up?

[markedwards]

Yes, this morning. It seemed to start after a VSCode update, after having worked consistently for a long time prior with no intentional changes on my side.

Again, I can connect using DOCKER_HOST or the docker context just fine from the command line, both in the VSCode terminal and in iTerm/Terminal. Its only the VSCode plugin UI which is timing out.

[bwateratmsft]

Is there any additional info if you hover over the error message in the explorer, or in the Output > Docker window?

What seems most likely to me is that there is something wrong with the SSH agent, despite it apparently being fine. The Docker CLI does not use it, only the extension UI, so that would explain the CLI working but the UI not.

[markedwards]

It also says "Failed to connect. Is Docker running?". But no additional info on hover.

I don't see any info about the issue in Output > Docker. All that has is:

The Docker extension will try to connect to 'foo', via context 'bar'.

Incidentally, I am still able to connect to a container on the remote machine if I first open a VSCode window to the machine via "Connect to host...", and then use the Docker extension from that window.

[bwateratmsft]

Well, my next best idea is to just kill and restart the SSH agent; I feel like I've seen it get "stuck" before in weird ways. Can you try these steps?

kill $(pgrep ssh-agent)
eval $(ssh-agent)
ssh-add <your key>
code

[markedwards]

Still not working. If its the SSH agent, shouldn't I also be failing to connect to a remote container from an ssh-connected VSCode window to the remote machine?

[bwateratmsft]

For a Remote - SSH session, the Docker extension essentially gets to pretend it is running locally on the SSH server, and runs no differently than it would if it were on your own machine with Docker Desktop present.

The reason that the Remote - SSH session can be established in the first place is because it doesn't actually use the SSH agent--rather, it just uses the ssh command line which has a different way of looking up SSH keys, I'm not familiar with how it works. The Docker CLI uses the same approach under the hood which is why the CLI doesn't need the SSH agent either.

Under the hood, we use Dockerode for connectivity to Docker in the explorer view, and several commands, which in turn uses ssh2 to talk via SSH to Docker hosts. ssh2 either needs to be fed the key directly--which dockerode doesn't have a way of doing--or have an SSH agent to give it the key. I'm not really fond of the experience but it's all we have to work with.

@karolz-ms do you have any ideas what might be happening here?

[karolz-ms]

I cannot think about anything we changed recently that would cause the SSH connectivity from Docker extension to fail. That said, in general, we recommend to use Remote - SSH and open a full session against the remote machine, for reasons that Brandon gives above. It is just a much more robust that trying to work around the limitations of Dockerode and ssh2.

Only thing that comes to my mind is if the SSH key has a passphrase, the timeout might be due to ssh agent trying to obtain that passphrase, but having no real means to do so. I have seen similar issues when using git/SSH as means to authenticate to GitHub.

[markedwards]

No passphrase on the key, and I haven’t changed the key for years.

I generally have been doing a full Remote SSH session, and that’s working. I was hoping for a lighter weight option, because I end up having rebuild two VSCode windows whenever I change networks.

[leimantas]

@markedwards do you have an SSH agent set up? More info here: https://code.visualstudio.com/docs/containers/ssh

This helped: ssh-add .ssh/id_rsa

[bwateratmsft]

I think the last idea I have left is that maybe the IP of the target VM changed and the extension is now trying to connect to the wrong place. Is your Docker context based on a hostname or IP address?

I verified that I am able to connect to an SSH host (local network, not cloud though) on OS X 11.5.1, VSCode 1.58.2, Docker extension 1.15.0.

[markedwards]

My Docker context is not based on an IP, its based on a hostname store in an SSH config Host, and I use the same Host to successfully connect everywhere else.

[bwateratmsft]

I just found another potential cause. I found that if you have not added an SSH key for the server, you get the same error message--"Request timed out."

I know you mentioned you have added the keys, and that's where another recurring Mac issue comes up (#2578, #2894), in which for some reason the VSCode process (and extension host processes) have a totally different value for PATH compared to the integrated terminal and other terminal windows.

So there's two things to look at--

1. Check what value is present for SSH_AUTH_SOCK in the terminal, with echo $SSH_AUTH_SOCK
2. Check what value VSCode has. Run the command "Developer: Toggle Developer Tools" (not to be confused with "Developer: Open Webview Developer Tools). It opens up a dev tools window where we can inspect the environment. Go to the Console tab (usually on the bottom), and type in process.env.SSH_AUTH_SOCK and hit enter, it should output some value.
   

My suspicion is that those two values don't match, and as a result, when you add the key via terminal, it gets added to the correct instance of the SSH agent...but for whatever reason is messing up environment variables, VSCode isn't looking at the correct instance.

[markedwards]

I figured out my issue, and it was stupid in the end.

I missed this note in the VSCode docs you pointed to: "Docker extension will not be able to use host aliases defined in the SSH configuration file."

Switched to use the hostname instead of the alias and it's working. What's surprising is (I think) it was actually working for a short time using ssh://user@alias set in docker.host. I had that set in a workspace that I saved, and then the workspace just stopped working. Unfortunately I can't go back in time and confirm this happened.

Anyway, doesn't matter, its sorted now.

Thanks everyone for the help on this, and sorry for the runaround.

[bwateratmsft]

Awesome, glad it's working now! Thanks for the update!