SSH Agent Forwarding not working with Remote SSH on MacOS (still a problem)

[mbmccoy]

The title is the same as microsoft/vscode-remote-release#2671 because I'm having precisely the same problem, and that issue was closed without a resolution. In short, ssh agent forwarding does not work in VSCode, despite working just fine when I SSH in via the terminal. It seems to be due to a stale / cached SSH_AUTH_SOCK.

The workaround suggested in microsoft/vscode-remote-release#2671 no longer works, but a minor modification does it for me. See below.

VSCode Version:

Version: 1.53.2
Commit: 622cb03f7e070a9670c94bae1a45d78d7181fbd4
Date: 2021-02-11T11:45:54.515Z
Electron: 11.2.1
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: 8.7.220.31-electron.0

OS: Darwin 20.6.0
Local OS Version: MacOS 11.6.1
Remote OS Version: Debian GNU/Linux 10 (buster) (GNU/Linux 4.19.0-18-cloud-amd64 x86_64)
Remote Extension/Connection Type: SSH
Steps to Reproduce:

1. set up ssh config with a host you want to forward your ssh agent keys to and set 'ForwardAgent yes'
2. Connect via terminal ssh and perform an operation requiring your forwarded key (in my case, ssh -T [email protected])
3. launch VSCode and validate that $SSH_AUTH_SOCK is set
4. Connect via Remote-SSH, then use the VSCode terminal to run ssh -T [email protected]. This time, I receive a permission denied error.

Does this issue occur when you try this locally?: NA
Does this issue occur when you try this locally and all extensions are disabled?: NA

Possible cause

The environment variable SSH_AUTH_SOCK appears to be stale. A workaround is to run

export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1)

which sets the socket to the newest one. Note that this workaround is quite fragile, as I had to modify the previous workaround to make it work now.

[tanhakabir]

Could it be an issue with needing to set an IdentityAgent to the appropriate authentication agent since you said the current SSH_AUTH_SOCK env variable seems to be stale?

More details from the SSH Man(5) page:

Specifies the UNIX-domain socket used to communicate with the authentication agent.

This option overrides the SSH_AUTH_SOCK environment
variable and can be used to select a specific agent.
Setting the socket name to none disables the use of an
authentication agent. If the string "SSH_AUTH_SOCK" is
specified, the location of the socket will be read from the
SSH_AUTH_SOCK environment variable. Otherwise if the
specified value begins with a ‘$’ character, then it will
be treated as an environment variable containing the
location of the socket.

Arguments to IdentityAgent may use the tilde syntax to
refer to a user's home directory, the tokens described in
the TOKENS section and environment variables as described
in the ENVIRONMENT VARIABLES section.

[mbmccoy]

Hmm.... I don't think that's the source of the problem. In particular, when I use ssh from the command line (using the same user/host that I use in VSC), I don't have this issue. I don't have any special settings for IdentityAgent in either my local or remote machine.

I'm often able to trigger the issue this way:

1. Open remote connection to server.
   • I can see that my identity is not appropriately forwarded via ssh -T [email protected] in VSCode terminal.
2. Run workaround in VSCode terminal: export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1).
   • Now ssh -T [email protected] succeeds.
3. Reload window (Developer: Reload Window).
   • Now ssh -T [email protected] fails.

Even weirder, after repeating steps 2 and 3 a few times, it's started using the correct socket after reload.

[roblourens]

What is $SSH_AUTH_SOCK before you run the workaround?

[mbmccoy]

I'm not able to trigger the issue right now, but $SSH_AUTH_SOCK was definitely set, and it looked structurally correct, like /tmp/ssh-cT3h6tFlau/agent.316, only the directory (and file) didn't exist.

[mbmccoy]

Ok, I'm able to reproduce now. These are all run from within the VSCode terminal. Here's the initial situation:

$ echo $SSH_AUTH_SOCK

/tmp/ssh-7G9TqKKzBp/agent.30503

$ ls -la /tmp/ssh*/**

srwxr-xr-x 1 xxx xxx 0 Feb  9 00:27 /tmp/ssh-9L8JuutTmn/agent.14198
srwxr-xr-x 1 xxx xxx 0 Feb  8 20:34 /tmp/ssh-jFxWY0hBUn/agent.16442

$ ssh -T [email protected]

[email protected]: Permission denied (publickey).

We can see that the SSH_AUTH_SOCK variable is not pointed at an extant ssh agent. Now we apply the workaround:

$ export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1)

$ ssh -T [email protected]

Hi mbmccoy! You've successfully authenticated, but GitHub does not provide shell access.

$ echo $SSH_AUTH_SOCK

/tmp/ssh-9L8JuutTmn/agent.14198

[roblourens]

Most likely you went through reconnection and are now connecting through a different ssh process, this means the original ssh auth sock is invalid. The extension tries to redirect SSH_AUTH_SOCK to a different path that we control, seems like that didn't work here. Do you have terminal.integrated.inheritEnv or remote.SSH.enableAgentForwarding disabled?

If not can you share the Remote-SSH log from the output panel?

[mbmccoy]

Here's the output for a run where I just had this issue:

https://gist.github.com/mbmccoy/331ba3674877b7c8769b482f78e467b7

Note that it does seem to have the correct agent in the logs (/tmp/ssh-vA5Po49GCF/agent.28474), but it was not correct in the integrated terminal because I had to update it using the fix mentioned above.

This makes me suspect that the terminal caches the SSH_AUTH_SOCK environment variable independent of the Remote SSH extension.

[roblourens]

Do you have terminal.integrated.inheritEnv or remote.SSH.enableAgentForwarding disabled?

Is this a terminal from a previous session that was reconnected, or a fresh terminal?

[github-actions]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[mbmccoy]

I suppose I dropped the ball on getting back to you on this. No, I don't have any settings that are related to the terminal or remote set in my settings.json.

The ssh command does include the -A flag.

Not sure if it matters, but I do log in to the machine from a terminal outside of VSCode (with agent forwarding enabled). It doesn't always seem to cause the issue.

I've updated to a recent version and I'm still occasionally seeing the issue.

Version: 1.66.0
Commit: e18005f0f1b33c29e81d732535d8c0e47cafb0b5
Date: 2022-03-30T05:50:04.419Z
Electron: 17.2.0
Chromium: 98.0.4758.109
Node.js: 16.13.0
V8: 9.8.177.11-electron.0
OS: Darwin x64 21.3.0

[roblourens]

Can you share the log from the Remote-SSH output channel?

Basically we should be trying to rewrite your SSH_AUTH_SOCK to something like /tmp/vscode-ssh-auth-sock-493210776 or /run/user/1000/vscode-ssh-auth-sock-493210776

[cdmistman]

I believe I'm experiencing the same issue

1. connect to remote host with ForwardAgent yes in ~/.ssh/config. terminal.integrated.inheritEnv and remote.SSH.enableAgentForwarding are both enabled in VSCode
2. verify a key is printed with ssh-add -L
3. echo $SSH_AUTH_SOCK
4. reload window
5. in the same terminal instance from steps 2 and 3, run ssh-add -L again - error code 2 Error connecting to agent: No such file or directory
6. again in the same terminal from step 5, echo $SSH_AUTH_SOCK - it prints the same value as step 3
7. open a new terminal (> Terminal: Create New Terminal) and execute ssh-add -L again - it should have the same result as step 2
8. in the same terminal from step 7, echo $SSH_AUTH_SOCK - it should print a different value from step 6

it seems terminals that are re-opened after a window reload don't get their SSH_AUTH_SOCK environment variable updated. The correct value is correctly set for all terminal instances that are created between window reloads

[mbmccoy]

Was able to trigger this just now. It may be related to the fact that I also connect (and forward my agent) from another terminal. (see note below) Here's the steps that I took:

1. Inside VSCode terminal connected to the remote, check that agent is forwarding:

    ssh -T [email protected]
    Hi mbmccoy! You've successfully authenticated, but GitHub does not provide shell access.
   

2. SSH into machine (forwarding the agent) from a non-VSCode terminal.

3. Reload window (Developer: Reload Window).

4. Check access:

    ssh -T [email protected]
    [email protected]: Permission denied (publickey).
   

Remote-SSH Logs here

NOTE (added in edit) This is also consistent with @cdmistman 's observation that

it seems terminals that are re-opened after a window reload don't get their SSH_AUTH_SOCK environment variable updated. The correct value is correctly set for all terminal instances that are created between window reloads

Opening a new terminal after reload gives access.

[davibe]

Same issue here, closing and recreating the terminal fixes it, but terminals cerated when vscode starts have a stale SSH_AUTH_SOCK

[snewell92]

The format of the fix above needed to be amended to match what I saw in my dev containers:

➜  ~ which ssh_vscode_fix
ssh_vscode_fix () {
	export SSH_AUTH_SOCK=$(ls -t /tmp/vscode-ssh-auth-*.sock | head -1)
}

[tamaszsellerp92]

hi
I'm facing a similar issue with VScode latest
I have the option ForwardAgent yes in my personal configuration file, however VSCode with Remote SSH seems to ignore it
I can't for example clone any git repo or jump over to other hosts with this key, the SSH_AUTH_SOCK env. variable is totally missing when connecting from VSCode terminal.
also it works from putty. I'm using windows (native win. SSH client, not WSL)
sum: connecting to the linux machine with key works fine from VScode, but agent forwarding is not taken into account
also using functions like git clone does not work from vscode, same issue - key is not forwarded. I can do it from terminal if I Set the SSH_AUTH_SOCK manually, but not from the vscode git function/plugin

any guess?

update: if I add export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1) to my .profile and relaunch vscode, it reads it and git clone works fine, but I would like to omit this step, as it should work without it.

[templela]

I'd also like to throw my 2c in the mix. Sometimes when the window is reloaded, or some other state change happens the SSH_AUTH_SOCK points at nothing. Also on every reload a new sock file is created at /tmp/vscode-ssh-auth-*.sock, this seems to be to mitigate the issues with the socket dsiappearing, but once the connection is gone none of the sockets are valid. Sometimes the variable points to a missing socket, sometimes the socket is invalid.

The symptom is similar to other people, not having a proper agent available.

For context the local machine is an M1 Pro mac on 12.5.1, the host is a Ubuntu 20.04.4 LTS machine. This is using both Remote - SSH and Dev Containers. The issue doesn't happen when only using Remote - SSH on the exact same host.

[roblourens]

What's supposed to happen is that when reloading the window, yes, the SSH_AUTH_SOCK in persisted terminals is incorrect. We set it to the /tmp/vscode-ssh-auth-*.sock which is a symlink to the real value. When going through reconnection (losing your connection and getting a new one set up automatically, without reloading the window), the symlink is fixed up to the new value automatically. But we can't do this when reloading the window and starting over from scratch, and SSH_AUTH_SOCK will point at something that doesn't work. That is probably what many of you see (but there's a lot going on in this issue).

I noticed that microsoft/vscode-remote-release#7642 was making this appear to work only because the old ssh process was still alive in the background, incorrectly.

And there's something that does not work: we also use the terminal EnvironmentVariableCollection API to set SSH_AUTH_SOCK in terminals, and it is supposed to show a warning icon on terminals that have already been started and now have a stale value for that variable. But, I don't see the warning icon @meganrogge @Tyriar any known issue? Repro steps for me,

• Set "remote.SSH.useLocalServer": false
• Connect to an ssh remote
• Open a terminal and type in it
• Reload the window
• That terminal should have the warning icon and it doesn't appear

This line in the log shows that we ran the replace method on the collection
[21:34:34.475] Updating terminal environments: {"SSH_AUTH_SOCK":"/run/user/1000/vscode-ssh-auth-sock-283918976"}

[meganrogge]

env var info comes back when I reload after changing it with debug toggle auto attach

env.mov

[Tyriar]

It's possible we don't restore the state correctly for remote terminals only?

[meganrogge]

Yeah the env var info widgets never appear for me in WSL

[lesca]

You are awesome! You saved my day!

[lesca]

Just chiming in, I have encountered this a few times now (including today, right now lol). Works for months, and suddenly the stale SSH behavior pops up. On 1.82.0 locally. My steps to reproduce:

1. Open remote with "Remote-SSH: Connect to Host..." to a server with fresh code server
2. In new window, try ssh -T [email protected], successful auth
3. Add seemingly any new folder to the workspace
4. Try authentication command again, fails this time
5. Delete the code server ~/.vscode-server
6. Go to 1.

Edit: to be clear, using Agent forwarding from the host that has worked previously

I was able to get agent forwarding working again by setting remote.SSH.useExecServer: false. It looks like the issue you and I are experiencing was reported here: microsoft/vscode-remote-release#8926

You are awesome!

+1 on this
In my case, agent is forwarding with ssh in cmd (this means config file is set correctly).
But in VS Code, agent not forwarding, until I changed this setting remote.SSH.useExecServer to false.

Update: unfortunately, it seems this works randomly. Now, ssh-add -l shows the key, but git pull fails with error "Permission denied (publickey)."

[ferreteleco]

Any update on this, it is getting very hard to work on development containers from windows (Windows host to WSL container).

[ascii78]

I've tried logging in with an old 1.6 vscode i had on a usb stick with static config. After logging in remote-ssh tells me the following symlinks are once again established:

[13:58:44.035] Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-XXXXiuO5un/agent.203985" "/run/user/1000/vscode-ssh-auth-sock-956309600"
[13:58:44.035] Using cwd: vscode-remote://ssh-remote%2Bhostname/
[13:58:44.035] Remote extension host environment: {"SSH_AUTH_SOCK":"/run/user/1000/vscode-ssh-auth-sock-956309600"}

cleaning up all the remote.SSH.* settings and workarounds on 1.8 and logging in again with that version the symlinks started working again.

[3f6a]

Was experiencing this issue today. Can confirm that "remote.SSH.useExecServer": false, fixes it.

[haplav]

Was experiencing this issue today. Can confirm that "remote.SSH.useExecServer": false, fixes it.

Same here right now!

FWIW,

>Remote-SSH: Uninstall VS Code Server from Host...

just before connecting works for me as well...

[micahcarroll]

Just in case anyone else has this issue, and "remote.SSH.useExecServer": false (or other things that were mentioned above) does not fix it for them:

Deleting ~/.vscode-server fixed it for me

UPDATE: it only fixed it temporarily 😢 Currently tried the "remote.SSH.useExecServer": false approach, and will report back if it doesn't work for me long-term

[3f6a]

Deleting ~/.vscode-server fixed it for me

At the remote, presumably? @micahcarroll

[ehoogeveen-medweb]

I started having this issue on Windows as well. Previously, closing and reopening the terminal helped, but that stopped working. Setting remote.SSH.useExecServer to false and restarting VSCode seems to have helped for now.

[david-boles]

I just experienced this on Windows 10 as well:

Version: 1.89.1 (user setup)
Commit: dc96b837cf6bb4af9cd736aa3af08cf8279f7685
Date: 2024-05-07T05:13:33.891Z
Electron: 28.2.8
ElectronBuildId: 27744544
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Windows_NT x64 10.0.19045

Remote-SSH version - v0.110.1

SSH_AUTH_SOCK was set to a path that didn't exist in new terminals. Manually setting SSH_AUTH_SOCK to the path in the Remote - SSH output worked fine. Disabling remote.SSH.useExecServer also seems to have worked.

[sbonds]

Deleting the remote $HOME/.vscode-server and letting it auto-install AFTER enabling ForwardAgent seems to have resolved the issue for me as well. It seems like if that initial connection is made without ForwardAgent set, future connections get stuck in that config even when ForwardAgent is enabled later.

[floroz]

I can confirm these steps solved it for me:

• remove forwarding from local/remote ssh configs (comment out)
• close all VSCode instances
• on the remote, delete /home/your-user/.vscode-server
• re-enable the AgentForwarding in both configs
• relaunch VSCode, it will re-install the VSCode server
• VSCode terminal should now have SSH forwarding enabled

[mabdulker]

Both solutions worked for me:

• Solution 1
• Solution 2

[kylebfredrickson]

Setting remote.SSH.useExecServer seemed to work initially but then failed. The only thing that reliably fixes the problem for me is to delete .vscode-server in home.

The problem changes depending on how remote.SSH.useExecServer is set though.

• if set, when echoing SSH_AUTH_SOCK I get a stale /tmp/ssh-*/agent.*
• if not set, I get a stale /run/user/1000/vscode-ssh-auth-sock-*.

[tweakyTweeter]

I also faced the same issue on MacOS. I was able to fix it by deleting ~/.vscode-server on remote machine and then reconnecting via Remote SSH extension.

[lobantseff]

The setting set which seems to work for my setup (macOS client -> Ubuntu host):

{
    "remote.SSH.enableAgentForwarding": true,
    "remote.SSH.useLocalServer": false,
    "remote.SSH.useExecServer": false,
    "remote.SSH.remoteServerListenOnSocket": true,
}

[loodvn]

Thanks @lobantseff, works for me! I'll update here if anything changes on my side though.
Could you please include in your comment some reasoning behind why it works, for posterity?

[PhoenixmitX]

The setting set which seems to work for my setup (macOS client -> Ubuntu host):

{
    "remote.SSH.enableAgentForwarding": true,
    "remote.SSH.useLocalServer": false,
    "remote.SSH.useExecServer": false,
    "remote.SSH.remoteServerListenOnSocket": true,
}

also fixes ubuntu -> ubuntu

[mach10x]

The workarounds mentioned here stops working for me (Windows client-> Linux Host) when I 'Reload Window'.
For me setting export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1) on .zshrc works.
To make that command run only in VSCode integrated terminal, I did it like https://stackoverflow.com/questions/45635168/vscode-how-to-run-a-command-after-each-terminal-open

UPDATE - Switching to the pre-release version of SSH Remote extension fixes this issue for me.
microsoft/vscode-remote-release#9662

[leohku]

The setting set which seems to work for my setup (macOS client -> Ubuntu host):

{
    "remote.SSH.enableAgentForwarding": true,
    "remote.SSH.useLocalServer": false,
    "remote.SSH.useExecServer": false,
    "remote.SSH.remoteServerListenOnSocket": true,
}

Apart from these settings, I also had to add

{
    "terminal.integrated.persistentSessionReviveProcess": "never",
    "terminal.integrated.enablePersistentSessions": false,
}

To prevent the stale terminal session from being restored. After that, agent forwarding worked flawlessly.

[pascal456]

So in my case it was the following (connecting to a RaspberryOS / Debian):

• ✅ Via PowerShell ssh-add -l showed the key in the list.
• ❌ via terminal that opens in vscode remote connection

Solution:

• set the option remote.SSH.remoteServerListenOnSocket": true on the remote/SSH host
• doing this via UI reveals that one should kill the vscode server on that host. So running the command ``:
  
  a list with the configured ssh hosts appears, select the one you are currently connect to / the remote host
• then forwarding was working again after running Reload Window

[Tyriar]

@joshspicer since you're very familiar with the SSH extension now, could you look into this issue and come up with recommendations to fix? Do we need to special case certain terminals to not revive the processes or something?