Remote SSH commands require double escaping before hitting the DefaultShell

[penguin359]

"OpenSSH for Windows" version
1.0.0.0

Server OperatingSystem
Windows Server 2016 Standard

Client OperatingSystem
Fedora 27

What is failing
Attempting to run remote commands using SSH with my DefaultShell set to powershell.exe requires escaping the command to pass through both my local shell and the Command Prompt on the remote Windows server before PowerShell handles the command. As a simple example, this is the command as run directly inside Powershell:

echo "`"hello`""

Which produces this output:
"hello"

Attempting to run the above command from a Bash script on Linux, I expect to have to use single quotes around the command to avoid Bash interpresting the double-quotes and back-ticks, but not much else:

ssh windows 'echo "`"hello`""'

It's hard to read, but that ends in back-tick, double-quote, double-quote, single-quote. However, I find that this only produces the following output:
hello`

Which seems to be due to the Command Prompt first interpreting the double-quotes before handing it off to PowerShell. Instead, I must quote the command for the Command Prompt and then for Bash in order for it to pass all the way to PowerShell unencumbered. The following command produces the correct output:

ssh windows '"echo \"`\"hello`\"\""'

This is far from intuitive and gets worse as I try to do more complex commands including pipes, where-object and foreach-object using the ?, %, and {} metacharacters.

Expected output
"hello"

Actual output
hello`

[manojampalam]

Behavior wrt to double escaping is the same even with cmd.exe. See the following from Windows client to Windows server.

hello` output seems to the behavior specific to powershell.exe

[penguin359]

My concern isn't about the client-side escaping, it's that I need two levels of server-side escaping. My example still applies when running ssh.exe from a Windows box. When running a remote PowerShell command with ssh, I first have to write it thinking about proper PowerShell syntax, naturally, but then I have to escape it to pass through the remote cmd.exe process that is started by the sshd.exe server on the Windows server which then runs powershell.exe as it's child. Then, as a third layer, I have to escape it to pass through the local cmd.exe shell. It's the middle layer, the remote server cmd.exe that I would like to eliminate. This is in stark contrast to how sshd runs on any UNIX box, whether it's Linux, macOS, or cygwin on Windows layer. If I use csh as my personal shell on a UNIX box, it has a distinct syntax incompatible from standard Bourne shell or it's derivatives like bash. I only have to worry about writing a valid csh command, and then escaping it for my local shell whether that's bash, csh, cmd.exe, or powershell.exe. The command, as passed through the ssh protocol is handed directly to the final shell that interprets it, which I would like to be powershell.exe and not a cmd.exe wrapping powershell.exe.

[penguin359]

Looking at the process hierarchy from an interactive PowerShell session on Windows, I see this:

  Id Level IndentedName             ParentId
  -- ----- ------------             --------
1676     2     sshd                      668
1032     3       sshd                   1676
 984     4         conhost              1032
3852     4         sshd                 1032
1920     5           ssh-shellhost      3852
1336     6             cmd              1920
  84     7               conhost        1336
 960     7               powershell     1336

It does indeed look like PowerShell is being run as a child of cmd.exe. This is an interactive session to a Linux server where the user's shell is csh, and not sh or bash as is typical:

  PID TTY      STAT   TIME COMMAND
18695 ?        Ss     0:00 sshd: user [priv]
18701 ?        S      0:00  \_ sshd: user@pts/86
18714 pts/86   Ss     0:00      \_ -csh
18817 pts/86   R+     0:00          \_ ps axwf

Where as the desired UNIX shell is invoked directly on a traditional UNIX\Linux box. Even if the cmd.exe layer on the server side can't be removed, it would be nice for it to escape commands going through it so it is transparent to the remote user running a command in the selected target shell, PowerShell or otherwise.

[penguin359]

Here's a better step-by-step example to demonstrate the different behavior of sshd.exe on Windows and sshd on a Linux box. The local and remote shell for Linux is bash, but for Windows, I'll use cmd.exe as the local shell for ssh.exe and powershell.exe for the server-side shell called by sshd.exe on Windows. The goal will be to echo out the string "hello" with quotes included in the output. In Bash, the double quotes have a special meaning and need to be escaped. When running the command locally, there are two ways to do it:

echo '"hello"'
echo \"hello\"

To run this remotely from a Linux box, I will need to escape the command so that the local Bash shell passes it through ssh to the remote Linux shell. The simplest mechanism is to use single-quotes with the second command above. Inside single-quotes no characters other than a single-quote, which ends the quoting, have any special meaning. That makes it pretty simple to implement:

ssh linux 'echo \"hello\"'

With just two layers of quoting, I can print "hello". Now, to do the same from a cmd.exe shell running ssh.exe, it's not much different, but the quoting rules are different. With cmd.exe, only two characters have any special meaning in quotes, the double-quote and backslash. To keep the example simple, I'll use the first command from above that only has double-quotes. To quote a double-quote in cmd.exe, it just needs to be doubled up, but the string much already be inside double quotes so I add double quotes around the whole command after doubling each double-quote in the original command:

ssh.exe linux "echo '""hello""'"

It looks a little confusing, but it's just two layers, still, of quoting rules to handle. Now, to do this same action in a local PowerShell, there are again two ways of writing it:

echo '"hello"'
echo `"hello`"

Both commands will print "hello" to the screen when run in a local PowerShell. If I run this from a cmd.exe shell on Windows, I would expect to be able run either by just adding the quoting needed to pass through the local cmd.exe shell. Taking the second command from above, I double up the quotes and then surround the entire command in quotes:

ssh.exe windows "echo `""`"""

But that produces hello` as output, not "hello". It turns out, I need to apply the cmd.exe quoting rules twice, once for the local cmd.exe and once for the remote cmd.exe:

ssh.exe windows """echo `""""`"""""""

So now I have 4 quotes for double-quotes that must appear for PowerShell, and 2 quotes where I'm wrapping the command for the remote cmd.exe giving 7 quotes at the end once I wrap it again for the local cmd.exe. Running the command from a local Linux box to a remote Windows server using PowerShell, I see the same behavior, but this time, I quote the PowerShell command to pass through cmd.exe, and then to pass through Bash by using single-quotes around it:

ssh windows '"echo `""hello`"""'

For every remote PowerShell command I run on Windows, I have to also quote to handle passing through cmd.exe regardless whether it's initiated from a Linux or Window box.

This whole thing above might seem like a silly example because it really is, so I'll give a more realistic example based on what we are actually doing. Here's a pretty short PowerShell command that we would like to run from a Linux box. This is the command as run on the target PowerShell:

Get-NetAdapter | ?{$_.MACAddress -eq "12-34-56-12-34-56"}

And it gives us the information we need as output. Trying this from ssh.exe inside cmd.exe, I know I need to double up the quotes and then surround it in double-quotes. Luckily, there's no backslashes to worry about. So that produces this command:

ssh.exe windows "Get-NetAdapter | ?{$_.MACAddress -eq ""12-34-56-12-34-56""}"

However, I find that this produced a syntax error since I failed to account for the remote cmd.exe interpreting this command. Taking that into account and escaping again produces this command:

ssh.exe windows "Get-NetAdapter | ?{$_.MACAddress -eq """"""12-34-56-12-34-56""""""}"

Which is starting to get hard to read. However, the whole reason I'm using ssh is to be able to run PowerShell commands like this from a Linux box. If I do the same thing from from ssh in Bash, I will run into that same syntax error if I don't first quote the PowerShell command for cmd.exe and then escape it for bash:

ssh windows 'Get-NetAdapter | ?{$_.MACAddress -eq """52-54-00-AB-A7-1F"""}'

I would like to be able to take a working PowerShell command and run it through ssh only worrying about my local shell or scripting language whatever that may be. Currently, I need to always add in this cmd.exe translation layer and don't always get it right.

[manojampalam]

Thank you for the detailed analysis. I understand the issue. Its stemming from the fact that while launching default shell (other than cmd.exe) with PTY, its actually launched via an intermediate cmd.exe and this is consuming the extra escape.

To fix this, we need to internally excape the shell payload before passing it to cmd.exe. AFAIK, we only need to escaple any double quotes ("s), by doubling them. Is this understanding correct?

Here' how the command payload would flow through:

ssh ""command"" -->
"command" (over the wire to sshd) -->
base64encoded("command") to ssh-shellhost -->
"command"(decoded) in ssh-shellhost -->
""command"" (after escape'ing) to cmd.exe --> //this is the escape step that needs to be added
"command" to default shell

@bagajjal thoughts?

[penguin359]

After a little more research, I'm thinking that this might have a little more to do with a fundamental difference in process creation between Windows and UNIX-like operating systems. The command-line is actually passed to it's child process as a single string which the child parses with it's own rules. What I'm seeing as the remote cmd.exe layer of quoting might actually be the C/C++ Runtime library inside PowerShell.exe following these rules:
https://msdn.microsoft.com/en-us/library/17w5ykft.aspx
In that case, it might require shell-host.c to escape the string it receives over SSH to pass through as the argument to /c for the child cmd.exe/powershell.exe. I should also test with cmd.exe and bash.exe and see how the quoting works for those remote shells.

Here one test I tried from a local cmd.exe shell:

powershell.exe /c "Get-NetAdapter | ?{$_.MACAddress -eq \"12-34-56-12-34-56\"}"

Using the expected level of quoting to handle the local invocation of PowerShel.exe and produces the desired output. Turning that into SSH directly, though breaks:

ssh.exe windows "Get-NetAdapter | ?{$_.MACAddress -eq \"12-34-56-12-34-56\"}"

As before, I have to double up the backslashes to handle two layers of C/C++ Runtime parsing the same argument:

ssh.exe windows "Get-NetAdapter | ?{$_.MACAddress -eq \\\"12-34-56-12-34-56\\\"}"

The equivalent test on Linux, for comparison, works with the same quoting level:

bash -c "/sbin/ip -o link | grep \"12:34:56:12:34:56\""
ssh linux "/sbin/ip -o link | grep \"12:34:56:12:34:56\""

[penguin359]

OK, you beat me to my last comment. Thanks for looking into this!

[bagajjal]

In case of no-pty (like this), we directly launch default-shell. Here is the code snippet

In case of pty (interactive ssh session), we launch default-shell through cmd.exe. Here is the code snippet

[manojampalam]

@bagajjal yes. Do you see any issues with escaping double quotes on the pty route?

[bagajjal]

I don't think there is a need to escape double quotes on pty route.
pty route should work without any issues.

In case of no-pty, the issue is the way we pass the received command to createprocessw().

[manojampalam]

Reported issue is with PTY - the fact that with Default Shell configured, we need 2 levels of escaping while with otherwise (cmd.exe), only one level is needed.

[manojampalam]

Reading the complete thread again, I was wrong in certain aspects. While its true that we have an issue in PTY logic as I described before, fixing that would not do any good to the original reported issue.

@bagajjal @penguin359, your analysis is accurate. The issue stems from the way the resulting shell in invoked via CreateProcess and how the command line arguments are interpreted by Windows CRT - https://msdn.microsoft.com/en-us/library/17w5ykft.aspx

I could think of one way to solve this - feed the "command +CRLF + exit + CRLF" via shell's stdin rather than CreateProcess's cmdline. This way the command gets executed as is within shell. The "exit" part is tricky as its again shell specific (shell's may have different quit commands).

[penguin359]

Now that I have a better understanding of what's happening, I think it comes down to a fundamental design difference between UNIX, where commands are broken up into parts and passed in from the parent, and Windows, where they are parsed and broken up by the child. For the no-pty case, where I'm primarily concerned, it seems to be that just escaping according to the rules of the standard MSVCRT parsing should handle the case of a child cmd.exe/powershell.exe shell. Based on the MSDN article @manojampalam posted, I believe this procedure, done in order, should properly protect it:

1. For every backslash in the original command string, insert a second backslash in front (keep them even).
2. For every double-quote in the original command string, insert a backslash in front.
3. Insert a double quote at the front and back of the command string.
4. Combine the shell, option, and command-line, in order and space separated.

From that, this command passed verbatim:

echo "Hello, $Env:USERNAME"

Would turn into the CreateProcess command-line:

powershell.exe /c "echo \"Hello, $Env:USERNAME\""

The difference in the output would be the Hello and username appearing on the same line as one argument to echo instead of two lines as it currently appears.

What do you think?

[danielboth]

@manojampalam @bagajjal, looking at #1212, is there any progress / eta on this issue?

[manojampalam]

Sorry @danielboth none yet. Please check back end of October.

[masaeedu]

@manojampalam

its actually launched via an intermediate cmd.exe and this is consuming the extra escape.

To fix this, we need to internally excape the shell payload before passing it to cmd.exe. AFAIK, we only need to escaple any double quotes ("s), by doubling them. Is this understanding correct?

Is it possible to simply avoid assuming anything about the shell and do no quote escaping at all? Ideally the default shell would receive exactly the same argument list that the ssh executable received on the other end. If some logic is needed to pass arguments to cmd without lots of nested escaping, this can be accomplished by creating wrappers around cmd/powershell/whatever you want and making those the default shell.

[masaeedu]

I created the following .NET console application and set it as the default shell.

using System;

namespace echoargs
{
    class Program
    {
        static void Main()
        {
            Console.WriteLine("Raw invocation:");
            Console.WriteLine(Environment.CommandLine);
        }
    }
}

Then I ran the following in zsh on a Linux client machine.

➜  ~ echo 'cmd /c "cd C:/ && dir"'
cmd /c "cd C:/ && dir"
➜  ~ ssh -i ~/.ssh/test_machine_rsa [email protected] 'cmd /c "cd C:/ && dir"'
Raw invocation:
c:\users\administrator\echoargs\binary\echoargs.dll -c "cmd /c cd" C:/ && dir

It looks like somewhere between the shell passing the arguments to ssh on the client machine and sshd passing the arguments to the default shell on the server, the escaping logic munges the quotes.

Just as a sanity check, I also tried starting the same executable directly on the server with the specified command string:

PS C:\Users\Administrator\echoargs> Start-Process .\binary\echoargs.exe -ArgumentList 'cmd /c "cd C:/ && dir"' -Wait -NoNewWindow
Raw invocation:
C:\Users\Administrator\echoargs\binary\echoargs.dll cmd /c "cd C:/ && dir"

You can see that the argument list string I provided was faithfully preserved, so I think it's unlikely that this is an issue with something inside the process (e.g. MSVCRT) mangling the quotes.

The server is Windows Server 2012 R2 Standard running OpenSSH 7.7.2.0. The client machine is Arch Linux with OpenSSH 7.8p1-1.

[masaeedu]

@penguin359

The command-line is actually passed to it's child process as a single string which the child parses with it's own rules. What I'm seeing as the remote cmd.exe layer of quoting might actually be the C/C++ Runtime library inside PowerShell.exe following these rules

I suspect that might not be the case, because if I write a .NET console application in which I log the raw command line string without any further processing (i.e. Environment.CommandLine) and then set that as the default shell, the quotes are still mangled.

I can try out a C++ application in which I call the relevant Windows API directly, I suspect I'll see the same results.

[masaeedu]

I went ahead and used the Windows API directly using P/Invoke and the results are a little different. The quotes are not being garbled, but they are being nested without any escaping. Here's the updated command line application:

using System;
using System.Runtime.InteropServices;

namespace echoargs
{
    class Program
    {
        static void Main()
        {
            var ptr = GetCommandLine();
            var commandLine = Marshal.PtrToStringAuto(ptr);

            Console.WriteLine(commandLine);
        }
        [DllImport("kernel32.dll", CharSet = CharSet.Auto)]
        private static extern System.IntPtr GetCommandLine();
    }
}

Here's the behavior I see from the Linux client:

➜  ~ echo 'cmd /c "cd C:/ && dir"'
cmd /c "cd C:/ && dir"
➜  ~ ssh -i ~/.ssh/test_machine_rsa [email protected] 'cmd /c "cd C:/ && dir"'
"c:\users\administrator\echoargs\binary\echoargs.exe" -c "cmd /c "cd C:/ && dir""

Here's the event viewer entry for the default shell being started by sshd (I enabled process creation auditing on the server):

I still think that the correct fix would be to remove all quote processing from sshd (including the addition of -c and the wrapping in quotes seen here), so that the command line passed to the SSH client is forwarded verbatim to the default shell. The default shell can prepend /c to the arguments, wrap in quotes, escape things etc. before passing on to a shell program like cmd or powershell if necessary. What do you think @manojampalam?

[manojampalam]

@bingbing8 you have any input to questions above ?

[bingbing8]

@manojampalam and @masaeedu , we have similar discussion on #1273, this is due the the rule of cmd. If you run the belows on cmd prompt:

echo 'cmd /c "cd C:/ && dir"'
'cmd /c "cd C:/ && dir"'

If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:

1.  If all of the following conditions are met, then quote characters
    on the command line are preserved:

    - no /S switch
    - exactly two quote characters
    - no special characters between the two quote characters,
      where special is one of: &<>()@^|
    - there are one or more whitespace characters between the
      two quote characters
    - the string between the two quote characters is the name
      of an executable file.

2.  Otherwise, old behavior is to see if the first character is
    a quote character and if so, strip the leading character and
    remove the last quote character on the command line, preserving
    any text after the last quote character.

@masaeedu , we still need to add quotes on client side, createprocess failed for those arguments with spaces in it. like #1211.
on serverside, if we take your approach (rely on custom default to process the argument), there is no existing shell directly work. user always required to write their own shell app, right? also It will totally depends on the custom default shell to choose which shell program to use.

[masaeedu]

@bingbing8 I didn't think about having Windows SSH on the client side, you're right.

Regarding the server side, yes, a custom shell app will be needed to wrap most shells, but you could just write and bundle these for the common shells, or allow the community to write them. I think having SSHD itself transparently propagate the arguments and not interfering with them provides the greatest degree of extensibility.

For my use case I need to be able to pass some arguments using NodeJS that will arrive in a remote NodeJS process in exactly the same form; figuring out how to formulate them so that they survive the SSHD wrapping/escaping process is a little tricky, so it would be easier if that extra layer didn't exist at all.

[bingbing8]

@masaeedu raised a good point. For those custom shell, which might have different rules of quotes and escaping than cmd or powershell, user might need to receive the raw cmdline from sshd. But I still think we need to process quotes and spaces to make common shells to work. How about we escape/process the commands for cmd.exe and powershell.exe, and for other customer shells, leave community and user to process?

int main()
{
    LPWSTR cmdline = GetCommandLineW();
    wprintf(L"Command line: %s\n", cmdline);

    int nArgs;
    LPWSTR *szArglist = CommandLineToArgvW(cmdline, &nArgs);
    if (NULL == szArglist)
    {
        wprintf(L"CommandLineToArgW failed\n");
        return 0;
    }
    else
    {
        for (int i = 0; i < nArgs; i++)
        {
            wprintf(L"argv[%d]: %s\n", i, szArglist[i]);
        }
    }
    LocalFree(szArglist);
    return 0;
}

[masaeedu]

@bingbing8 I think it would be a good idea to put the escaping logic for built in shells like cmd and powershell into small executables as well. These can be built and distributed alongside win32-openssh and set up as the DefaultShell out of the box. It seems to me this would make things easier and more consistent in terms of implementation, since you don't need to inspect the shell or command string and make decisions inside SSHD. There is however the marginal cost of an additional process call.

It's essentially an implementation detail though, so as an end user it wouldn't matter if it was baked into SSHD either (so long as it doesn't accidentally catch and process command strings intended for other shells).

[bingbing8]

@maertendMSFT, we still need to add double quotes to the executable, in this case, the full path to the default shell. otherwise, it may not even find the path to the executable.

[bingbing8]

@masaeedu, here is what will do,

1. for existing common shells, like powershell, bash, cygwin, we add double quotes to the path of default shell and its arguments, escape the double quotes and backslash inside args.
2. For cmd and ssh-shellhost.exe, we simplily add double quotes to shell and its arguments without any escaping.
3. For shell other than above, we add a registry value DefaultShellEscapeArguments" under "HKEY_LOCAL_MACHINE\SOFTWARE\OpenSSH". The default value when missing is 1, which mean the escaping will happen like no.1 above. If user does not want escaping, they need to explicitly set this value to 0. in later case, only double quotes are added to shell and args, like no.2 above.

The PR is at: PowerShell/openssh-portable#349

[unhammer]

@bingbing8 will 1. help with git clone windowsmachine/c/Users/user/repo.git turning into ''/c/Users/user/repo.git'' (extra single-quotes)? Cf. https://stackoverflow.com/questions/53834304/how-do-i-git-clone-from-a-windows-machine-over-ssh/53847484

[bingbing8]

@unhammer, openssh code does not add extra single-quotes. I believe the single quotes are added by git. I close this double quotes escaping issue now. If want, we can discuss in a different issue for the extra single quotes issue.

[unhammer]

That make sense – I managed to avoid the issue by switching the openssh defaultshell from cmd.exe to bash :)

[bingbing8]

@unhammer, look like the issue you saw are due to the issue single quoted are not interpreted as file path by cmd and ssh-shellhost.exe. singe quotes are added by git client, but cmd.exe and ssh-shellhost.exe does not recognize the quoted string as file path. #895.
Please refer to Setting up a Git server on Windows using Git for Windows and Win32_OpenSSH for workaroud. setting default shell to powershell should also works.

[bielawb]

Hi

When can we expect that fix in the supported version (Windows Server 2019)?
I don't feel comfortable running beta versions in production environments...

Regards
Bartek

[cpriest]

Here's a quick workaround AutoHotkey script that takes care of the problem for me:
https://gist.github.com/cpriest/b08543648ba513d977e6c54e23f9f682