Escaping Vim system() Calls

You can run non-interactive shell commands in Vim via the system(command, input) function. The commands must of course be escaped correctly for the shell, and different shells have different escaping rules.

In constructing and running a command, two levels of escaping are done:

1. You have to use shellescape() to escape special characters in the arguments, e.g. file paths, of the command you pass to system().
2. Vim escapes the entire command appropriately for your shell.

This works pretty well in recent versions of Vim on non-Windows systems. Windows is problematic because its shells' escaping rules are odd.

The way Vim escapes commands for your shell has evolved over time. Consider these patches for example:

Patch	Description
7.3.443	MS-Windows: 'shcf' and 'shellxquote' defaults are not very good. Make a better guess when 'shell' is set to "cmd.exe".
7.3.445	Can't properly escape commands for cmd.exe. Default 'shellxquote' to '('. Append ')' to make '(command)'. No need to use "/s" for 'shellcmdflag'.
7.3.446	Win32: External commands with special characters don't work. Add the 'shellxescape' option.
7.3.447	Win32: External commands with "start" do not work. Unescape part of the command.
7.3.448	Win32: Still a problem with "!start /b". Escape only '|'.
7.3.450	Win32: Still a problem with "!start /b". Fix pointer use.

You can read some of the discussion around the patches if you're feeling brave, and a follow-up.

The problem for plugin authors is using system() in a way that works whatever version of Vim their users have.

The only way to do it is to handle as much of the escaping in your own VimL code, effectively backporting Vim's escaping logic. This is non-trivial and, as far as I can tell, there is no consensus on the best approach.

I would love to see a definitive shell-escaping plugin which everybody else can depend on once and for all. It's silly for every plugin author to reinvent the wheel, especially when it's so tricky to get right.

Below we look at how several popular plugins handle escaping. But first here's an overview of how Vim constructs the commands it passes to the shell.

How Vim Constructs Shell Commands

The command executed in constructed using several options (source: system() documentation):

'shell' 'shellcmdflag' 'shellxquote' command 'shellredir' tmp 'shellxquote'

– where command is the string you passed to system() and tmp is an automatically generated file name. On Unix braces are put around command to allow for concatenated commands.

For example, when I run :echo system('ls') using Vim 7.4.052 with Bash on OS X:

/bin/bash -c "(ls) >some_tmp_file 2>&1"

You can see everything after shellcmdflag by setting Vim's verbosity to anything greater than 3: set verbose=4.

Here are the relevant options, as of 7818ca6de3d0 (11 December 2013):

(Shell constraints are shown in square brackets.)

Option	Description	Default Unix	Default Windows
shell	Name of the shell to use	$SHELL or sh	command.com or cmd.exe
shellcmdflag	Flag passed to the shell	-c	[contains sh]: -c; otherwise /c
shellpipe	String to use to put output of :make in error file	[csh, tcsh]: |& tee; [sh, ksh, mksh, pdksh, zsh, bash]: 2>&1| tee; otherwise | tee	>
shellquote	Quoting character(s) surrounding command passed to shell excluding redirection		[contains sh]: "
shellredir	String to use to put output of a filter command in a temporary file	[csh, tcsh, zsh]: >&; [sh, ksh, bash]: >%s 2>&1; otherwise >	[cmd]: >%s 2>&1; same as unix
shellslash	Only when a backslash can be used as a path separator: when set, a forward slash is used when expanding file names. Useful when a Unix-like shell is used on Windows.	off	off
shelltemp	When set, use temp files for shell command; otherwise use a pipe	on	on
shelltype	(Amiga only)	off	off
shellxescape	When shellxquote is set to (, the characters listed in this option will be escaped with ^		"&|<>()@^
shellxquote	Quoting character(s) surrounding command passed to shell including redirection	when using system(): "	[cmd.exe]: (; [contains sh]: "

You also need to know the rules for including whitespace in a string option value.

And here's the shellescape(str) function, as of 350272cbf1fd (23 January 2014):

Escape str for use as a shell command argument. On Windows, when shellslash is not set, it encloses str in double quotes and doubles all double quotes within str. For other systems, it encloses str in single quotes and replaces all ' with '\''.

Overall there are quite a few moving parts to account for.

How Several Popular Plugins Handle Escaping

I think it instructive to examine how several popular plugins handle escaping. Popular plugins are, by definition, widely used and therefore have had to learn to cope with the wide variety of Vim versions and shells in the wild. As you will see, they take different approaches ;)

vim-dispatch

vim-dispatch provides a function to escape a command invoked on Windows with silent execute '!start cmd.exe ...':

function! s:escape(str)
  if &shellxquote ==# '"'
    return '"' . substitute(a:str, '"', '""', 'g') . '"'
  else
    let esc = exists('+shellxescape') ? &shellxescape : '"&|<>()@^'
    return &shellquote .
          \ substitute(a:str, '['.esc.']', '^&', 'g') .
          \ get({'(': ')', '"(': ')"'}, &shellquote, &shellquote)
  endif
endfunction

The if block doubles up any " characters when shellxquote is " – which sounds like shellescape() on Windows when shellslash is off. The else block implements shellxescape's escaping rules.

vim-fugitive

vim-fugitive implements its own shellescape(arg):

function! s:shellesc(arg) abort
  if a:arg =~ '^[A-Za-z0-9_/.-]\+$'
    return a:arg
  elseif &shell =~# 'cmd'
    return '"'.s:gsub(s:gsub(a:arg, '"', '""'), '\%', '"%"').'"'
  else
    return shellescape(a:arg)
  endif
endfunction

And its own s:fnameescape(file):

function! s:fnameescape(file) abort
  if exists('*fnameescape')
    return fnameescape(a:file)
  else
    return escape(a:file," \t\n*?[{`$\\%#'\"|!<")
  endif
endfunction

Finally, here is an excerpt from the s:ReplaceCmd() function:

if &shell =~# 'cmd'
  let cmd_escape_char = &shellxquote == '(' ?  '^' : '^^^'
  call system('cmd /c "' . prefix . s:gsub(a:cmd,'[<>]', cmd_escape_char.'&') . ' > ' . tmp . '"')
else
  call system(' (' . prefix . a:cmd . ' > ' . tmp . ') ')
endif

This looks a little like an implementation of shellxescape combined with shell and shellcmdflag.

vundle

vundle provides a Windows-aware cd function:

func! g:shellesc(cmd) abort
  if ((has('win32') || has('win64')) && empty(matchstr(&shell, 'sh')))
    if &shellxquote != '('      " workaround for patch #445
      return '"'.a:cmd.'"'      " enclose in quotes so && joined cmds work
    endif
  endif
  return a:cmd
endf

func! g:shellesc_cd(cmd) abort
  if ((has('win32') || has('win64')) && empty(matchstr(&shell, 'sh')))
    let cmd = substitute(a:cmd, '^cd ','cd /d ','')  " add /d switch to change drives
    let cmd = g:shellesc(cmd)
    return cmd
  else
    return a:cmd
  endif
endf

The g:shellesc() function looks like a partial implementation of shellescape() when shellslash is off.

The g:shellesc_cd() function adds the /d switch to support Windows' drives and then calls g:shellesc().

It is invoked in the s:sync() function as follows (omitting some irrelevancies):

if ...
  let cmd = 'cd '.shellescape(a:bundle.path()).' && git pull && git submodule update --init --recursive'
  let cmd = g:shellesc_cd(cmd)
  let get_current_sha = 'cd '.shellescape(a:bundle.path()).' && git rev-parse HEAD'
  let get_current_sha = g:shellesc_cd(get_current_sha)
  let initial_sha = s:system(get_current_sha)[0:15]
else
  let cmd = 'git clone --recursive '.shellescape(a:bundle.uri).' '.shellescape(a:bundle.path())
endif
let out = s:system(cmd)

vim-signify

vim-signify provides a replacement for shellescape() for use with file paths:

function! sy#util#escape(path) abort
  if exists('+shellslash')
    let old_ssl = &shellslash
    if fnamemodify(&shell, ':t') == 'cmd.exe'
      set noshellslash
    else
      set shellslash
    endif
  endif

  let path = shellescape(a:path)

  if exists('old_ssl')
    let &shellslash = old_ssl
  endif

  return path
endfunction

An example invocation from autoload/sy/repo.vim is:

let root = finddir('.git', fnamemodify(b:sy.path, ':h') .';')
let root = fnamemodify(root, ':h')
let output = system('cd '. sy#util#escape(root) .' && git diff --numstat')

The justifications for setting noshellslash before invoking shellescape() were:

But many people use [shellslash] even with cmd.exe because it lets you use forward slashes within Vim easier, and most (but not all) Windows programs in cmd.exe will work with both forward and backwards slashes.

It would probably be best, if you also test the value of 'shell' to see whether it actually is still set to start with "command" or "cmd"; shellslash can remain set if a Unix-like shell is actually being used. The reason for resetting 'shellslash' if cmd.exe or similar is in use, is that shellescape() assumes a Unix-like shell if shellslash is set.

Source: issue #15

And:

So, basically, when one starts Vim from Command Prompt (cmd.exe), then &shell points to cmd.exe indeed, and paths for system() call have to be escaped with noshellslash. However, when one starts Vim from Sh, Bash, Ksh, Csh, etc., then &shell points to one of them, and, in this case, paths for system() call have to be escaped with shellslash, otherwise everything breaks because of backward slashes \ in paths.

Source: issue #99

What Now?

I would like a VimL replacement for shellescape(str) and possibly one for escaping the entire command passed to system().

These should work on Unix and Windows regardless of the user's Vim version.

Reading the discussions on the vim-dev mailing list, it feels like a perfect solution is unlikely. And a lot depends on the complexity of the commands invoked with system().

However I hope we can get 90%-95% of the way there.

What Can You Do?

Please send me your suggestions!