"opam config env" returns cygwin PATH without --sexp and MinGW paths with --sexp

[MSoegtropIMC]

Dear Opam for Windows Team,

opam config env has a strange difference when called with and without --sexp:

MinGW PATH with --sexp

$ opam config env --safe --sexp
(
  ("OPAM_SWITCH_PREFIX" "C:\\bin\\cygwin_coq64_ms_dev_opam2\\home\\soegtrop\\.opam\\ocaml-variants.4.07.1+mingw64c")
  ("OCAMLLIB" "C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/ocaml")
  ("CC" "x86_64-w64-mingw32-gcc.exe")
  ("CPP" "x86_64-w64-mingw32-cpp.exe")
  ("CAML_LD_LIBRARY_PATH" "C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/stublibs;C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/ocaml/stublibs:C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/ocaml:C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/stublibs")
  ("OCAML_TOPLEVEL_PATH" "C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/toplevel")
  ("MANPATH" ":/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/man;C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/man")
  ("PATH" "C:\\bin\\cygwin_coq64_ms_dev_opam2\\home\\soegtrop\\.opam\\ocaml-variants.4.07.1+mingw64c\\bin;C:\\bin\\cygwin_coq64_ms_dev_opam2\\usr\\local\\bin;C:\\bin\\cygwin_coq64_ms_dev_opam2\\bin;C:\\bin\\coq64_ms_dev_opam2\\bin;C:\\bin\\cygwin_coq64_ms_dev_opam2\\usr\\x86_64-w64-mingw32\\sys-root\\mingw\\bin;C:\\Windows\\system32;C:\\Windows")
)

vs cygwin PATH without --sexp

$ opam config env --safe
OPAM_SWITCH_PREFIX='C:\\bin\\cygwin_coq64_ms_dev_opam2\\home\\soegtrop\\.opam\\ocaml-variants.4.07.1+mingw64c'; export O
PAM_SWITCH_PREFIX;
OCAMLLIB='C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/ocaml'; export OCAMLLI
B;
CC='x86_64-w64-mingw32-gcc.exe'; export CC;
CPP='x86_64-w64-mingw32-cpp.exe'; export CPP;
CAML_LD_LIBRARY_PATH='C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/stublibs;C
:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/ocaml/stublibs:C:/bin/cygwin_coq6
4_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/ocaml:C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtr
op/.opam/ocaml-variants.4.07.1+mingw64c/lib/stublibs'; export CAML_LD_LIBRARY_PATH;
OCAML_TOPLEVEL_PATH='C:/bin/cygwin_coq64_ms_dev_opam2/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/lib/toplevel';
export OCAML_TOPLEVEL_PATH;
MANPATH=':/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/man:/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/ma
n'; export MANPATH;
PATH='/home/soegtrop/.opam/ocaml-variants.4.07.1+mingw64c/bin:/usr/local/bin:/usr/bin:/cygdrive/c/bin/coq64_ms_dev_opam2
/bin:/usr/x86_64-w64-mingw32/sys-root/mingw/bin:/cygdrive/c/Windows/system32:/cygdrive/c/Windows'; export PATH;

This has the effect that e.g. user-setup for emacs does not work because the opam-user-setup.el file installed by opam user-setup uses the --sexp form and then sets the PATH variable to teh MinGW path after which no command is found any more (all further shell commands return 127).

I would say for this opam living between two worlds, it would be best to have an option if MinGW or cygwin paths are wanted. Depending on the situation both might make sense. As default I would probably use cygwin paths since I guess the intention is that opam is mostly used from cygwin, just the created executables are used from MinGW.

Also not so nice is that \ and / widely mixed and that in the non --sexp case MinGW and cygwin paths are mixed. In woudl recommend to always use / since native Windows APIs do understand /.

Btw.: I am the maintainer of the Coq Windows build (since 4 years or so) and think about switching to your opam for building the official Coq releases for Windows. Currently we use shell scripts which replicate a lot of the information already known to opam. Definitely I want to use it to setup the Coq Windows developer environment and to use development builds of Coq.

[fdopen]

The current behavior was deliberately chosen.

The output of opam config env --safe is intended to be used inside (unix) shell scripts. Therefore some environment variables like PATH (see https://cygwin.com/cygwin-ug-net/setup-env.html ) are converted to a format that is expected by cygwin; all other variables are left untouched.

The output of opam config env --safe --sexp can however be used by all programs. My guess was, that most of them will not be installed through cygwin, but through opam install .... They are native windows programs and don't understand cygwin's conventions. So no conversion is applied in this case.

You are probably trying to use the emacs build, that is shipped by cygwin.com. However, there is a also a native Windows version of Emacs. The former probably doesn't understand windows paths, the latter won't understand cygwin conventions. Does cygwin's emacs work with a native windows version of merlin? ocamlformat? utop? I can add a switch for easier integration with cygwin based tools, but I'm not sure, if cygwin's emacs is a wise choice and if similar situations will be a common use case. (sexp expressions are not commonly used for such purpose outside the OCaml community)

Regarding the strange mixture of forward slashes and backslashes: Variables like CAML_LD_LIBRARY_PATH are set through opam files (see e.g.: https://github.com/fdopen/opam-repository-mingw/blob/e9b5b29ab38cb6b638fe88d0620d1bbe907dc269/packages/ocaml-variants/ocaml-variants.4.07.1%2Bmingw64/opam). If pre-configured variables like %{lib}% would expand to paths with backward slashes, this would break a lot build instructions (shell-scripts, Makefiles,...), that always never quote their arguments properly (at the time I made this decision, dune didn't exists and custom Makefiles and configure scripts were much more common). Other variables are however (partially) inherited or they are set by the user through opam config set .... They might use backslashes instead. I could enforce forward slashes for special variables like PATH, but in the general case I don't know anything about these variables and were they came from. I can't know, when it is safe to convert backslashes to forward slashes and vice versa. (backslashes and forward slashes are not fully compatible. The Windows API often only accepts backslashes, cmd.exe can't properly deal with forward slashes, ...)

[MSoegtropIMC]

Many thanks for the detailed explanations! It is indeed a good question if one should use a native Windows or a cygwin emacs. Just FYI: the cygwin emacs does work with the native merlin after patching the emacs config files generated by opam user-setup a bit. I did just quick tests, though. But it is interesting to know that you recommend using the native Emacs.

In general my preferred setup would be one where all build tools are native cygwin applications (say ocamlc, opam, dune, remake, make) and are creating MinGW binaries in a cross compilation sense, similar to the MinGW gcc provided by cygwin. In such a setup a cygwin emacs makes most sense. This would of cause require two ocaml compilers, one for build tools and one for resulting binaries. And in the Coq world possibly a third one which is a MinGW app when one wants to have native compute in Coq.

Interestingly ocamlc, opam and dune as created by this opam repo are native Windows apps but still can be used reasonably well as if they would be cygwin apps since they mostly use relative paths. There are only very minor hickups. But since obviously this isn't your working model, it does definitely make sense for me to explore a "as little cygwin as possible" workflow, which would lead to a native Windows Emacs. I am currently in an experimentation phase and am not yet sure what works best.

May I ask where - from a design point of view - you draw the line between cygwin and MinGW? What should be cygwin apps and what should be MinGW apps in your opinion? E.g. do you use a cygwin git or a MinGW git?

Back to the topic: I think at least for the kind of experiments I am doing, that is shifting around the border between cygwin and MinGW, it would make sense to have a bit more flexibility, and possibly also predictability, in what path format configuration queries to opam return. So something like a --cygwin and --mingw option which would lead to /cygdrive/c/bin and C:/bin results would be most helpful. I fully agree that only in very rarely makes sense to use backward slashes. My comment was more towards "always use forward slashes and don't mix". Still a --windows option, which then returns only backward slashes, also might make sense in some cases.

I will follow your advice and see how life with a native Windows Emacs is.

[fdopen]

I've now played around with user-setup ( ca6263d and 37ea05d ) and at least temporary added a --cygwin switch to opam env --sexp.
I'm not longer convinced, that such a switch is helpful. If you need to patch your files anyway, in order to distinguish between usage with native programs and usage with cygwin, you can also modify your files in such a way, that the result is piped thorough cygpath for the cygwin case.

May I ask where - from a design point of view - you draw the line between cygwin and MinGW? What should be cygwin apps and what should be MinGW apps in your opinion? E.g. do you use a cygwin git or a MinGW git?

Native tools usually have better performance, cygwin gives you greater conformity to Unix standards. The graphical installer will installs cygwin's version of git. But this is just for easier maintenance. Git for Windows is faster, but it ships its own, customized fork of cygwin. You would end up with different version of various tools (git, bash,...) in your PATH. That's a possible source of errors. It's easier to just test everything with one environment. (WSL wouldbe the better choice for many use cases anyway, but WSL is not available for all Windows versions ...).

[MSoegtropIMC]

Thanks, I will experiment with the new option and try your changes to user-setup and merlin. It is indeed hard to tell in which place this is best fixed. Right now I hack the scripts with patch and sed after creation, which is for sure least elegant and most fragile.

I followed your advice and tried the native emacs. It works, but what I don't like is that it installs almost a complete Linux - more than 30.000 files, quite heavy weight even for an advanced editor. It is similar to the MinGW git. As you say, one ends up with 10 Linux like systems on Windows.

WSL is definitely an option. What I like about cygwin is that it is lightweight and that I can easily have 10 cygwins for different uses / experiments in parallel on one system. E.g. I tend to have one for development which is quite rich and another one for making releases which is as streamlined as possible to ensure I know what contributes to a release (important to fulfill open source requirements) and also for fast operation in CI. A release build of Coq is always done on a completely fresh cygwin to ensure reproducibility. This is way harder to achieve with WSL (well not that I really tried ...).