m2-bash: looses PATHs, prevents any use as a shell to run other commands from conda

[yarikoptic]

(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>echo %PATH%
C:\Users\DataLad\miniconda3\envs\datalad-2;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\mingw-w64\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\usr\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Library\bin;C:\Users\DataLad\miniconda3\envs\datalad-2\Scripts;C:\Users\DataLad\miniconda3\envs\datalad-2\bin;C:\Users\DataLad\miniconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Git\cmd;C:\Users\DataLad\AppData\Local\Microsoft\WindowsApps;.

(datalad-2) C:\Users\DataLad\AppData\Local\Temp\datalad_temp_tree_hufbz7i1>bash -c "echo $PATH"
/c/Users/DataLad/miniconda3/envs/datalad-2:/mingw-w64/bin:/usr/bin:/bin:/c/Users/DataLad/miniconda3/envs/datalad-2/Scripts:/c/Users/DataLad/miniconda3/envs/datalad-2/bin:/c/Users/DataLad/miniconda3/condabin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/Program Files/Git/cmd:/c/Users/DataLad/AppData/Local/Microsoft/WindowsApps:.

you can see that C:\Users\DataLad\miniconda3\envs\datalad-2\Library\usr\bin and C:\Users\DataLad\miniconda3\envs\datalad-2\Library\bin are not translated, so anything (e.g. 7z from 7zip) installed in those paths, is not accessible inside that bash session. That prevents any reliance on bash as the general shell to execute commands from conda.

Moreover, even conda is not executable, likely because it is originally conda.bat so it might be a different issue altogether.

[mingwandroid]

Please provide detailed reproduction instructions.

[yarikoptic]

here is an example with trying to use 7z from outside and within bash

(base) C:\Users\DataLad\Downloads>conda create -n try7zip -c conda-forge 7zip
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.9.1

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: C:\Users\DataLad\miniconda3\envs\try7zip

  added / updated specs:
    - 7zip


The following NEW packages will be INSTALLED:

  7zip               conda-forge/win-64::7zip-19.00-h74a9793_2
  vc                 conda-forge/win-64::vc-14.1-h869be7e_1
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.16.27012-h30e32a0_2


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate try7zip
#
# To deactivate an active environment, use
#
#     $ conda deactivate


(base) C:\Users\DataLad\Downloads>conda activate try7zip

(try7zip) C:\Users\DataLad\Downloads>7z --version


Command Line Error:
Unknown switch:
--version

(try7zip) C:\Users\DataLad\Downloads>conda install m2-bash
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.9.1

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: C:\Users\DataLad\miniconda3\envs\try7zip

  added / updated specs:
    - m2-bash


The following NEW packages will be INSTALLED:

  m2-bash            pkgs/msys2/win-64::m2-bash-4.3.042-5
  m2-msys2-runtime   pkgs/msys2/win-64::m2-msys2-runtime-2.5.0.17080.65c939c-3
  msys2-conda-epoch  pkgs/msys2/win-64::msys2-conda-epoch-20160418-1


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

(try7zip) C:\Users\DataLad\Downloads>bash -c 7z --version
bash.exe: warning: could not find /tmp, please create!
--version: 7z: command not found

so distilled set of commands to try

conda create -n try7zip -c conda-forge 7zip
conda activate try7zip
conda install m2-bash
7z  # would work
bash -c 7z  # would not find it

[mingwandroid]

It is required to use -lc here, this is an MSYS2-ism.

[mingwandroid]

Also you must set MSYS2_PATH_TYPE to inherit, again, an MSYS2-ism.

[yarikoptic]

Well, that would still preclude any "seamless" use of bash as interpreter for shell scripts which are already there and used by conda installed tools - we cannot modify shebangs for all of them to carry -l and then modify all the tools to export environment variable.

Moreover, it seems still to not work:

(try7zip) C:\Users\DataLad\Downloads>set MSYS2_PATH_TYPE=inherit

(try7zip) C:\Users\DataLad\Downloads>bash -lc "7z --help"
bash: 7z: command not found       

[mingwandroid]

OK, I'll look into this when I am back on Windows.

[dhirschfeld]

FYI: It's a somewhat common idiom on conda-forge to use build.sh to actually build a package and treat bld.bat as a shim which just calls bash -lc "./build.sh"

When doing so it first sets the env vars Ray mentioned:
https://github.com/conda-forge/dotnet-feedstock/blob/6f344ecc9dd46904045500cd9829ba850e6abd98/recipe/build-runtime.bat#L2-L11

This seems to work well for that use-case at least.

[yarikoptic]

@dhirschfeld thank you for the pointer! in our case entry points would be git, git-annex and datalad calls... may be at some point I should look into making .bat shims which would have the bash -lc invocation (and invoked only when not already in bas) thus jumping entire "tree" of processes into that bash "logged in" environment... may be that would provide some remedy.

[yarikoptic]

@mingwandroid Any guidance on how to make bash -l (want "interactive" one) to inherit PATHs properly , as I have mentioned above even with MSYS2_PATH_TYPE=inherit it doesn't inherit paths such as ...\Library\bin leading to many conda-installed tools to not be found.

[mattiasljungstrom]

I'm also trying to get interactive bash to work on win10/conda. It looks like "/c/Users/USERNAME/Miniconda3/Library/bin" is missing in bash. Maybe other too, but that's where git is installed for instance.

Adding PATH=$PATH:/c/Users/USERNAME/Miniconda3/Library/bin to .bashrc is my current workaround.