./configure fails on Windows/MSYS2

[petrochenkov]

The PR causing this is #34055
cc @brson @alexcrichton

How to reproduce: simply run ./configure on fresh or not so fresh clone of Rust repo using relatively recent MSYS2 (msys2-x86_64-20160205).
cmake package used: mingw64/mingw-w64-x86_64-cmake 3.4.1-1
Errors:

-- Constructing LLVMBuild project information
CMake Error: File /home/we/rust/src/llvm/LLVMBuild.txt does not exist.
CMake Error at C:/msys64/home/we/rust/x86_64-pc-windows-gnu/llvm/LLVMBuild.cmake:28 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:489 (include)


CMake Error: File /home/we/rust/src/llvm/bindings/LLVMBuild.txt does not exist.
CMake Error at C:/msys64/home/we/rust/x86_64-pc-windows-gnu/llvm/LLVMBuild.cmake:30 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:489 (include)


// ...
// Lots of "File does not exist" errors are skipped
// ...

CMake Error: File /home/we/rust/src/llvm/utils/llvm-build/llvmbuild/componentinfo.py does not exist.
CMake Error at C:/msys64/home/we/rust/x86_64-pc-windows-gnu/llvm/LLVMBuild.cmake:362 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:489 (include)


CMake Error: File /home/we/rust/src/llvm/utils/llvm-build/llvmbuild/util.py does not exist.
CMake Error at C:/msys64/home/we/rust/x86_64-pc-windows-gnu/llvm/LLVMBuild.cmake:364 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:489 (include)


-- LLVMHello ignored -- Loadable modules not supported on this platform.
-- Targeting X86
-- Targeting ARM
-- Targeting AArch64
-- Targeting Mips
-- Targeting PowerPC
-- BugpointPasses ignored -- Loadable modules not supported on this platform.
-- Configuring incomplete, errors occurred!
See also "C:/msys64/home/we/rust/x86_64-pc-windows-gnu/llvm/CMakeFiles/CMakeOutput.log".
See also "C:/msys64/home/we/rust/x86_64-pc-windows-gnu/llvm/CMakeFiles/CMakeError.log".
configure: error: LLVM cmake configure failed

All the files reported by CMake as missing actually exist, so there is probably some mis-translation between Linux-style msys file paths and Windows paths or something like that.

Non-mingw cmake package (msys/cmake 3.2.3-1) gives other errors (EDIT: irrelevant):

configure:  -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_DOCS=OFF -DLLVM_ENABLE_ZLIB=OFF -DWITH_POLY=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_ENABLE_LIBEDIT=OFF -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ '-DCMAKE_C_FLAGS=' '-DCMAKE_CXX_FLAGS=' -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_TARGETS_TO_BUILD='X86;ARM;AArch64;Mips;PowerPC' -G 'Unix Makefiles' /home/we/rust/src/llvm/
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- The ASM compiler identification is GNU
-- Found assembler: /c/mingw-w64/x86_64-4.9.3-win32-seh-rt_v4-rev1/mingw64/bin/gcc.exe
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Check for working C compiler: /c/mingw-w64/x86_64-4.9.3-win32-seh-rt_v4-rev1/mingw64/bin/gcc.exe
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Check for working C compiler: /c/mingw-w64/x86_64-4.9.3-win32-seh-rt_v4-rev1/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /c/mingw-w64/x86_64-4.9.3-win32-seh-rt_v4-rev1/mingw64/bin/g++.exe
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Check for working CXX compiler: /c/mingw-w64/x86_64-4.9.3-win32-seh-rt_v4-rev1/mingw64/bin/g++.exe -- works
-- Detecting CXX compiler ABI info
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for dirent.h
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Looking for dirent.h - found
-- Looking for dlfcn.h
System is unknown to cmake, create:
Platform/MINGW64_NT-6.3 to use this system, please send your config file to [email protected] so it can be added to cmake
-- Looking for dlfcn.h - not found

// ...
// etc
// ...

[petrochenkov]

Non-MSYS2 cmake from cmake.org installed globally gives the same errors as mingw-w64-x86_64-cmake 3.4.1-1

[petrochenkov]

Ha, it's actually Python is at fault here, not CMake.
Any CMake seems to work only with native Windows paths, however the path list [/home/we/rust/src/llvm/LLVMBuild.txt, /home/we/rust/src/llvm/bindings/LLVMBuild.txt, ...] is created by a Python script (llvm\utils\llvm-build\llvmbuild\main.py) and the format of paths used in this list depends on used version of Python. MSYS2's Python used by default outputs paths in Unix-style format, while native Windows Python outputs Windows-style paths.

So the solution is to make the LLVM build system to use (roughly speaking) "C:\Python27\python.exe" instead of "/usr/bin/python2" somehow. I solved this locally by hard-coding the PYTHON_EXECUTABLE CMake variable in LLVM cmake files to my Python path, but this is certainly not an appropriate solution.
I'm also not sure why ./configure works on buildbots without going through all these tricks. Waiting for what @alexcrichton will say.

[retep998]

@petrochenkov Maybe the buildbots just don't have msys python installed, or if they do they have the standard windows python in the PATH first so it gets used instead of msys python?

[petrochenkov]

@retep998

Maybe the buildbots just don't have msys python installed

This is very likely. If we look at some random buildbot logs, like http://buildbot.rust-lang.org/builders/auto-win-gnu-64-opt/builds/4667/steps/configure/logs/stdio we see than a custom version of Python is not passed to configure, but CFG_LLDB_PYTHON is detected as /c/Python27/python2.7 and the way configure detects Python puts MSYS2 Python first if it's installed, because MSYS2 puts its own paths before inherited Windows paths in $PATH, so MSYS2 Python is not installed on buildbots.

However, Python is installed with base-devel and that means that buildbot setup either picks tools like make one by one and not in a base-devel batch, or uninstalls Python after installing base-devel. Both ways differ from README.md recommendations and common practice and thus put buildbots further from reality.

[retep998]

@petrochenkov Which is why I don't bother building Rust with the configure/make hell and just invoke rustbuild directly. Solves all sorts of problems, such as this issue.

[petrochenkov]

@retep998
rustbuild doesn't affect the way in which LLVM build system finds Python (it finds it with FindPythonInterp using find_program unless PYTHON_EXECUTABLE explicitly specified), so ./configure --enable-rustbuild && make <anything> fails too with the same errors.

[retep998]

@petrochenkov See, you're still using configure/make hell when you do it like that. I'm invoking python .\src\bootstrap\bootstrap.py directly and I'm doing it in a standard Windows command line without any of the msys/mingw stuff in my path.

[brson]

Maybe the buildbots just don't have msys python installed

Yes, buildbots are using windows python.

[brson]

I can't reproduce it on my own local install using whatever old msys2 I'm on. Have to set up a new env.

[alexcrichton]

@petrochenkov interesting! This sounds like what this block is trying to catch as this error looks very familiar to me. (tracked down to python when we failed to build LLVM awhile ago as well)

[alexcrichton]

Note that if the default instructions on how to build Rust on Windows are now broken in the README that's pretty serious and we should try to fix that pronto (e.g. correct the docs).

@petrochenkov I'm curious, does that check not work for you? That is, is it not firing when it perhaps should be firing?

[petrochenkov]

This sounds like what this block is trying to catch

I must be blind, I never noticed this check o_O

I'm curious, does that check not work for you?

It fires only for MSVC flavor builds, but after #34055 the problem exists for GNU builds as well (I use GNU, readme instructions also refer to GNU).

[petrochenkov]

Note that if the default instructions on how to build Rust on Windows are now broken in the README that's pretty serious and we should try to fix that pronto (e.g. correct the docs).

I think what should be done is 1) docs fixed 2) Python version check performed for GNU builds 3) my_python from ./configure --python="my_python" should be passed to LLVM build so people could use MSYS2 Python, but override it when building Rust.

If you are building under msys2 try installing the mingw-w64-x86_64-python2
package instead of python2:

I never tried this actually, I'll check if this works. This is probably the best thing to put into README.md if it does, much simpler than installing base-devel tools one by one and using Windows Python.

[alexcrichton]

@petrochenkov oh duh, right! If you have that check for both GNU and MSVC builds, does it prevent the build altogether?

If that works then I agree with your 1/2/3!

[petrochenkov]

If you have that check for both GNU and MSVC builds, does it prevent the build altogether?

Yes.

[alexcrichton]

Excellent! Do you want to send a PR to the configure script or shall I?

[petrochenkov]

Could you do this?
I have only couple of evenings left (then I'm leaving on vacation, etc) and I'd like to finish my other PRs.

[alexcrichton]

Sure, thanks for the investigation and the report!

[petrochenkov]

If you are building under msys2 try installing the mingw-w64-x86_64-python2
package instead of python2:
I never tried this actually, I'll check if this works. This is probably the best thing to put into README.md if it does, much simpler than installing base-devel tools one by one and using Windows Python.

mingw64/mingw-w64-x86_64-python2 2.7.11-1 is confirmed to work.