Is get_msvcr still relevant?

[GalaxySnail]

I'm running tests on Windows with this patch:

diff --git a/distutils/ccompiler.py b/distutils/ccompiler.py
index f4a8a897..6b0be67a 100644
--- a/distutils/ccompiler.py
+++ b/distutils/ccompiler.py
@@ -1058,7 +1058,7 @@ _default_compilers = (
     ('cygwin.*', 'unix'),
     # OS name mappings
     ('posix', 'unix'),
-    ('nt', 'msvc'),
+    ('nt', 'mingw32'),
 )

There are 2 failed tests. Except for a failed test in distutils/tests/test_msvccompiler.py because of DistutilsPlatformError: Unable to find vcvarsall.bat, the other one is in distutils/tests/test_ccompiler.py::test_has_function_prototype:

---------------------------- Captured stderr call -----------------------------
...
ld.exe: cannot find -lvcruntime140: No such file or directory
collect2.exe: error: ld returned 1 exit status
------------------------------ Captured log call ------------------------------
INFO     root:spawn.py:38 gcc -mdll -O -Wall -c AppData\Local\Temp\abortuj8w3z0y.c -o appdata\local\temp\abortuj8w3z0y.o
INFO     root:spawn.py:38 gcc -s appdata\local\temp\abortuj8w3z0y.o -lvcruntime140 -o a.out.exe

While building extensions, ld doesn't complain -lvcruntime140 because vcruntime140.dll can be found in -LC:/Program Files/PythonXY/. However, get_msvcr() seems to be considered as a system libaray, which is linked unconditionally without library_dirs.

AFAIK mingw-w64 never links to msvcrXY or vcruntime140, it only links to either msvcrt or ucrt (which is talked about in #196), and -lvcruntime140 also seems to be a no-op. I don't know why get_msvcr is added at that time, I guess it was for the old mingw and is no longer needed for mingw-w64 nowadays. The old mingw is dying, there's no reason to use it instead of mingw-w64 and it's OK to drop support for it.

The question is, should we simply remove get_msvcr and rename Mingw32CCompiler to Mingw64CCompiler? The compiler type mingw32 can be deprecated and kept for backwards compatibility. I'm not sure how will it affect cygwin, I haven't investigate it. After it getting done, we can add mingw-w64 compiler to CI tests (Clarification: It's different from #184, that PR is for mingw-w64 compiled CPython, while what suggest here is for the "official" MSVC compiled CPython.)

Any ideas?

[alvinhochun]

For Krita we compile sip and PyQt with mingw-w64 (mingw-builds in the past, now moved to llvm-mingw toolchain) so we have been monkey patching cygwinccompiler.get_msvcr to return an empty list to work around this issue (initially for -lucrt when we were still using mingw-w64 with msvcrt but I see that it just got removed in #202). For our use case (and in my point of view) the existing get_msvcr is simply wrong (or being used for the wrong purpose) so I would be happy to see it removed.

The question is, should we [...] rename Mingw32CCompiler to Mingw64CCompiler?

I personally think there is no need. Doing so would just breaks whatever code that references it (if there are any). If you do decide to do the rename, then it should probably be MingwW64CCompiler to stand for mingw-w64. Mingw-w64 do support 32-bit, so "Mingw64" may be confusing (e.g. MSYS2 uses MINGW64 to refer to the 64-bit mingw-w64-with-msvcrt environment). But to be fair the name MinGW-w64 itself is also confusing...

AFAIK mingw-w64 never links to msvcrXY or vcruntime140, it only links to either msvcrt or ucrt (which is talked about in #196),

To be pedantic, mingw-w64 is able to use newer versions of msvcrxx and also UCRT (with the --with-default-msvcrt configure option), and builds also ship the import libs for other CRT versions (to link a non-default CRT properly, the code will need to be built with the proper __MSVCRT_VERSION__ define manually set). There is no libvcruntime140.a however, because mingw-w64 when using UCRT does not (and cannot) depend on vcruntime140. Also note that in mingw-w64, libmsvcrt.a is a copy of the default CRT import lib; to explicitly use the system CRT (msvcrt.dll) one needs to link to libmsvcrt-os.a instead.

and -lvcruntime140 also seems to be a no-op.

I am not so sure. Looks like vcruntime140.dll exports some standard C library functions (e.g. memcpy), so -lvcruntime140 probably causes these functions to be imported from vcruntime140.dll instead of what the current mingw-w64 toolchain uses.

I don't know why get_msvcr is added at that time, I guess it was for the old mingw and is no longer needed for mingw-w64 nowadays. The old mingw is dying, there's no reason to use it instead of mingw-w64 and it's OK to drop support for it.

(My note back when our monkey patch was added was "the original get_msvcr function is supposed to return the name of the specific version of MS C runtime so that it can be (?) copied over" but I don't remember how I got that.)

[GalaxySnail]

I personally think there is no need. Doing so would just breaks whatever code that references it (if there are any). If you do decide to do the rename, then it should probably be MingwW64CCompiler to stand for mingw-w64. Mingw-w64 do support 32-bit, so "Mingw64" may be confusing (e.g. MSYS2 uses MINGW64 to refer to the 64-bit mingw-w64-with-msvcrt environment). But to be fair the name MinGW-w64 itself is also confusing...

OK, I agree. Making the docstring more clear is good enough.

To be pedantic, mingw-w64 is able to use newer versions of msvcrxx and also UCRT (with the --with-default-msvcrt configure option), and builds also ship the import libs for other CRT versions (to link a non-default CRT properly, the code will need to be built with the proper __MSVCRT_VERSION__ define manually set).

Thanks, I didn't know that. It looks like that -lmsvcrXX is as bad as -lucrt, and those who want to link them should rebuild mingw-w64 with --with-default-msvcrt or use gcc -dumpspecs and modify it.

I am not so sure. Looks like vcruntime140.dll exports some standard C library functions (e.g. memcpy), so -lvcruntime140 probably causes these functions to be imported from vcruntime140.dll instead of what the current mingw-w64 toolchain uses.

If libvcruntime140.a doesn't exist, -lvcruntime140 looks like a no-op. It seems that the linker is linking vcruntime140.dll, I don't know what does it mean to link a dll (instead of a .a lib).

// test.c
#include <stdio.h>
#include <string.h>

int main(void)
{
    const char hello[] = "Hello world!";
    char str[16];
    memcpy(str, hello, sizeof(hello));
    puts(str);
}

$ export SOURCE_DATE_EPOCH=$(date +%s)  # See https://reproducible-builds.org/docs/source-date-epoch/
$ gcc test.c -o a.exe
$ objdump -p a.exe | grep 'DLL Name'
        DLL Name: KERNEL32.dll
        DLL Name: msvcrt.dll
$ ./a
Hello world!
$ gcc test.c -L. -lvcruntime140 -o b.exe
C:/msys2/mingw64/x86_64-w64-mingw32/bin/ld.exe: cannot find -lvcruntime140: No such file or directory
collect2.exe: error: ld returned 1 exit status

$ cp /c/Windows/System32/vcruntime140.dll .
$ gcc test.c -L. -lvcruntime140 -o b.exe
$ sha256sum a.exe b.exe
6825bff922c3e393380409f4e0fdc767413f7635219d0798510327def9b45800 *a.exe
6825bff922c3e393380409f4e0fdc767413f7635219d0798510327def9b45800 *b.exe

If libvcruntime140.a exists, it will really link to vcruntime140.dll, but I think there may be some incompatibility (just like -lucrt).

$ gendef vcruntime140.dll
 * [vcruntime140.dll] Found PE+ image
$ dlltool -D vcruntime140.dll -d vcruntime140.def -l libvcruntime140.a
$ gcc test.c -L. -lvcruntime140 -o c.exe
$ ./c
Hello world!
$ sha256sum a.exe b.exe c.exe
6825bff922c3e393380409f4e0fdc767413f7635219d0798510327def9b45800 *a.exe
6825bff922c3e393380409f4e0fdc767413f7635219d0798510327def9b45800 *b.exe
8b159a5b2fd0666cc503ee2a7be2a0460abe90b5debc853993bff8681d468005 *c.exe
$ objdump -p c.exe | grep 'DLL Name'
        DLL Name: vcruntime140.dll
        DLL Name: KERNEL32.dll
        DLL Name: msvcrt.dll
$ objdump -p c.exe | grep -A4 'vcruntime140.dll'
        DLL Name: vcruntime140.dll
        vma:  Hint/Ord Member-Name Bound-To
        82b0        9  __C_specific_handler
        82c8       61  memcpy

[mstorsjo]

Thanks, I didn't know that. It looks like that -lmsvcrXX is as bad as -lucrt, and those who want to link them should rebuild mingw-w64 with --with-default-msvcrt or use gcc -dumpspecs and modify it.

Rebuilding mingw-w64 is generally the only safe way to do it - switching defaults via GCC spec files doesn't help since you at the very least would want to rebuild your libgcc/libstdc++ with the new target. If libgcc/libstdc++ only are linked statically, and only switching between msvcrt.dll and msvcrXX.dll, it may be ok ABI wise to switch without recompiling everything, but at least with msvcrt.dll vs UCRT, I would recommend against switching without rebuilding the toolchain's runtimes.

I am not so sure. Looks like vcruntime140.dll exports some standard C library functions (e.g. memcpy), so -lvcruntime140 probably causes these functions to be imported from vcruntime140.dll instead of what the current mingw-w64 toolchain uses.

If libvcruntime140.a doesn't exist, -lvcruntime140 looks like a no-op. It seems that the linker is linking vcruntime140.dll, I don't know what does it mean to link a dll (instead of a .a lib).

In most cases, linking directly against a DLL is the same as linking against the import library for that library. (For i386, the linker needs to do more guesswork for e.g. stdcall function suffixes though - for those cases, there are details that only are visible in a proper import library but not in the DLL itself.) However your nicely reproducible example does indeed show that it differs, subtly.

I would have expected it to link symbols in the same order and with the same priority as the option that resolved to a DLL(i.e. if the full linker invocation is ... -lfoo -lbar, then a symbol is pulled in from foo if it exists there, only then taken from bar), but it seems like the binutils implementation works more as if -lfoo resolves to foo.dll, it links as if ... -lbar ... -limplicit_generated_foo. I don't think this distinction is intentional, but more of an implementation detail. When doing the same experiment with LLVM's lld instead of GNU binutils (where lld does support the same feature of linking directly against DLLs), it seems to use the same linking order/priority as the original command line order.

(With lld, it seems like one has to add -Wl,--no-insert-timestamp to get reproducible outputs.)

[GalaxySnail]

Thanks for the explanation!

Rebuilding mingw-w64 is generally the only safe way to do it - switching defaults via GCC spec files doesn't help since you at the very least would want to rebuild your libgcc/libstdc++ with the new target. If libgcc/libstdc++ only are linked statically, and only switching between msvcrt.dll and msvcrXX.dll, it may be ok ABI wise to switch without recompiling everything, but at least with msvcrt.dll vs UCRT, I would recommend against switching without rebuilding the toolchain's runtimes.

It seems that libgcc_eh.a is exactly the same on msys2 mingw64 prefix and ucrt64 prefix (so is libgcc.a), so I assume that it would be safe if you are in pure C and libgcc is linked statically. But it may be an implementation detail, rebuilding mingw-w64 is indeed the most recommanded.

In most cases, linking directly against a DLL is the same as linking against the import library for that library. (For i386, the linker needs to do more guesswork for e.g. stdcall function suffixes though - for those cases, there are details that only are visible in a proper import library but not in the DLL itself.) However your nicely reproducible example does indeed show that it differs, subtly.

I would have expected it to link symbols in the same order and with the same priority as the option that resolved to a DLL(i.e. if the full linker invocation is ... -lfoo -lbar, then a symbol is pulled in from foo if it exists there, only then taken from bar), but it seems like the binutils implementation works more as if -lfoo resolves to foo.dll, it links as if ... -lbar ... -limplicit_generated_foo. I don't think this distinction is intentional, but more of an implementation detail. When doing the same experiment with LLVM's lld instead of GNU binutils (where lld does support the same feature of linking directly against DLLs), it seems to use the same linking order/priority as the original command line order.

Thanks, I didn't know there is a difference. I agree that we shouldn't depend on this implementation detail. Anyway, it sounds like a good idea to remove -lvcruntime140.

(With lld, it seems like one has to add -Wl,--no-insert-timestamp to get reproducible outputs.)

Clang 16 will support SOURCE_DATE_EPOCH. if ldd could support that, it would be a nice enhancement for lld.

[mstorsjo]

It seems that libgcc_eh.a is exactly the same on msys2 mingw64 prefix and ucrt64 prefix (so is libgcc.a), so I assume that it would be safe if you are in pure C and libgcc is linked statically. But it may be an implementation detail, rebuilding mingw-w64 is indeed the most recommanded.

Yes, exactly. In practice, there's probably a bit of things that are safe to use for a different target (roughly speaking, statically linked libgcc for C executables, but nothing more than that), but that's a really fuzzy area, so nothing one really can advertise to users - since it's incredibly easy to overstep the boundaries into something which is known to have subtle bugs.

(With lld, it seems like one has to add -Wl,--no-insert-timestamp to get reproducible outputs.)

Clang 16 will support SOURCE_DATE_EPOCH. if ldd could support that, it would be a nice enhancement for lld.

Oh, thanks for the reference, I hadn't noticed that Clang had gotten this. I'll see if I could look into implementing that in lld at some point.

[qwenger]

Hi,

Adding my grain of salt, as a follow-up of pypa/setuptools#4101.

I maintain https://github.com/PetterS/quickjs. On Windows we build with MinGW, mainly because QuickJS is difficult to get to compile with MSVC (there are attempts, but they require lots of patches on upstream).

For years (https://stackoverflow.com/a/57109148) we have monkey-patched get_msvcr to return an empty list. Else we get ld.exe: cannot find -lvcruntime140. With that patching It Works™.

For reference here is how we set up MinGW.

I'm in favor of removing get_msvcr for MinGW compiling. I cannot judge whether it is useful for Cygwin.

[jaraco]

@DWesl or @lazka, do you have any opinion or insight into this issue and what should happen next?

[lazka]

I'm looking into adding some coverage for MSVC CPython + mingw, and then deprecating and making get_msvcr return an empty list.

I've looked at references to get_msvcr() and dll_libraries in the top 8000 pypi packages and the only users I've found monkey patch it to remove it:

_top/mpi4py-3.1.6.tar.gz: mpi4py-3.1.6/conf/mpidistutils.py: del compiler.dll_libraries[:]
_top/gssapi-1.8.3.tar.gz: gssapi-1.8.3/setup.py: cygwinccompiler.get_msvcr = lambda *a, **kw: []

[lazka]

I've created #274

[DWesl]

I think it's possible to use MSVCR directly on Cygwin, but I don't think it's recommended except in very specific circumstances (i.e. functions not currently available through the Cygwin C runtime).

Directly using the Windows GUI functions instead of the Cygwin X server, or getting process information not exposed by the Cygwin APIs are the big examples I can think of. I don't know of Python packages that do that and use the distutils function to do so.