Build with Microsoft's UCRT (Visual Studio 2015 and above)

[mingwandroid]

I apologize if discussion has already happened on this feature request.

If someone wants to use a C library that uses C99 or a C++ library that uses C++11 or later, that is not possible with Python 2.7 because it (and every C/C++ compiled module) are built with Visual Studio 2008. Building with the UCRT would offer a way forward for people who want Python 2.x and modern C and C++ libraries to work together on Windows.

[naftaliharris]

Hmm. I don't personally know much at all about Windows development. Does Python 3 do this?

[mingwandroid]

Yes. It would really help to modernize this 2.8 version for C and C++ modules. I believe @ukoethe has done this before?

[kovidgoyal]

See https://github.com/kovidgoyal/cpython

[mingwandroid]

Thanks. Any chance of submitting a PR?

[kovidgoyal]

Sure, but I'd need to hear from @naftaliharris if he is willing to accept such a PR before I spend the time.

[naftaliharris]

@kovidgoyal, I don't understand this issue. (I don't know much about Windows development, I'm afraid). Is there some place I can read a bit more about this? If this is something it makes sense to do, (which I don't know enough to decide), I would be very grateful for a PR and be very happy to review it.

[kovidgoyal]

See the original issue I opened on the python bug tracker for more details: http://bugs.python.org/issue25759 for details on the patches.

As for why it is needed a brief background: On windows every different compiler (and there are dozens) usually ships with its own incompatible C library (think gnu libc). This even includes different versions of the same compiler. All native code used in a single program should be compiled against the same C library, otherwise you can end up with crashes in code that uses C library functions, which is pretty much all code, if it uses datastructures from one C library passed to another C library.

As a result, python and all its extension modules have to be compiled with the same version of Visual Studio (the "official" compiler to be used on windows). If you change the version of visual studio everything has to be recompiled. python 2.7 is stuck on Visual Studio 2008 which is very old and has no support for things like C++11. So if you use any modern C++/C99 libraries, such as, for example, Qt, ten you cannot compile them with VS 2008 and so cannot use them with python 2.7.

Feel free to ask if you have more questions.

[kovidgoyal]

Also as of Visuual Studio 2015, microsoft has moved to a "universal" C library that is (almost) entirely decoupled from visual studio versions and is updated as part of the OS itself -- this functions more like gnu libc on linux, where programs compiled against older versions continue to work against newer versions.

[kovidgoyal]

So let me know if this is something you want to accept in tauthon. Also, it would be nice to setup appveyor CI integration to test tauthon on windows alongside linux and OS X. It's my TODO list to setup that up in my cpython fork in any case, so if you want this PR I'll probably create it only after I have that working.

[naftaliharris]

So if somebody has a C-extension that works with Python 2.7, and we start building with VS 2015, what would you need to change about the C-extension in order to use it with tauthon? Any code changes, or just recompiling with VS 2015?

Also, it would be nice to setup appveyor CI integration to test tauthon on windows alongside linux and OS X.

That would be excellent!

[naftaliharris]

Another windows-related issue I'd love to have help on is #16, which will help us backport asyncio for windows.

[kovidgoyal]

On Thu, Feb 23, 2017 at 09:53:32AM -0800, Naftali Harris wrote: So if somebody has a C-extension that works with Python 2.7, and we start building with VS 2015, what would you need to change about the C-extension in order to use it with tauthon? Any code changes, or just recompiling with VS 2015?

It depends. If the C-extension does not use anything that has changed between VS 2008 and 2015 (highly likely) then just a recompile. Otherwise, they will need to port it just as you would need to port any C code when your compiler/stdlib changes.

[kovidgoyal]

On Thu, Feb 23, 2017 at 09:55:38AM -0800, Naftali Harris wrote: Another windows-related issue I'd love to have help on is #16, which will help us backport asyncio for windows.

Let me first say that I am not a windows expert by any means. All my windows knowledge comes from having maintained a cross platform product written in python (https://github.com/kovidgoyal/calibre) for about a decade. That said, I would be happy to help as when time permits. I am committed to maintaining python 2 for the foreseeable future for calibre in any case. I have no problems with contributing my efforts to tauthon.

[naftaliharris]

So let me know if this is something you want to accept in tauthon. Also, it would be nice to setup appveyor CI integration to test tauthon on windows alongside linux and OS X.

Thanks for explaining the issue to me! Yes, I will accept a PR for this, (but lets get CI for windows working first).

[kovidgoyal]

OK -- probably sometime this weekend, although given that it is also my daughter's annual day on Saturday, maybe not :)

[kovidgoyal]

Well I have it building on AppVeyor, but one of the ctypes tests is causing a hard crash. I cannot reporduce on my windows machine and trying to debug a crash on appveyor is impossible, so I'm stuck. All the other tests pass on AppVeyor.

[kovidgoyal]

I tried to reproduce the ctypes crash on three different windows VMs, failed in all, so Ihave just skipepd the test on AppVeyor, now the build succeeds: https://ci.appveyor.com/project/kovidgoyal/cpython

Note that I have skipped the following tests:

1. ctypes, for reasons noted above
2. bsddb: not of interest to me
3. tcl/tk: not of interest to me

Other than that, everything is built and all tests are run (except the very slow/memory intensive ones)

[naftaliharris]

Ok, I'm trying this out on Tauthon: https://ci.appveyor.com/project/naftaliharris/tauthon

Since backwards compatibility is a critical feature of this project, I am going to need all of the tests to pass.

[naftaliharris]

Added a new issue for this the CI #69

[kovidgoyal]

All tests do pass, just not on appveyor. It's likely something about the appveyor environment, but its hard to figure out what.

[arizvisa]

Also with regards to this, it'll also screw up the building of modules as distutils and such will need to be updated. I submitted a patch to 2.7 but as expected it was ignored.

https://bugs.python.org/issue35172

I regularly build my CPython 2.7 with VS2012 and VS2015, if there's still interest in this issue I can take the time to submit PRs.

[stefantalpalaru]

Please submit that patch here. Any way to control the Visual Studio version used by "PCbuild/build.bat" and ".github/appveyor.yml"?

[arizvisa]

@stefantalpalaru, not directly as the environment variables that point to the SDK and Header files are assigned by vcvarsall.bat via a parameter for 32-bit/64-bit. So one would need to locate that file based on which version the user has installed (probably searchable via the registry). But there isn't a general way as I think the user is expected to explicitly choose which version of VS they want to build under.

I'm not too familiar with appveyor, but docs seem to claim that you can specify the VS image to use via the "Image:" parameter like:

# Build worker image (VM template)
image: Visual Studio 2015

[arizvisa]

Btw, PR #120 adds the support for other versions of VS to distutils.

[arizvisa]

I submitted PR #121 which explicitly adds a platform-toolset parameter to PCbuild/build.bat. This should make it easier to specify the toolset to link with.

Keep in mind though that although later versions of Visual Studio do support C++11 (and more), Visual Studio 2015 and below don't fully support Expression SFINAE and a handful of other things that some of the cooler C++ libraries use. (Ran into this problem as using some of the cooler C++ libraries w/ Python was the primary reason for building it w/ later versions of Visual Studio).

[cemeyer]

Keep in mind though that although later versions of Visual Studio do support C++11 (and more), Visual Studio 2015 and below don't fully support Expression SFINAE and a handful of other things that some of the cooler C++ libraries use. (Ran into this problem as using some of the cooler C++ libraries w/ Python was the primary reason for building it w/ later versions of Visual Studio).

@arizvisa , there is always mingw64-gcc :-).

[stefantalpalaru]

Implemented by #120