-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compile in universal2 mode for macOS Python >= 3.9.1. #114
Conversation
@miketimofeev Prod prod... |
@bwoodsend sorry, we didn't have a chance to take a look yet, but it's on our radar |
Just a +1 for this - it would be great to have! |
@bwoodsend we have announced macOS-10.14 image deprecation so in the scope of switching python packages generation to macOS-10.15 we will consider adding this change as well |
Hmm, not entirely sure what that implies but it sounds like good news ... 🙂 |
@@ -6,7 +6,7 @@ stages: | |||
- stage: Build_Python_MacOS | |||
dependsOn: [] | |||
variables: | |||
VmImage: 'macOS-10.14' | |||
VmImage: 'macOS-10.15' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One side effect of bumping the build image version is that, because the MACOSX_DEPLOYMENT_TARGET
isn't set anywhere, the Pythons and therefore any wheels produced will only be compatible with macOS >= 10.15. Is there a reason MACOSX_DEPLOYMENT_TARGET
isn't used? The python.org builds use minimum macOS version from 10.6 to 10.9 (which admittedly is overkill).
Due to `setup-python@v2` limitations, only x64 binary are created for macOS, universal binaries will be available once the following PR is merged: actions/python-versions#114
builders/macos-python-builder.psm1
Outdated
### For Python versions which support it, compile a universal2 (arm64 + x86_64 hybrid) build. The arm64 slice | ||
### will never be used itself by a Github Actions runner but using a universal2 Python is the only way to build | ||
### universal2 C extensions and wheels. | ||
if ($this.Version -ge "3.9.1") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May I ask why the Python version need to be greater than 3.9.1
? In official Python download page, you can have universal Python for version 3.8.10
. See below:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you're right. I hadn't realised that they backported to Python 3.8.x as well. Note though that Python 3.9.0
does not support universal2
so the conditional would have to be:
if ($this.Version -ge "3.8.10" && $this.Version -ne "3.9.0") {
(assuming that that is the correct powershell syntax).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think 3.8.10
is the only 3.8
version with universal2
. So a ==3.8.10
or >=3.9.1
will work I guess.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can compile universal2
for 3.8.11
- it's just by that point, python.org stops providing precompiled installers of any kind.
Using `universal2` Python environments is currently the only way to build Python wheels which support macOS M1.
That last change was just a rebase plus adding Python 3.8.10+ to the Python versions to use |
As a bonus point:
|
That is the best piece of news I've heard for ages. I've been facing using Jenkins indefinitely at the day job since it's the closest we can get to CI/CD on M1. |
I take it that there's still no VMware in there so no job isolation and therefore no chance of this being available without self-hosting? |
Thanks! I read the code node and that comment worries me a little "The arm64 slice will never be used itself by a Github Actions runner". I wonder that comment was before the news went out, like https://www.protocol.com/bulletins/microsoft-azure-arm-chips-ampere --- With Microsoft building their own arm based servers for azure, I believe that in 1-2 years we will likely see runners on arm, as being cheaper. Another case of never say never? ;) |
Despite it having the same name and instruction set, Apple M1 isn't really |
the fix in actions/python-versions#114 only applies to >= 3.9.1
Is it expected that this might break anything in users' builds? We (pyca/cryptography) are seeing builds failing with 3.10.5 (the first version to be built with this patch as far as I can tell). here's what the failures look like: https://github.com/pyca/cryptography/runs/6948897569?check_suite_focus=true And here's a build demonstrating that 3.10.4 is clean: pyca/cryptography#7344 The big difference one see is in the CLI args that are passed to the compiler when building C extensions. Here's what a diff of those looks like between passing and a failing build: --- old 2022-06-18 11:24:34.000000000 -0400
+++ new 2022-06-18 11:24:43.000000000 -0400
@@ -7,6 +7,12 @@
-fwrapv
-O3
-Wall
+-arch
+arm64
+-arch
+x86_64
+-isysroot
+/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
-I/usr/local/opt/sqlite/include
-I/usr/local/opt/sqlite/include
-I/Users/runner/openssl-macos-x86-64/include
@@ -19,12 +25,11 @@
-march=core2
-DUSE_OSRANDOM_RNG_FOR_TESTING
-I/Users/runner/work/cryptography/cryptography/.tox/py310/include
--I/Users/runner/hostedtoolcache/Python/3.10.4/x64/include/python3.10
+-I/Users/runner/hostedtoolcache/Python/3.10.5/x64/include/python3.10
-c
-build/temp.macosx-10.15-x86_64-cpython-310/_openssl.c
+build/temp.macosx-10.15-universal2-cpython-310/_openssl.c
-o
-build/temp.macosx-10.15-x86_64-cpython-310/build/temp.macosx-10.15-x86_64-cpython-310/_openssl.o
+build/temp.macosx-10.15-universal2-cpython-310/build/temp.macosx-10.15-universal2-cpython-310/_openssl.o
-Wconversion
-Wno-error=sign-conversion
|
I think that the only expected failures are if you're trying to link against single arch binaries (possibly installed via brew) or if you land with an XCode < 12.2. Neither of those look like the problem you're facing. |
We are actually trying to link against a single arch binary -- the OpenSSL we're linking there is x86_64 only. However, even if we switch to linking against a universal2 OpenSSL (see pyca/cryptography#7346) the issue with the sysroot exists. Is it possible these builds aren't compatible with the macos-12 images we're using? |
I think that it's more likely just to be a bug either in XCode or in one of the layers of build tools if it's pointing to non-existant XCode SDKs. I'll fire up my not so beloved macOS and see if I can reproduce locally. |
I've put together a minimal reproducer that demonstrates that universal2 + macos-12 images aren't working for some usecases: https://github.com/alex/temp-gha-macos-python/actions/runs/2523945774 |
I think 3.10.4 must have gotten rebuilt as universal2 in the last day or so, as it's now failing the same way 3.10.5 does. |
Not sure if I misunderstood, but won't this break building wheels for python projects that also compile and link to native code? To provide context, I'm a maintainer for feast-dev/feast. We have some latency sensitive code that's been ported to go and we call into it using gopy. Up until the last release we attempted, we'd correctly produce wheels with the go libraries embedded for x86_64, and other platforms needed to install feast from source. This change ended up breaking our wheel building process and required us to do some digging to figure out what had changed. I'd love to know if there's a better way for us to be building wheels or setting up our toolchain. |
Yeah, I'm having second thoughts about not making this optional. I believe that the status quo in PyPA land is very much You can opt-out of |
Explicitly providing |
Can you share a build URL for that? |
pyca/cryptography#7346 (it's going to take a minute for the macOS 3.10 job to fail, I previously reverted the commit so the CI needs to re-run) Edit: Specific job https://github.com/pyca/cryptography/runs/7046642554?check_suite_focus=true |
Hi, this change has also broken the build process of a C extension of mine, because now In my case setting setting |
@miketimofeev Would you be OK with this change becoming optional and opt-in. i.e. Anyone wanting - uses: setup-python@v3
with:
macos_arch: universal2 It feels like the safest option to me. |
I know that this might be inconvenient for some projects that failed to adopt the universal2, but I am against not making the default The opt-in should be about not avoiding |
If universal2 is going to be the default, the fact that it does not work on GHA's macOS 12 needs to be resolved. Until that happens, I encourage reverting the default. |
@bwoodsend looks like it will be better to disable |
oh reverting that change will break https://github.com/cielavenir/python-slz/actions/workflows/wheel.yaml |
Absolutely agree. PS: Having multiple builds ( |
I understand why you wanted to disable the default |
Fixes actions/runner-images#4133.
For Python versions that support it, build in universal2 mode so that package authors can build macOS M1 compatible wheels. As it is at the moment, I've made the switch mandatory because I can't think of any reason (minus the increase in size) that a universal2 Python would cause any harm.