Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cp38-macosx universal2 and arm64 Python pkg URL #1169

Closed
wants to merge 2 commits into from

Conversation

hombit
Copy link

@hombit hombit commented Jul 7, 2022

Previous URLs pointed to x86_64 version

@hombit hombit changed the title Fix cp38-macosx_universal2 Python pkg URL Fix cp38-macosx universal2 and arm64 Python pkg URL Jul 7, 2022
@joerick
Copy link
Contributor

joerick commented Jul 7, 2022

Unfortunately we don't want to upgrade the bundled Python because it would increase the minimum MacOS version for the x86_64 arch from 10.9 to 11.0, which is a pretty massive jump.

@hombit
Copy link
Author

hombit commented Jul 7, 2022

Sorry, could you please explain how arm64 Python URL affects x86_64 minimum macOS version?

{ identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg" },
{ identifier = "cp38-macosx_arm64", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg" },
{ identifier = "cp38-macosx_universal2", version = "3.8", url = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg" },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here you update also python to build universal2 wheels. universal2 wheels are wheels that contain code for both x86 and arm. This python version does not allow to build wheels against x86 for macos 10.9.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I update arm64 URL only?

Copy link
Author

@hombit hombit Jul 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't understand, sorry. For Python 3.9 and 3.10 both arm64 and universal2 URLs points to universal2 packages, while all python 3.8 URLs are for the x86_64 package. Why is it different for different Python versions?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not play too much with ARM on macOS so let's wait for what @joerick will tell.

@joerick
Copy link
Contributor

joerick commented Jul 8, 2022

Ah, I understand what's going on. We still have no arm64 CI, so we've never actually had a chance to test running all this on arm64. The version of 3.8 that we install only runs on x86_64. So when you run it on arm64, it's emulating x86_64. The wheel that you built is fine, but you're not able to test it.

This PR seems to only change the universal2/arm64 build identifiers. I don't think that will change anything, as on macOS you can only have one minor version of Python installed at once, and the first one that cibuildwheel finds, it will install. So if we were going to make this change we'd have to change all three 3.8 URLs.

The Python 3.8 arm64 support was added in #704. I don't have time right now but perhaps there are some clues in there why we're not using the macos11 versions.

@henryiii
Copy link
Contributor

henryiii commented Jul 8, 2022

We tried very hard to use this when it came out, but it doesn’t work. CPython only has partial support for 3.8 on ARM - it was squeezed in at the last minute in the last binary release. IIRC you can’t produce 10.9-10.15 compatible wheels for Intel from the ARM version since Python didn’t include a 10.9 compatible Intel version in the binary, like they normally do (and do in 3.9+). You can look at the PR where 3.8 was added, I think.

I think to make this “work” we’d need to special case when running on ARM and have the (big) caveat that you can’t build for macOS 10 from AS (but only for 3.8). Is 3.8 still the version Apple ships? I’m traveling and won’t be my AS machine till the 27th.

@henryiii
Copy link
Contributor

henryiii commented Jul 8, 2022

Read here:

I think we should see if we can build from AS too, but we haven’t had CI runners for it and it will take some changes and should error out of macOS version is set <11.

I’m pretty sure we use the AS version from this table when building pure AS wheels from Intel.

@mayeut
Copy link
Member

mayeut commented Jul 9, 2022

IIRC you can’t produce 10.9-10.15 compatible wheels for Intel from the ARM version since Python didn’t include a 10.9 compatible Intel version in the binary, like they normally do (and do in 3.9+).

Now that MACOSX_DEPLOYMENT_TARGET is always set, I don't think there's a big issue on what you can target (i.e. will still be 10.9 for x86_64) unless I'm missing something. However, using a macOS 10.x runner to do the build won't be possible anymore with this installer which requires macOS 11+.

@henryiii
Copy link
Contributor

henryiii commented Jul 9, 2022

From what I understand, you can't set MACOSX_DEPLOYMENT_TARGET lower than whatever CPython was prepared with, otherwise we'd not need to download the official binaries at all. I could be wrong, but I'm pretty sure MACOSX_DEPLOYMENT_TARGET can only be increased.

@Czaki
Copy link
Contributor

Czaki commented Jul 9, 2022

yes. here is a wheel code fragment with a warning about this:

https://github.com/pypa/wheel/blob/9d4264d2f23ea44dd85639a27fa6b284261921cd/src/wheel/macosx_libfile.py#L411-L419

@joerick
Copy link
Contributor

joerick commented Jul 10, 2022

Thanks all for the informative discussion. I guess the upshot is that we keep the macos10.9 installer, and we can't test 3.8 arm64 wheels, even when running on arm64.

However, we should change the behaviour when running on Apple Silicon, to not try to test the arm64 3.8 wheel, and include an informative message explaining why.

joerick added a commit that referenced this pull request Jul 11, 2022
I've added a warning for users, so they know that it doesn't get tested.
See discussion in #1169 for the reason why.

Fixes #1168, fixes #1169
@mayeut mayeut closed this in #1171 Jul 14, 2022
mayeut pushed a commit that referenced this pull request Jul 14, 2022
I've added a warning for users, so they know that it doesn't get tested.
See discussion in #1169 for the reason why.

Fixes #1168, fixes #1169
mayeut added a commit to mayeut/cibuildwheel that referenced this pull request Sep 25, 2022
Per discussion in pypa#1169, the default installer used for cp38 is an Intel installer.
It makes sense to skip testing arm64 wheels in this case.

However, if the user choose to manually install the universal2 CPython version, then, tests shall be run on arm64.

This allows users that either target 11.0+ on intel, universal2 or only build for arm64 to get the arm64 wheel tested on AppleSilicon.
mayeut added a commit to mayeut/cibuildwheel that referenced this pull request Sep 25, 2022
Per discussion in pypa#1169, the default installer used for cp38 is an Intel installer.
It makes sense to skip testing arm64 wheels in this case.

However, if the user choose to manually install the universal2 CPython version, then, tests shall be run on arm64.

This allows users that either target 11.0+ on intel, universal2 or only build for arm64 to get the arm64 wheel tested on AppleSilicon.
mayeut added a commit to mayeut/cibuildwheel that referenced this pull request Sep 25, 2022
Per discussion in pypa#1169, the default installer used for cp38 is an Intel installer.
It makes sense to skip testing arm64 wheels in this case.

However, if the user choose to manually install the universal2 CPython version, then, tests shall be run on arm64.

This allows users that either target 11.0+ on intel, universal2 or only build for arm64 to get the arm64 wheel tested on AppleSilicon.
ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Jan 16, 2023
ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Jan 22, 2023
ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Dec 22, 2023
- not testable due to cpython38 constraints (pypa/cibuildwheel#1169)
ravnoor added a commit to ravnoor/ANTsPy that referenced this pull request Dec 24, 2023
- not testable due to cpython38 constraints (pypa/cibuildwheel#1169)
@joerick
Copy link
Contributor

joerick commented Feb 2, 2024

Just a note for anyone finding this now that arm64 runners are more widely available. If it's important to you to test the cp38 arm64 wheels, there's a workaround.

You can choose to install a different version of Python 3.8 on your runner before invoking cibuildwheel:

curl -fsSLO https://www.python.org/ftp/python/3.8.10/python-3.8.10-macos11.pkg
sudo installer -pkg python-3.8.10-macos11.pkg -target /
# ... invoke cibuildwheel ...

Just be aware that if you use this Python to build x86_64 wheels too, this increases the minimum supported macOS version on x86_64 to 11.0.

@joerick
Copy link
Contributor

joerick commented Feb 2, 2024

For reference, this gives an idea of which Python 3.8 users are using which OS.

image

That's the last 30 days, from our stats spreadsheet

@henryiii
Copy link
Contributor

henryiii commented Feb 2, 2024

Do you have your old python versions the last month plot? That was interesting to see the fraction of 3.8 ARM users. I also don't remember what the default Python version was for those versions. I'm on the latest macOS and I see 3.9 in my /usr/bin, so the've finally updated (a little).

@joerick
Copy link
Contributor

joerick commented Feb 2, 2024

image

Though I have no idea what's going on with the Python 2.7 arm64 users!? I guess that Homebrew offers a build of that?

@henryiii
Copy link
Contributor

henryiii commented Feb 2, 2024

I don't think they do, I don't think there's an open source 2.7 for AS, and homebrew doesn't offer 2.7 anymore. But Apple provided their own patched version in macOS 11 - 12.3.

skirpichev added a commit to skirpichev/gmpy that referenced this pull request Mar 9, 2024
marcelm added a commit to marcelm/dnaio that referenced this pull request Apr 23, 2024
jeanchristopheruel added a commit to Innoptech/OpenSTL that referenced this pull request May 28, 2024
jeanchristopheruel added a commit to Innoptech/OpenSTL that referenced this pull request May 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants