Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Universal2 wheels on macOS #484

Merged
merged 66 commits into from
Jan 31, 2021
Merged

Universal2 wheels on macOS #484

merged 66 commits into from
Jan 31, 2021

Conversation

joerick
Copy link
Contributor

@joerick joerick commented Dec 16, 2020

Add support for building arm64 and universal2 wheels on macOS 11+.

  • Building arm64 and universal2 on x86_64 runners using CIBW_ARCHS
  • Testing universal2 wheels
  • Docs update
    • CIBW_ARCHS option
    • 'How to' build for Apple Silicon, (probably including the nuances around testing)
  • Fix up CIBW_TEST_SKIP support for specifying parts of universal2 wheels
  • Examples update?

Closes #473.

@joerick
Copy link
Contributor Author

joerick commented Dec 16, 2020

Hmm, due to actions/runner-images#2247 this is gonna be slow going... it might be better to start work on this next week...

@henryiii
Copy link
Contributor

henryiii commented Dec 16, 2020

We will have to download the new images for Python 3.9 specifically https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg (which, regardless of the name, does support 10.9+ and Intel too). I think 3.8 may get a release as well.

I think the builds are a bit slower, but still faster than Travis!

@henryiii
Copy link
Contributor

henryiii commented Dec 16, 2020

Do the 10.6 32/64 fat binaries not work on macOS 11? Is that perhaps the issue with 3.5? 2.7 does seem to be working, but not 3.5, if I read the log correctly? Possibly it's the workaround we have in place for SSL?

Just a thought, how about we drop 3.5 on Windows and macOS, but keep it on Linux for a little while longer (April 2021, EOL for Ubuntu 16.04, or whenever manylinux wheels drop it - ideally, I think we should match exactly what they are doing on Linux)

@Czaki
Copy link
Contributor

Czaki commented Dec 16, 2020

Do the 10.6 32/64 fat binaries not work on macOS 11? Is that perhaps the issue with 3.5? 2.7 does seem to be working, but not 3.5, if I read the log correctly?

https://discourse.jupyter.org/t/jupyter-notebook-error-dyld-library-not-loaded-corefoundation-after-macos-big-sur-upgrade/6838
https://stackoverflow.com/questions/64916693/jupyter-notebook-error-dyld-library-not-loaded-corefoundation-after-macos-big

So it looks like the bigger problem with all not patched python versions..

@henryiii
Copy link
Contributor

I'm not sure. It's not affecting 2.7, and Big Sur comes with 3.8.2 (CLT), so I think it's worth checking to see if 3.6 builds, anyway - maybe just reverse the order and do 3.9 first, then 3.8, etc. :) It might be that the Universal2 3.9 wheel will need to be in a separate job for now, anyway - but that shouldn't stop us from working out the 11.0 issues. @joerick let me know if / when it's okay to push to this branch. It's better if you do it for now, as I have to prepare to record some CMake training videos, but I'd like to help this along when you aren't working on it.

@joerick
Copy link
Contributor Author

joerick commented Dec 16, 2020

We will have to download the new images for Python 3.9 specifically https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg (which, regardless of the name, does support 10.9+ and Intel too). I think 3.8 may get a release as well.

I think the builds are a bit slower, but still faster than Travis!

Hm, I wonder if you're talking about native ARM support, or the ability to build universal2 wheels? I was under the impression that wheel and the Xcode toolchain was cross-compiling both x86_64 and arm by default, regardless of the Python version?

I'll disable the 3.5 macOS build for now. Feel free to hack on this branch @henryiii, it just takes about half an hour for each macOS build to queue so it's not a lot of fun :)

@henryiii
Copy link
Contributor

PyPy is not macOS 11 ready either? At least all the normal wheels built. Though are they univesal2? Don't see a difference in naming.

@joerick
Copy link
Contributor Author

joerick commented Dec 16, 2020

Building wheel for spam (setup.py): started
316
    Building wheel for spam (setup.py): finished with status 'done'
317
    Created wheel for spam: filename=spam-0.1.0-cp37-cp37m-macosx_10_9_x86_64.whl size=2382 sha256=efb4f386693223f8a943c5b60d7571318f0258eda5a7339992c62e957e40d493
318
    Stored in directory: 

Yeah, maybe you're right that we need the new Python macos11 builds

@Czaki
Copy link
Contributor

Czaki commented Dec 16, 2020

PyPy is not macOS 11 ready either? At least all the normal wheels built. Though are they univesal2? Don't see a difference in naming.

In the current release, of the wheel depends on get_platform from distutils.util.
Its output depends on the installer (it is hardcoded, not calculated).

So without changing the installer platform tag will be x86.

As I do not understand, there is no plan for changing this behavior.

@henryiii
Copy link
Contributor

henryiii commented Dec 17, 2020

...built_wheel/spam-0.1.0-cp39-cp39-macosx_10_9_universal2.whl ...

Looks like that did the trick. :) How do we want to switch to using the new download? Always, with a flag, etc? Can you build an Intel (or Apple Silicon) wheel from the universal Python binary? If so, I don't see a reason to not always download it?

Also, PyPy and Python 3.5 don't support macOS 11, but I think that's okay - we could either recommend they be skipped for now until 11.0 becomes the default for macOS-latest on one of the CI systems, and by then, hopefully PyPy will have fixed their issues. @mattip is this a known issue with PyPy? Also will it be able to make Universal2 wheels too eventually? I feel like I have seen this in homebrew too for the 11.0 (but could be mixing it with Apple Silicon support - that I know is missing currently for PyPy). GitHub is pretty fast, so maybe macOS-latest will change sooner than I think, but I bet we have some time.

@henryiii
Copy link
Contributor

More wheel related, if you need a really new pip to use a universal2 wheel, I expect most projects will provide both for the near future - in which case, it seems much more sensible to provide a Intel and an Apple Silicon wheel instead of an Intel and a Universal2 wheel. So the question remains, can you set a target?

@henryiii
Copy link
Contributor

Also, can we silence all the silly warnings from pip? The Python 2 warning, the newer version available warning? We pin pip, so these warnings just get in the way when trying to find the real error.

@henryiii
Copy link
Contributor

And, final message tonight - do we have any 10.x builds left? Azure and GHA ran on 11, and I can't see CircleCI, so can't tell if this still works on 10.15 or if using the new Python broke it.

@mattip
Copy link
Contributor

mattip commented Dec 17, 2020

Also, PyPy and Python 3.5 don't support macOS 11

Is that this issue with _ctypes and SCDynamicStoreCopyProxies or are there more problems?

Edit: this issue

@Czaki
Copy link
Contributor

Czaki commented Dec 17, 2020

Looks like that did the trick. :) How do we want to switch to using the new download? Always, with a flag, etc? Can you build an Intel (or Apple Silicon) wheel from the universal Python binary? If so, I don't see a reason to not always download it?

I do not know the method to do this. As far as I know, there is a need to check the Xcode version. Maybe @ronaldoussoren will know more.

Maybe the best workaround will be updated delocate.

@joerick
Copy link
Contributor Author

joerick commented Dec 17, 2020

perhaps you quoted the wrong text there, @Czaki ?

Also, can we silence all the silly warnings from pip? The Python 2 warning, the newer version available warning? We pin pip, so these warnings just get in the way when trying to find the real error.

If we can be selective enough about the warnings, yes please :)

I can't see CircleCI, so can't tell if this still works on 10.15 or if using the new Python broke it.

Wait, you can't see the CircleCI logs? I thought it was open-source. I do have to login with Github, maybe you need to do that?

To answer your question, Circle's mac builds are still on xcode:9.4.1, macOS 10.13. Which makes this test failure all the more interesting:

        # check that the expected wheels are produced
        expected_wheels = utils.expected_wheels('spam', '0.1.0')
>       assert set(actual_wheels) == set(expected_wheels)
E       AssertionError: assert {'spam-0.1.0-..._64.whl', ...} == {'spam-0.1.0-..._64.whl', ...}
E         Extra items in the left set:
E         'spam-0.1.0-cp39-cp39-macosx_10_9_universal2.whl'
E         Extra items in the right set:
E         'spam-0.1.0-cp39-cp39-macosx_10_9_x86_64.whl'
E         'spam-0.1.0-cp35-cp35m-macosx_10_9_x86_64.whl'
E         'spam-0.1.0-pp27-pypy_73-macosx_10_9_x86_64.whl'
E         Use -v to get the full diff

To me, that looks like it produced a universal2 wheel, even though it was built on xcode9 which almost certainly doesn't support Apple Silicon! Wat.

More wheel related, if you need a really new pip to use a universal2 wheel, I expect most projects will provide both for the near future - in which case, it seems much more sensible to provide a Intel and an Apple Silicon wheel instead of an Intel and a Universal2 wheel. So the question remains, can you set a target?

Did a little searching though the CPython source and found _PYTHON_HOST_PLATFORM: https://github.com/python/cpython/blob/master/Lib/distutils/util.py#L47 (thanks @Czaki for the pointer). And I remembered that we already do this on CPython 3.5, I think @mayeut figured it out. So I guess the same approach would work here.

If that works, we might as well select the universal2 Python distribution where it's available.

@Czaki
Copy link
Contributor

Czaki commented Dec 17, 2020

@joerick I think I quote too long text. I comment mainly:

Can you build an Intel (or Apple Silicon) wheel from the universal Python binary?

cibuildwheel/macos.py Outdated Show resolved Hide resolved
cibuildwheel/macos.py Outdated Show resolved Hide resolved
@joerick
Copy link
Contributor Author

joerick commented Jan 30, 2021

Okay, this is ready for review!

The only think that will probably change is the removal of the wheel rename hack - judging by pypa/pip#9533 it looks like pip will soon support universal2 on arm64, and then we can remove that too. Thank you @henryiii for pushing that forward!

@joerick joerick marked this pull request as ready for review January 30, 2021 11:03
cibuildwheel/macos.py Outdated Show resolved Hide resolved
cibuildwheel/macos.py Outdated Show resolved Hide resolved
@henryiii
Copy link
Contributor

Let's bump our pins and dump the hack!!!

pypa/pip#9282 (comment)

@joerick
Copy link
Contributor Author

joerick commented Jan 30, 2021

I can't actually run bin/update_dependencies.py locally anymore - a combination of no Docker on M1 and no CPython 3.5 on macOS 11 - so I've done it in GHA instead. I wonder if this works...

@joerick
Copy link
Contributor Author

joerick commented Jan 30, 2021

Ah, it does not want to make a PR on a PR. Hey ho. I'll do it on master instead and merge that in.

@henryiii
Copy link
Contributor

You could let it make it on master then redirect it. But not really any different.

@henryiii
Copy link
Contributor

Actually, you can just cherry pick the change here and keep going. :)

cibuildwheel/macos.py Outdated Show resolved Hide resolved
@@ -180,6 +238,41 @@ def setup_python(python_configuration: PythonConfiguration,
# https://github.com/python/cpython/blob/a5ed2fe0eedefa1649aa93ee74a0bafc8e628a10/Lib/_osx_support.py#L260
env.setdefault('ARCHFLAGS', '-arch x86_64')

if python_configuration.version == '3.9':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we have access to packaging from #536 , then we should do a >= 3.9 here so we don't have to change it when we add 3.10. But this is fine until then.


# Different versions of Xcode contain different SDK versions...
# we're happy with anything newer than macOS 11.0
arm64_compatible_sdks = [s for s in sdks if not s.startswith('macosx10.')]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we sort and get the newest one? Not a change for this PR, but selecting the latest might be better than selecting randomly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can see on https://xcodereleases.com/ , each version of Xcode only ships with one macOS SDK version. I couldn't find a case where there are multiple. But I wasn't 100% sure, hence the use of lists here.

Also not sure what the selection criteria would be... would we want the newest , or the oldest? I tend to find that binaries are more often forward-compatible than backward-compatible. So then we'd want the oldest? ¯\_(ツ)_/¯

@YannickJadoul
Copy link
Member

Wow. I haven't really been following (partially because I didn't feel like I wanted more macOS issues in my life) this intensively, but great job, everyone! :-)

@joerick joerick merged commit b95dca5 into master Jan 31, 2021
@joerick
Copy link
Contributor Author

joerick commented Jan 31, 2021

Aaaand it's in! Thanks everyone for the input, especially @henryiii , @Czaki , @ronaldoussoren and @mayeut !

@joerick joerick deleted the macos-universal2 branch January 31, 2021 12:22
@ogrisel
Copy link

ogrisel commented Jan 31, 2021

Thanks very much to everybody involved in making this happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for macOS universal2 builds for ARM-based Macs
8 participants