macOS: wheels names disregarding deployment target for cp38+

[boschmitt]

I have a Github workflow that builds wheels for macOS. Since my library depends on C++17 for both core language features and library features, I set MACOSX_DEPLOYMENT_TARGET to be fairly high, i.e., 10.15.

When building the wheels, I get wheels with the correct name for CPython versions 3.6 and 3.7. However, the other versions are named *10.9*:

5 wheels produced in 30 minutes:
  tweedledum-1.1.1-cp310-cp310-macosx_10_9_x86_64.whl
  tweedledum-1.1.1-cp36-cp36m-macosx_10_15_x86_64.whl
  tweedledum-1.1.1-cp37-cp37m-macosx_10_15_x86_64.whl
  tweedledum-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl
  tweedledum-1.1.1-cp39-cp39-macosx_10_9_x86_64.whl

I believe the culprit code is:

cibuildwheel/cibuildwheel/macos.py

Lines 286 to 290 in 2709a49

	elif python_configuration.identifier.endswith("x86_64"):
	# even on the macos11.0 Python installer, on the x86_64 side it's
	# compatible back to 10.9.
	env.setdefault("_PYTHON_HOST_PLATFORM", "macosx-10.9-x86_64")
	env.setdefault("ARCHFLAGS", "-arch x86_64")

As a workaround, I have manually set _PYTHON_HOST_PLATFORM to be macosx-10.15-x86_64, but I'm not sure if there might be other implications.

[henryiii]

I feel like this was a bug that we were going to fix, but seems to have not been fixed. It should be fixed, or we should use pypa/wheel#407 which I haven't had a chance to implement yet.

[joerick]

Agreed, this looks like a bug to me. I think that we should be setting _PYTHON_HOST_PLATFORM to the maximum of our known-good values for the arch and the user supplied MACOSX_DEPLOYMENT_TARGET. In fact, we already have something like that implemented in the tests...

cibuildwheel/test/utils.py

Lines 89 to 99 in 02652e6

	def _get_arm64_macosx_deployment_target(macosx_deployment_target: str) -> str:
	"""
	The first version of macOS that supports arm is 11.0. So the wheel tag
	cannot contain an earlier deployment target, even if
	MACOSX_DEPLOYMENT_TARGET sets it.
	"""
	version_tuple = tuple(map(int, macosx_deployment_target.split(".")))
	if version_tuple <= (11, 0):
	return "11.0"
	else:
	return macosx_deployment_target

So I think we should bring the same logic into the code, too.

Do you have any opinion on this @mayeut? I seem to remember discussing _PYTHON_HOST_PLATFORM and MACOSX_DEPLOYMENT_TARGET, and I think you added the original _PYTHON_HOST_PLATFORM override a few years back. Was there a particular reason that it was static and not tied to MACOSX_DEPLOYMENT_TARGET?

Edit: whoops!

[henryiii]

Please see #819 :)

[joerick]

I'm working through my emails and I now see #819 with a @mayeut review! :)

[joerick]

@boschmitt would you mind taking a look at #820? I'm having trouble recreating this bug.

[boschmitt]

@joerick I had a look at #820, but I couldn't figure it out why the errors is not happening.

First, I thought that it was because I'm using [email protected] (sorry I failed to mention it before) and that branch might be using newer (unreleased) code. So I run my workflow again using cibuildwheel@main:

I tried to simplify the workflow as much as possible: Here is the result

The relevant parts:

Run pypa/cibuildwheel@main
  with:
    package-dir: .
    output-dir: wheelhouse
  env:
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.8.11/x64
    CIBW_BUILD: cp36-macosx_x86_64 cp39-macosx_x86_64
    CIBW_ARCHS: x86_64
    CIBW_TEST_REQUIRES: pytest
    CIBW_TEST_COMMAND: pytest --import-mode importlib -v {project}/python/test/
    CMAKE_BUILD_PARALLEL_LEVEL: 2
    MACOSX_DEPLOYMENT_TARGET: 10.15

Produced wheels:

2 wheels produced in 18 minutes:
  tweedledum-1.1.0-cp36-cp36m-macosx_10_15_x86_64.whl
  tweedledum-1.1.0-cp39-cp39-macosx_10_9_x86_64.whl

I'm doing some others tests on my side and I will try to look a bit further on #820.

EDIT: It might be a long shot, but: Is there a possibility that this only happens with 10.15? Unfortunately, I cannot try other deployment targets because my code won't build with anything lesser than that.

[henryiii]

CMAKE_BUILD_PARALLEL_LEVEL - Hmm, that looks like you are not doing a standard setuptools build. It might handle the naming correctly but other tools might not. What are you using?

[henryiii]

scikit-build >= 0.12.0 - Ouch, I could have sworn we were supposed to handle this correctly. I expect https://github.com/scikit-build/scikit-build/blob/c0741f9a1de9dd97e0a1e6f0b3b02fcd76221418/skbuild/command/bdist_wheel.py#L69 might be the culprit? Possibly setuptools is taking this exactly if forced in this way.

[henryiii]

Probably ideally should be fixed in both libraries, cibuildwheel and scikit-build.

[boschmitt]

CMAKE_BUILD_PARALLEL_LEVEL - Hmm, that looks like you are not doing a standard setuptools build. It might handle the naming correctly but other tools might not. What are you using?

I think its a standard setuptools build. I pretty much copied what is in the documentation. For these last tests, I removed anything that might be source of "weirdness", including CMAKE_BUILD_PARALLEL_LEVEL. Results still the same.

scikit-build >= 0.12.0 - Ouch, I could have sworn we were supposed to handle this correctly. I expect https://github.com/scikit-build/scikit-build/blob/c0741f9a1de9dd97e0a1e6f0b3b02fcd76221418/skbuild/command/bdist_wheel.py#L69 might be the culprit? Possibly setuptools is taking this exactly if forced in this way.

I'm don't have the required knowledge about the internals of cibuildwheel and skbuild to give a good answer here. However, since manually setting _PYTHON_HOST_PLATFORM to be macosx-10.15-x86_64 seems to do fix the problem, I would be "suspicious" of the parts of the code which deal if it.

[joerick]

Thanks for the recreation @boschmitt!

Hmm... personally, I'm not sure we should really 'fix' this issue unless we can pin the problem down to cibuildwheel. For me, the question is: is the 'correct' value of _PYTHON_HOST_PLATFORM is really related to MACOSX_DEPLOYMENT_TARGET?

The _PYTHON_HOST_PLATFORM variable is picked up by the function distutils.util.get_host_platform(). On my machine, which is running macOS 11.5, here's what that function returns without _PYTHON_HOST_PLATFORM set:

$ python3.9 -c 'import setuptools._distutils.util; print(setuptools._distutils.util.get_host_platform())'
macosx-10.9-universal2
$ MACOSX_DEPLOYMENT_TARGET=10.15 python3.9 -c 'import setuptools._distutils.util; print(setuptools._distutils.util.get_host_platform())'
macosx-10.9-universal2

These values come from sysconfig, they appear to be baked in when python was compiled. So I'm not sure that I'd consider setting _PYTHON_HOST_PLATFORM to MACOSX_DEPLOYMENT_TARGET to be any more 'correct'.

So my feeling is to stick with our current approach, which is as close to the python-default behaviour as possible, while doing the minimum to support cross-compiling.

[makslevental]

I'm bitten by this in the far flung future of 2024:

https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/macos.py#L245-L260

    env.setdefault("MACOSX_DEPLOYMENT_TARGET", "11.0" if config_is_arm64 else "10.9")

    if python_configuration.version not in {"3.6", "3.7"}:
        if config_is_arm64:
            # macOS 11 is the first OS with arm64 support, so the wheels
            # have that as a minimum.
            env.setdefault("_PYTHON_HOST_PLATFORM", "macosx-11.0-arm64")
            env.setdefault("ARCHFLAGS", "-arch arm64")
        elif config_is_universal2:
            env.setdefault("_PYTHON_HOST_PLATFORM", "macosx-10.9-universal2")
            env.setdefault("ARCHFLAGS", "-arch arm64 -arch x86_64")
        elif python_configuration.identifier.endswith("x86_64"):
            # even on the macos11.0 Python installer, on the x86_64 side it's
            # compatible back to 10.9.
            env.setdefault("_PYTHON_HOST_PLATFORM", "macosx-10.9-x86_64")
            env.setdefault("ARCHFLAGS", "-arch x86_64")

I don't understand why MACOSX_DEPLOYMENT_TARGET isn't injected into _PYTHON_HOST_PLATFORM here?

Maybe the solution is to move from cibuildwheel to scikit-build but right now I am procrastinating on that...

[henryiii]

Scikit-build is a build backend, cibuildwheel is a build front end that produces wheels. You can’t switch between them. What is your build backend? Sounds like it’s not respecting this, which means it’s probably one that was not really designed for binaries. I plan to fix this in hatchling in the future, and poetry-core probably doesn’t care.

[makslevental]

Scikit-build is a build backend, cibuildwheel is a build front end that produces wheels.

Thanks for clarifying that.

What is your build backend?

https://github.com/makslevental/mlir-wheels/blob/main/pyproject.toml#L13

build-backend = "setuptools.build_meta"

which either means setuptools or pip? I'm not sure how the responsibility is distributed amongst those two given the frontend/backend distinction you've clarified here...

[henryiii]

Setuptools is a build backend. I think your problem is you don't set MACOSX_DEPLOYMENT_TARGET. Set and use the standard variable for that (not OSX_VERSION), otherwise cibuildwheel sets it for you, which is what you are seeing. Most proper build backends (including setuptools) use MACOSX_DEPLOYMENT_TARGET instead of trusting what's in _PYTHON_HOST_PLATFORM.

Also, you should use macos-14 for the Apple Silicon jobs instead of cross-compiling, simpler and faster.

I expect you are not making usable SDists, since there's no MANIFEST.in. Scikit-build-core would be quite a bit simpler!

[makslevental]

I think your problem is you don't set MACOSX_DEPLOYMENT_TARGET. Set and use the standard variable for that (not OSX_VERSION), otherwise cibuildwheel sets it for you, which is what you are seeing.

Sorry I don't understand - I am setting CMAKE_OSX_DEPLOYMENT_TARGET but you're talking about the env MACOSX_DEPLOYMENT_TARGET that then sets the default for the cmake. But who's reading MACOSX_DEPLOYMENT_TARGET? I don't see that in cibuildwheel? Oh you're right it's setuptools. Sneaky sneaky...

Also, you should use macos-14 for the Apple Silicon jobs instead of cross-compiling, simpler and faster.
I expect you are not making usable SDists, since there's no MANIFEST.in. Scikit-build-core would be quite a bit simpler!

Yea it's on my todo but my todo gets longer every day and CI is not ever at the top...

Thanks for the pointers! I think I'll be able to find my way out now.