Poetry does not install from wheels on macOS Big Sur

[williammck]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Big Sur 11.0.1 on Apple Silicon

• Poetry version: 1.1.4

• Link of a Gist with the contents of your pyproject.toml file: https://github.com/crimsobot/crimsoBOT/blob/c00966011f5e829c097f0f1ce77e0b785d0d23ab/pyproject.toml

Issue

When using Poetry on Big Sur, poetry.installation.chooser.Chooser is not recognizing available macOS wheels.
e.g. Poetry uses cryptography-3.0.tar.gz instead of cryptography-3.0-cp27-cp27m-macosx_10_10_x86_64.whl.

This has been fixed in pypa/packaging#319 (v20.5 and above), however it appears that Poetry is using packaging v20.4.
I've tracked it down to https://github.com/python-poetry/poetry-core vendoring in packaging v20.4, as poetry.utils.env.VirtualEnv.get_supported_tags is using ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/_vendor/packaging/tags.py as the script source on my machine.

I'm happy to put up a PR for this both here and in poetry-core, I just wasn't sure how to proceed with the vendoring there and how it ties back to here.

If there's any clarification I can provide, let me know :)

[donaldguy]

For a sketchy workarount in the meantime

pip=$(which pip)
pushd $PWD
cd "$(dirname $(which poetry))/../lib/poetry/_vendor/"
cd "py$( python -V | grep -o '[23]\.[0-9]\+')"
cd poetry/core/_vendor
sed -i '' 's/packaging==20.4/packaging==20.5/' vendor.txt
$pip install -t $PWD --upgrade -r vendor.txt
unset pip
popd

[williammck]

For a sketchy workarount in the meantime

Feels weird but works (changed mine to install packaging==20.7 since that's latest).
Also calling out that virtualenv still has wheels for pip 20.2.2 packaged with it, I needed to run poetry run pip install -U pip so that pip would install macOS wheels without erroring.

But with those two changes, seems like things are working... so one upstream dependency and then figuring out what all's in Poetry that needs changing.

[emanueljacob]

I can confirm that the workaround by @donaldguy works well on MacOS 11.0 . I created a PR for that (in fact for packaging 20.8 not just 20.5), the tests have run successfully locally. Seems to be a small PR for me and therefore would be nice to see this reviewed and merged soon :)

[henryiii]

The old version of packaging also means poetry can't install NumPy on Big Sur (Intel) with the last released version (the build fails since it's not getting a wheel, and there's some setup required to add a BLAS library since Apple Accelerate is buggy).

For Big Sur Apple Silicon, you'll want pypa/packaging#380 which has just landed and is not yet in a released version of packaging (or pip, obviously), as otherwise it still will not be able to pick up normally-named binary wheels.

[barentsen]

Using poetry 1.1.4 with Python 3.8.6 (intel i386) on MacOS 11.1 (Big Sur), poetry also fails to install numpy for me with the following error:

numpy.distutils.system_info.NotFoundError: No lapack/blas resources found. Note: Accelerate is no longer supported.

As pointed out above, the problem is that poetry attempts to install packages from source rather than using the appropriate wheel, and the solution is to upgrade the version of packaging that is vendored with poetry.

Solution

Inspired by #3458 (comment) above, the following steps resolve the issue for me by updating the version of packaging that is embedded within poetry-core. These steps assume you already have poetry installed in your python environment:

git clone https://github.com/python-poetry/poetry-core.git
cd poetry-core
sed -i '' 's/packaging = "^20.1"/packaging = "^20.8"/g' vendors/pyproject.toml
python -m pip install vendoring  # required for the next step
make vendor/sync                 # updates the dependencies listed in vendors/pyproject.toml
python -m pip install .          # move the updated dependencies to site-packages

I'd open a PR to update the version of packaging that is included within poetry-core, but I think I better leave it to a maintainer with a good understanding of these vendored dependencies? I say this because running make vendor/sync also updated/modified some of the other embedded dependencies, and it's unclear to me whether that is desirable?

[sinoroc]

@barentsen As far as I understood, there is already a PR: #3530. If you get the chance, give it a try, leave some feedback, review the code.

[barentsen]

@sinoroc Excellent point! I forgot to mention: I evaluated PR #3530 and it did not appear to resolve the issue for me. I posted details here: #3530 (comment).

[sinoroc]

@barentsen

Excellent point! I forgot to mention: I evaluated PR #3530 and it did not appear to resolve the issue for me. I posted details here: #3530 (comment).

I see. My bad.

[henryiii]

There should be a new release of packaging somewhat soon, and it will fix a bug that causes correctly named universal2 wheels to not be loaded on Apple Silicon, but it looks like it will also drop Python 2.7, 3.4, and 3.5 support, so it might be something to watch for, as I think Poetry partially supports 2.7 and fully supports 3.5, at least that was my impression. Not that I've tried that :). Going up to 20.8 would at least fix the issue for all macOS 11 users on Intel that can't get things like NumPy wheels with Poetry currently! See pypa/packaging#385 and pypa/packaging#376 .

[nilnullzip]

Same issue on Intel silicon (16" 2019 MBP). Numpy and others don't download MacOS wheels and the attempted compile fails. My workaround is installing the failed packages at the specified revision with pip, after which running Poetry install/update succeeds. Make sure you also update pip if you see a message to that effect. I've had problems if I don't, although not sure if related.

[henryiii]

It's the same issue if you use pip 20.2 (which tends to be default some places, like the CLT venv), it's due to old versions of packaging. Pip updated the vendored packaging in 20.3, and poetry needs to, too.

[williammck]

I've fixed this locally for the time being by setting an alias for Poetry...
alias poetry='env SYSTEM_VERSION_COMPAT=1 poetry'

Setting SYSTEM_VERSION_COMPAT=1 instructs macOS to return 10.16 for any system version lookups, which fixes the issues with older versions of https://github.com/pypa/packaging.

[damienrj]

Nice, I will have to give this a try. I had been exporting to requirements.txt to get things to work for now.

[damienrj]

This solution worked better than most but fails with xgboost even with the new poetry core. I am sticking with converting to requirements.txt for now it seems.

[maxmorlocke]

For those still encountering this and finding this lovely thread, I had success getting things installed with a M2 running Monterrey using:

brew install mupdf swig freetype
poetry add PyMuPDF

Installing mupdf alone didn't cover it all.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.