Black install via pip fails on embedded Python on Windows using latest pip version (20.3)

[robintw]

Describe the bug
A new version of pip has recently been released, which is a lot stricter in many ways. When installing the latest version of Black on a Windows machine using the embedded Python distribution, I get an error:

ERROR: Requested black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has different version in metadata: '0.0.0'

The package it is trying to download is the latest source release (20.8b1), and from what I can gather, pip finds that the version in the filename (20.8b1) conflicts with the version in the metadata inside the package - which it says is 0.0.0.

To Reproduce

1. Install the embedded Python interpreter on Windows (https://docs.python.org/3/using/windows.html#the-embeddable-package)
2. Install pip in the embedded Python interpreter (download https://bootstrap.pypa.io/get-pip.py and run it using the embedded Python interpreter's python.exe)
3. Run python.exe -m pip install black --no-cache-dir
4. You will see the following output:

c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Requested black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has different version in metadata: '0.0.0'

If the latest version which has a wheel release published to PyPI is specified instead, then it works fine:

c:\pepys-deploy\pepys-import>python -m pip install black==19.10b0 --no-cache-dir
Collecting black==19.10b0
  Downloading black-19.10b0-py36-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.1 MB/s
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black==19.10b0) (1.4.1)
Requirement already satisfied: toml>=0.9.4 in .\python\lib\site-packages (from black==19.10b0) (0.10.2)
Requirement already satisfied: regex in .\python\lib\site-packages (from black==19.10b0) (2020.11.13)
Requirement already satisfied: attrs>=18.1.0 in .\python\lib\site-packages (from black==19.10b0) (20.3.0)
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black==19.10b0) (1.4.4)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black==19.10b0) (0.8.1)
Requirement already satisfied: click>=6.5 in .\python\lib\site-packages (from black==19.10b0) (7.1.2)
Installing collected packages: black
  WARNING: The scripts black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-19.10b0

So, it seems to be something to do with the metadata checking before building the wheel which is going wrong.

The latest version of Black worked fine with the previous version of pip which didn't use the new strict metadata checker.

I can't seem to reproduce this in non-embedded Python, but it may be an issue there too. I'm aware that pip isn't officially supported for embedded Python on Windows, but it is widely used and all other packages seem to install fine:

Expected behavior
Expected to install successfully.

Environment (please complete the following information):

• Version: 20.8b1
• OS and Python version: Windows embedded Python 3.7

Does this bug also happen on master?
N/A as related to PyPI releases

[cooperlees]

/me boots Windows VM to debug tonight or some time soon hopefully

[robintw]

Just as an additional piece of information: the install works when when you tell pip to use it's legacy resolver which isn't as strict about metadata (see below). However, you can see that it is definitely confused about versions - it states Successfully installed black-0.0.0 at the end.

c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir --use-deprecated=legacy-resolver
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in .\python\lib\site-packages (from black) (0.10.2)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black) (0.8.1)
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black) (1.4.1)
Requirement already satisfied: regex>=2020.1.8 in .\python\lib\site-packages (from black) (2020.11.13)
Requirement already satisfied: mypy-extensions>=0.4.3 in .\python\lib\site-packages (from black) (0.4.3)
Requirement already satisfied: click>=7.1.2 in .\python\lib\site-packages (from black) (7.1.2)
Requirement already satisfied: typing-extensions>=3.7.4 in .\python\lib\site-packages (from black) (3.7.4.3)
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-0.0.0-py3-none-any.whl size=124180 sha256=9d02c9cda7598e7caf1f73046db0bb8db2a1baf63bb54ac8c606c7ea8b143f82
  Stored in directory: C:\Users\ROBINW~1\AppData\Local\Temp\pip-ephem-wheel-cache-h9rra3jz\wheels\c5\85\79\f3af8daaf8037c0bf14beb3b7a1511a39b6e6902ca2aaf494e
Successfully built black
Installing collected packages: black
  WARNING: The scripts black-primer.exe, black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-0.0.0

Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

[cooperlees]

Just as an additional piece of information: the install works when when you tell pip to use it's legacy resolver which isn't as strict about metadata (see below). However, you can see that it is definitely confused about versions - it states Successfully installed black-0.0.0 at the end.

c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir --use-deprecated=legacy-resolver
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in .\python\lib\site-packages (from black) (0.10.2)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black) (0.8.1)
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black) (1.4.1)
Requirement already satisfied: regex>=2020.1.8 in .\python\lib\site-packages (from black) (2020.11.13)
Requirement already satisfied: mypy-extensions>=0.4.3 in .\python\lib\site-packages (from black) (0.4.3)
Requirement already satisfied: click>=7.1.2 in .\python\lib\site-packages (from black) (7.1.2)
Requirement already satisfied: typing-extensions>=3.7.4 in .\python\lib\site-packages (from black) (3.7.4.3)
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-0.0.0-py3-none-any.whl size=124180 sha256=9d02c9cda7598e7caf1f73046db0bb8db2a1baf63bb54ac8c606c7ea8b143f82
  Stored in directory: C:\Users\ROBINW~1\AppData\Local\Temp\pip-ephem-wheel-cache-h9rra3jz\wheels\c5\85\79\f3af8daaf8037c0bf14beb3b7a1511a39b6e6902ca2aaf494e
Successfully built black
Installing collected packages: black
  WARNING: The scripts black-primer.exe, black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-0.0.0

Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

Yup sure thing. I'll fix and push for a release before then. Thanks for raising.

[ichard26]

I'll also look into this soon. I think this has something to do with setuptools-scm breaking as that's how the version is resolved in our source distributions. Why is it not working as expected? I don't know any answers but I'll do some digging.

[robintw]

I imagine a partial solution would be to publish a wheel for the latest version - installing the latest version that had a wheel (19.x) worked fine for me. Obviously it'd be good to work out why it's actually happening too though!

[ichard26]

We want to do more releases more often so I'm totally +1 for a new release. Need to check that the CHANGELOG is ready though. One thing, I can't do a release. I clearly have some power to push for a release, but no power to create and push one. That power lies with the other collaborators :-)

@cooperlees, what do you think?

[pradyunsg]

Since @cooperlees asked nicely...

You seem to be hitting pypa/setuptools-scm#386. The main solution is listed in pypa/setuptools-scm#386 (comment). I'm not a 100% sure how you're generating release artifacts, (I don't see your release process documented -- it's usually a good idea to do so, to avoid mistakes!) but it does seem that the sdist doesn't have the metadata that setuptools_scm needs to properly version it, likely due to setuptools_scm not being installed/used in the build environment.

Also, even though this is a pure-Python project, I strongly recommend uploading wheels to PyPI. Most users who use pip won't really download/uses the sdist if there's a wheel, so you'd also buy some time to investigate the underlying issue here by uploading a well-formed wheel. I'll leave it up to the maintainers to decide whether they want the 20.8 beta wheel built (checkout the 20.8 tag and build the wheel with pip wheel .?) or if they wanna put out a completely new release. :)

[RonnyPfannschmidt]

at first glance i believe a part of the problem is that setup_requires is in setup.cfg but the recent enough setuptools version is not ensured
a second part is that its observable that pip nukes the ability of setuptools_scm to fetch version data when it manages copying things to tmpdirs (and pip does not provide a way to get access to that metadata)

[sz-cgt]

FWIW, I just ran into this issue on an AWS EMR cluster linux install as well. Pinning pip to 20.2.4 worked around the issue for me

[dciborow]

Was able to get around this with all of my other packages by switching from their src dist to their binary. This was the only package that had the issue, and did not have a binary.
I would prefer to install from source, so would still use that if this is fixed.
Using legacy-resolver for now.

[reata]

ran into the same issue here for my package building via GitHub Actions

UPDATE:
My situation may not be the same with this issue. See #1933 for more info. And for me it's back to normal now with new setuptools release 51.3.1.

[robintw]

@cooperlees We're getting fairly close to the time that the legacy resolver in pip will be going away (the last information I could find said Jan 2021). Has there been any progress on fixing this? From what I can see from the discussion above, uploading a wheel would probably fix it for the moment, giving more time for a more general fix involving setuptools-scm.

If there's anything I can do to help get this done soon then just let me know

[cooperlees]

Unfortunately only @ambv can release or I would have a long time ago. @ambv - What's the plan?

[pradyunsg]

FWIW, https://pypi.org/project/black/#data tells me that there's quite a few folks who can make the release. It might be worth reaching out to them as well.

[lucinvitae]

I'm seeing a similar issue when pip-installing black within the python:3.8-slim docker image:

Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--no-deps', 'black==20.8b1'] errored with the following return code 1, and output: 
Looking in indexes: https://pypi.org/simple/
Collecting black==20.8b1
  Downloading black-20.8b1.tar.gz (1.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  WARNING: Requested black==20.8b1 from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea, but installing version 0.0.0
WARNING: Discarding https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea (from https://pypi.org/simple/black/) (requires-python:>=3.6). Requested black==20.8b1 from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has inconsistent version: filename has '20.8b1', but metadata has '0.0.0'
ERROR: Could not find a version that satisfies the requirement black==20.8b1
ERROR: No matching distribution found for black==20.8b1

The issue has been occurring across our CI clusters for a month now, since mid-January.

I don't think this issue is limited to Windows.

[lucinvitae]

For folks looking to workaround this issue, downgrading has fixed the install issues in CI for our company:

$ pip list | grep black
black                    19.10b0

[dciborow]

Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

This is no longer a working solution. Our build server went belly up. Could we get a binary release of black? That is how I am resolving for all my other packages...

[robintw]

Pinging a few other people listed as maintainers on the PyPI page, in case they can help get a wheel release out, while we wait for a full fix. It seems that this is failing on multiple platforms, and causing issues for CI for a lot of people, and (I hope) should be a fairly simple fix to upload a binary wheel release.

Is there anything you can do to help get a new release out as soon as possible: @ambv, @autophagy @carljm @willingc ?

[cooperlees]

I'd love to. I just don't have access. I've been personally trying to nudge @ambv ...

Gonna be a rocky release tho. We have almost ~8 months of merges so people will need to be ready for that.

[JelleZijlstra]

@cooperlees I do have PyPI access, let's talk with the other maintainers to see if we can get a release out.

[ichard26]

De-assigning myself since I do not have enough time to debug (also debugging is annoying since I dual boot Ubuntu and Windows but I only daily drive Ubuntu). Hopefully GH-2125 happens soon so this is resolved (or I guess worked around?).

[pradyunsg]

FWIW, I just saw that this involves using an embedded interpreter. Using pip with an embedded Python interpreter is explicitly not supported, by Python or pip.

From the link in OP:

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates to users.

I'd suggest closing this issue, since the specific usecase being asked for is not supported by the tooling being used.

[cooperlees]

Agree. The pip king has spoken.