[24.1] name== hack for listing package versions no longer works

[stretch4x4]

Description

Previously you could skip the version definition to get a list of version numbers that were compatible with your environment.
This was really helpful for debugging or checking if a mirror had picked up an upstream package yet.

Pip 24.0:

(pip_test) $ pip install pip==
Looking in indexes: https://token:****@dl.cloudsmith.io/basic/___/pre-release/python/simple/
ERROR: Could not find a version that satisfies the requirement pip== (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1,
 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8
.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 20.1, 20.1.1, 20.2b1, 20.
2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 20.3b1, 20.3, 20.3.1, 20.3.2, 20.3.3, 20.3.4, 21.0, 21.0.1, 21.1, 21.1.1, 21.1.2, 21.1.3, 21.2, 21.2.1, 21.2.2, 21.2.3, 21.2.4, 21.3, 21.3.1, 22.0, 22.0.1, 22.0.2, 22.0.3, 22.0.4, 22.1b1, 22.1, 22.1.1, 22.1.2, 22.2, 22.2.1, 22.2.2, 22.3, 22.3.1, 23.0, 23.0.1, 23.1, 23.1.1, 23.1.2, 23.2, 23.2.1, 23.3, 23.3.1, 23.3.2, 24.0, 24.1b1, 24.1b2, 24.1, 24.1.1, 24.1.2)
ERROR: No matching distribution found for pip==

Pip 24.1.2 (happens from 24.1b1 onwards)

(pip_test) $ pip install pip==
ERROR: Invalid requirement: 'pip==': Expected end or semicolon (after name and no valid version specifier)
    pip==
       ^

I assume this was a bit of an undocumented feature and is part of the Remove support for legacy versions and dependency specifiers. change but I can't find an alternative way to get this information.

This is quite similar to #12749 but focusing on the version numbers rather than the urls

Expected behavior

A list of potential package versions that are compatible with my system.

pip version

24.1.*

Python version

3.11.5

OS

Ubuntu 22.04

How to Reproduce

• Install pip 24.0 (or below)
• run pip install <package name>== eg pip install pip==
• Install pip 24.1.*
• run pip install <package name>== eg pip install pip==

Output

$ pip install pip==
Looking in indexes: https://token:****@dl.cloudsmith.io/basic/___/pre-release/python/simple/
ERROR: Could not find a version that satisfies the requirement pip== (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1,
1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8
.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 20.1, 20.1.1, 20.2b1, 20.
2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 20.3b1, 20.3, 20.3.1, 20.3.2, 20.3.3, 20.3.4, 21.0, 21.0.1, 21.1, 21.1.1, 21.1.2, 21.1.3, 21.2, 21.2.1, 21.2.2, 21.2.3, 21.2.4, 21.3, 21.3.1, 22.0, 22.0.1, 22.0.2, 22.0.3, 22.0.4, 22.1b1, 22.1, 22.1.1, 22.1.2, 22.2, 22.2.1, 22.2.2, 22.3, 22.3.1, 23.0, 23.0.1, 23.1, 23.1.1, 23.1.2, 23.2, 23.2.1, 23.3, 23.3.1, 23.3.2, 24.0, 24.1b1, 24.1b2, 24.1, 24.1.1, 24.1.2)
ERROR: No matching distribution found for pip==
$ pip install pip==
ERROR: Invalid requirement: 'pip==': Expected end or semicolon (after name and no valid version specifier)
pip==
^

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[ichard26]

pip index versions <project> should be a suitable replacement for this trick, although it was definitely unintentional to break this.

[stretch4x4]

Oh neat! That one isn't in the docs (I assume because it is experimental) that is exactly what I need :D

Is there any way to make it use my pip.conf by default so I don't need to specify my index and cert every time?

But yeah would also be good if we can get that original behaviour back, at least until this command becomes official.

Thanks!

[NoahGorny]

@uranusjr @pradyunsg @ichard26 although this breakage probably was not intentional- lets stabilize pip index! (#12342)
I think this command has enough milage in order to be considered "official". Let me know what you think 😄

[pradyunsg]

I've never really liked this hack and I strongly prefer stabilizing pip index versions <name> over re-enabling this hack again.

I'd pushed back on this when we added this back into the new resolver (which also "broke" this hack) and my position now is basically the same as it was then: The error we're presenting here is absolutely the right thing to be doing here, since it's specific and provides guidance on what went wrong.

We had no reasonable means to get this information earlier but we do now, and I'd really like to stop having people rely on a hack that is non-discoverable via any sensible means, that is exceedingly easy for us to break, that forces us to not be able to improve how the "distribution not found" error is presented and will require some sort of bodge throughout the various pieces to enable.

I think this command has enough milage in order to be considered "official". Let me know what you think 😄

I think the main thing there is whether the output format is correct and whether we should do something like provide JSON or something by default there.

[stretch4x4]

I think the main output looks good to me, however a parameter for some for of machine readable version like JSON seems like a good addition as suggested in these two tickets:
#10983
#12749
The other minor upgrade I would like to see, is using pip.conf by default to feed in some of the params. Without this I have to supply my pip index with token and ssh path every time.
Thanks

[NoahGorny]

I think the main thing there is whether the output format is correct and whether we should do something like provide JSON or something by default there.

I think that we should stabilize the current main output and then work on additional json api as discussed in #10983
If we won't publish the current output people are going to complain that the hack is broken and we will have to re-instate it.

The other minor upgrade I would like to see, is using pip.conf by default to feed in some of the params. Without this I have to supply my pip index with token and ssh path every time. Thanks

@stretch4x4 Can you provide reproducible example on what happens in that case? I think that can be easily fixed 😄

[stretch4x4]

@stretch4x4 Can you provide reproducible example on what happens in that case? I think that can be easily fixed 😄

@NoahGorny not a problem

First example is equivalent of how I would use the hack ie pip install lib_io== and it grabs the two configs from my pip.conf
If I manually add the flags it works ok.
The == behaviour also includes pre-releases by default which is useful for how I use it but there isa --pre flag so that is not a massive issue 👍

(pip_test) user@MACHINE:~/repos/pip_test$ pip index versions lib_io
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
ERROR: No matching distribution found for lib_io

(pip_test) user@MACHINE:~/repos/pip_test$ pip index versions lib_io --index-url=https://token:**********************@dl.cloudsmith.io/basic/company/pre-release/python/simple/
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /basic/company/pre-release/python/simple/lib-io/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /basic/company/pre-release/python/simple/lib-io/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /basic/company/pre-release/python/simple/lib-io/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /basic/company/pre-release/python/simple/lib-io/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))': /basic/company/pre-release/python/simple/lib-io/
Could not fetch URL https://token:****@dl.cloudsmith.io/basic/company/pre-release/python/simple/lib-io/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(
host='dl.cloudsmith.io', port=443): Max retries exceeded with url: /basic/company/pre-release/python/simple/lib-io/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))) - skipping
ERROR: No matching distribution found for lib_io

(pip_test) user@MACHINE:~/repos/pip_test$ pip index versions lib_io --index-url=https://token:**********************@dl.cloudsmith.io/basic/company/pre-release/python/simple/ --cert=~/.config/pip/cert_file.cer
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
lib_io (6.9.1)
Available versions: 6.9.1, 6.9.0, 6.8.1, 6.7.0, 6.6.2, 6.6.1, 6.6.0, 6.5.3, 6.5.2, 6.5.1, 6.5.0, 6.4.0, 6.3.2, 6.3.1, 6.3.0, 6.2.0, 6.1.0, 6.0.1, 6.0.0, 5.1.0, 5.0.1, 5.0.0, 4.1.1, 4.1
.0, 4.0.1, 4.0.0, 3.7.2, 3.7.1, 3.7.0, 3.6.5, 3.6.4, 3.6.3, 3.6.2, 3.6.0, 3.5.0, 3.4.0, 3.3.3, 3.3.1, 3.3.0, 3.2.1, 3.2.0, 3.1.0, 3.0.0, 2.7.3, 2.7.2, 2.7.1, 2.7.0, 2.6.0, 2.5.5, 2.5.4, 2.5.3, 2.5.2, 2.5.1, 2.5.0, 2.4.0, 2.3.1, 2.3.0, 2.2.0, 2.1.3, 2.1.2, 2.1.1, 2.1.0, 2.0.0, 1.3.0, 1.2.0, 1.1.1, 1.1.0, 1.0.1, 1.0.0

[lopusz]

Hi All,

I recently updated pip to 24.1. I also saw the removal of informative error message (with package versions) for pip install abc==.

Would it be possible to have it back?

In my opinion it is not a hack it is a very useful feature that people use. It has been in pip for years.

See also, some discussions when it was removed in the past: #9139

Thank you very much for looking into this.

[joshcooper]

If we won't publish the current output people are going to complain that the hack is broken and we will have to re-instate it.

Hi I'm the lead maintainer for puppet and we've started getting support requests because puppet can no longer retrieve the list of available packages using pip install abc==versionplease. Having a non-experimental command like pip index versions abc that output JSON would be ideal, but AFAIK the current command is experimental and doesn't output JSON.

We've been using this hack since 2016 (see puppetlabs/puppet@152299cc859fc74343c697841848086d4e41b6f8) so it would be great if this hack could be re-enabled, though I understand why you might not want to do that.

[notatallshaw]

Reading through this, this is my understanding of the issue, let me know if I’m wrong:

1. Since pip 24.1 the requirement is more strictly parsed
2. This parsing comes from packaging which more strictly conforms to requirement and version standards
3. pip presents this error directly rather than somehow trying to determine the requirement name and presenting the old error

If packaging doesn’t parse the requirement it’s not clear, to me, how pip would be able to extract the requirement name and check what versions are available.

However, there is a fairly simple workaround for users who want to minimally change their workflow (i.e. not change to pip index versions {package}), provide a version that is valid but will never be used, e.g. pip install {package}==1+1, or to triply ensure it won't exist on PyPi pip install '{package}==9!0dev0+x'

$ pip -V
pip 24.2 from /home/dshaw/.pyenv/versions/3.11.8/lib/python3.11/site-packages/pip (python 3.11)

$ pip install 'requests==9!0dev0+x'
ERROR: Ignored the following yanked versions: 2.32.0, 2.32.1
ERROR: Could not find a version that satisfies the requirement requests==9!0dev0+x (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.8.8, 0.8.9, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.10.8, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.14.0, 0.14.1, 0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.8.0, 2.8.1, 2.9.0, 2.9.1, 2.9.2, 2.10.0, 2.11.0, 2.11.1, 2.12.0, 2.12.1, 2.12.2, 2.12.3, 2.12.4, 2.12.5, 2.13.0, 2.14.0, 2.14.1, 2.14.2, 2.15.1, 2.16.0, 2.16.1, 2.16.2, 2.16.3, 2.16.4, 2.16.5, 2.17.0, 2.17.1, 2.17.2, 2.17.3, 2.18.0, 2.18.1, 2.18.2, 2.18.3, 2.18.4, 2.19.0, 2.19.1, 2.20.0, 2.20.1, 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.25.1, 2.26.0, 2.27.0, 2.27.1, 2.28.0, 2.28.1, 2.28.2, 2.29.0, 2.30.0, 2.31.0, 2.32.2, 2.32.3)
ERROR: No matching distribution found for requests==9!0dev0+x

This works because:

1. 9! is the epoch, a part of the version specification I've never seen anyone use (please let's keep it that way)
2. dev0 refers to being the first possible development version, my understanding is dev versions are not allowed to be uploaded to PyPI
3. +x is a local version identifier, my understanding is local identifiers are not allowed to be uploaded to PyPI

Hope that helps someone.

[juntaosun]

Name: pip
Version: 24.2

pip index versions pip 
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
pip (24.2)
Available versions: 24.2, 24.1.2, 24.1.1

I really don't know why you keep changing this most basic command. Now using index versions will prompt a warning.

WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.

pip install pip==      
ERROR: Invalid requirement: 'pip==': Expected end or semicolon (after name and no valid version specifier)
    pip==
       ^

This most basic command has been used for many years, why was it suddenly changed?

[juntaosun]

pip index versions <project>should be a suitable replacement for this trick, although it was definitely unintentional to break this.

WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.

This command now prompts that it will be deleted.

[notatallshaw]

This most basic command has been used for many years, why was it suddenly changed?

It was an error message not a command, relying on the specific output of user facing error messages between tool versions is a recipie for disaster: https://xkcd.com/1172/. The command still works and still gives you an error.

If you want the same error message workarounds have been given: #12852 (comment)

This command now prompts that it will be deleted.

No, it prompts that the command is experimental, I think a PR to mark it as stable would be accepted at this point.

[notatallshaw]

I don't think this issue serves any further constructive discussion.

Pip can no longer parse arbitrary strings, such as an empty string, like they are real package versions. This comes from packaging which no longer supports parsing versions that don't comply with the spec (PEP 440 and onwards).

The primary command to use this now is pip index versions <project>, if there are requests or bugs with that command they should be opened as a new issue and not lost in this issue.

As a workaround, you can specify a valid version, but highly unlikely to ever actual exist, e.g. pip install <package>==3+3 (version 3, local segment version string "3"). Again though, this is not guaranteed to remain that exact same error message for the rest of time.

[cederom]

• I just got this problem.
• There are TONS of scripts out there that used pip install package== syntax for years.
• You broke the backward compatibility in a MINOR version release with no prior warning or notification.
• You mark this "unintentional breakage" as "not planned" to fix.
• A solution that you propose is using "experimental command. It may be removed/changed in a future release without prior warning".
• I avoided JavaScript because of self-incompatibility, but Python went the same drain.
• I can't even say how NOT COOL that is.

[notatallshaw]

You broke the backward compatibility in a MINOR version release with no prior warning or notification.

Pip follows Calver, there are no backwards compatibility guarantees outside patch releases: https://pip.pypa.io/en/stable/development/release-process/. I recommend you pin your pip version if you prioritize stability over new features and improvements.

There was various announcements that this version of pip may include unexpected breaking changes and there were multiple beta releases:

• https://github.com/pypa/pip/blob/main/NEWS.rst#241b2-2024-06-12
• https://discuss.python.org/t/call-for-beta-testing-of-pip-24-1/53599
• https://pradyunsg.me/blog/2024/05/13/pip-24-1-betas/
• https://www.reddit.com/r/Python/comments/1clx454/pip_241_beta_released_and_its_a_big_one/

No one during this beta period raised an issue that the error message produced by "package==" was relied on.

A solution that you propose is using "experimental command. It may be removed/changed in a future release without prior warning".

As previously discussed, I'm sure a PR removing its experimental status would be accepted.

[cederom]

You "unintentionally" broke the command that was working fine for years. Why not just bring it back?

Win-Win: You can easily bring back pip install package== as an alias to pip index versions package.

[potiuk]

Commment from a side.

You "unintentionally" broke the command that was working fine for years. Why not just bring it back?

Your request @cederom is quite similar to this XKCD comic:

Another variant of talking about it is Hyrum's law https://www.hyrumslaw.com/ - which I very much agree with - which explains that with sufficient complexity "any change is breaking".

While pip does not use semver, even SemVer is very clear that backwards-compatibility promises are about intentional used and not unintentional side-effects. See https://semver.org/#why-use-semantic-versioning

So I'd say you (and maybe few others) "unintentionally" used something that was never documented.

There are TONS of scripts out there that used pip install package== syntax for years.

I would be very interested to see some stats, do you know how many people do it and why you think a "TON" of usages are there? (note that with pip user base - even reports from few 100s of users would not mean a TON. This feature seems like a hack (and even the title mentions it's a hack).

Also I think it's good if you exercise some empathy towards pip maintainers - they attempt to keep their codebase clean - because it makes it easier to develop future versions. Removing a "hack" like that rather than keeping it running will make the more productive and do "more with less" - and their work enables millions of people out there. So any "saving" they do and allow them release pip faster will be multiplied by that for the whole ecosystem.

Converseley - you and few other people changing your scripts to be "proper" - for example use simple API to Pypi in json to retrieve the versions you need (which is about one line of curl + jq or single-liner in Python) is likely something that will take eveyyone a few minutes (but it won't be multiplied by millions) - so this is a "win-win" situation - you get cleaner, future-proof code, and pipmaintainers are not slowed down.

I think you should try to see that from this perspective.

[notatallshaw]

You "unintentionally" broke the command that was working fine for years. Why not just bring it back?

As already discussed, because packaging no longer parses it, please read through the previous comments so you can have informed comments.

[potiuk]

As already discussed, because packaging no longer parses it, please read through the previous comments so you can have informed comments.

Just to add - I actually read those earlier comments @cederom - I proposed you to use simple repository API https://packaging.python.org/en/latest/specifications/simple-repository-api/ that does the "trick" in a non-tricky way - because it is actually standardized an it is an "API" in fact.

I guess you probably have not read what I wrote, and downvoted my and @notatallshaw comments without tryiing to understand what I wrote. I personally prefer to solve a problem rather than win an argument, and you got reasonable proposal on how to do it, you can follow it to solve the problem or try to win an argument - up to you.