Warn users if there is a higher package version that cannot be installed

[JustAnotherArchivist]

What's the problem this feature will solve?

I'm the developer of a package whose current version requires at least Python 3.8. This is reflected in python_requires in setup.py. The last version to support 3.7 is rather old and broken in many ways. When people run into issues with it, the advice is naturally to upgrade to the latest version.

The problem here is that if you run pip install --upgrade $PACKAGE under Python 3.7 (tested with Python 3.7.9, pip 21.3.1, setuptools 60.5.0), it will simply report that everything's fine ('Requirement already satisfied'). Most importantly, it does not indicate at all that there is a newer version of the package which is incompatible with the Python version. pip index versions $PACKAGE, pip install ${PACKAGE}==, and friends also only list the compatible versions. There are very good reasons for this, of course. However, in particular for people unfamiliar with python_requires, this results in them thinking they already have the current version of a package when they don't.

Describe the solution you'd like

I would like to see a warning on at least pip install --upgrade and pip index versions if higher version numbers get ignored, e.g.

WARNING: Ignoring newer versions of $PACKAGE (1.2.3, 1.2.4, ...) which are not compatible with your version of Python.

I'm not all that familiar with the intricacies, so please excuse my ignorance about other reasons why package versions might get pruned from the installation candidates. A more generic warning than just the Python version might be needed instead. However, from a user perspective, ideally it would tell all the relevant details, e.g. also 'requires at least Python 3.8' in this particular case.

Alternative Solutions

Other than checking from within the package what the current version is and printing a warning at execution time (as PRAW is doing with update_checker, for example), which has always annoyed me as a user and is definitely not a route I want to take, I cannot think of other approaches since the package code is not involved at all in this step of the installation process to my knowledge.

Additional context

The specific package, if relevant, is snscrape. Version 0.3.4 is the last to support Python 3.7.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[mnot]

This appears to be similar to #6695.

I think it's also a documentation issue -- currently, python_requires documentation implies that a version mismatch

will prevent pip from installing the project on other Python versions.

What's happening is not prevention of installation; it's a silent downgrade, which is confusing for users and maintainers alike.

[pfmoore]

@mnot Agreed the guide isn't clear here. The field applies to the individual file it's specified in, not to the project as a whole. I'm sure the packaging user guide would welcome a PR improving the text.

[mnot]

Nod will see if I can get to that sometime soon.

I think the underlying issue still needs to be addressed -- as it is, packages that require later versions of python after their first release need to do awkward things like this.

[pfmoore]

Just to clarify, the behaviour of python_requires is well-defined, and the docs should explain what it does (it works per file, not per version or per project). Hopefully better-worded documentation will address the issue of users thinking it works differently than it does (which is the root problem here).

The problem that python_requires is awkward to use for certain situations is a different matter, and while I sympathise with the difficulty, it's not easy to fix, unfortunately.

From the original issue description:

I would like to see a warning on at least pip install --upgrade and pip index versions if higher version numbers get ignored,

The problem here is that there's a misunderstanding of what's going on. Pip isn't ignoring "higher version numbers", it's ignoring individual files that are marked as incompatible with the relevant Python version. Pip doesn't actually know (without doing extra work that it currently doesn't do, and which is tricky to get right) that it's discarded all the files for a later version, so a message along the lines of "I ignored version 2.0 because all of its files were marked as only for Python 3.10 or later" is a lot harder to implement than it seems, if you think of python_requires as metadata about the project version, rather than about the individual file.

[JustAnotherArchivist]

@mnot My bad, yes, this is the same as #6695. Not sure how I missed that when searching.

@pfmoore I'm not sure I understand what you mean by that. python_requires is part of the project metadata, right? In what situation would it only apply to some but not all files in a project (or a version of that project)?

[pfmoore]

The metadata is stored in the distribution file (the sdist file or the wheel). There is no guarantee that the metadata stored in the sdist is the same as the metadata in the win32 wheel, or the manylinux wheel, etc.

In practice, projects don't put different metadata in different files. But it's possible.

For example, suppose you build a wheel on your Windows PC, with python_requires >=3.10. Then you copy the sources to your Linux machine and edit them to say python_requires >= 3.9. Then you build a wheel from that. You now have 2 wheels, both for version 1.0 of your project, with different python_requires values. You can successfully upload these to PyPI, and nothing will complain. You now have a project that will install 1.0 on Python 3.9 on Linux, but not on Windows.

[JustAnotherArchivist]

Ah, I was thinking of source files within the project, not distribution files. Yeah, that makes sense. And I suppose there could even be cases where that might be a reasonable choice, e.g. if Unixoid-specific code in a project requires something in fcntl that can't be (easily) emulated in earlier versions of Python but the Windows equivalent works on lower versions.

And yeah, I can see how that makes things more complicated for a warning at the version level. How about warnings at the dist file level then?

WARNING: Ignoring $PACKAGE version $VERSION distribution file $FILENAME which is incompatible with your Python version.

This would still give the user a fairly obvious indication that there might be a newer version available, and I'd think it would be much easier to add.

[uranusjr]

Warning on the dist file level would be much too noisy because it’s not uncommon for projects to publish tens of files for one particular version (for example, Numpy 1.22.3 contains 20 files, if I counted correctly). Some aggregation will be needed for the message to make sense to the user.

[JustAnotherArchivist]

That's true, but shouldn't there generally only be one or two candidates for bdists after filtering for OS and machine architecture? For example, Numpy has separate wheels for three CPython versions (3.8 through 3.10), three operating systems, and x64/ARM as far as I can see (plus one for PyPy and the sdist). In practice, only one of the bdists and the sdist would be relevant for an actual installation.

[pfmoore]

We could definitely do better at summarising what pip considered, what it discarded, and why. Some of the criteria are:

• Not too verbose, if we generate a wall of text the user will simply ignore it.
• Understandable by the user, as this thread shows there are subtleties that are non-obvious.
• Actionable, the user needs to be able to work out what to do next, to fix the issue.

If someone were to tackle this, that would be great, but it's not going to be simple to get right. I, for one, don't have anything like the UX expertise to do a good job on this 🙁

[pradyunsg]

In practice, only one of the bdists and the sdist would be relevant for an actual installation.

No. It's totally possible to have multiple compatible wheels (eg: multiple manylinux versions), or because you have custom build numbers in wheels or local version specifiers for custom wheels in addition to upstream wheels (eg: that's what Compute Canada does).

[mnot]

Just thinking out loud --

It seems like what's required is one of the following:

1. The ability to emit a warning based upon metadata in a later distribution that pip has discarded (something like a warn_if_discarded flag, possibly with an argument that contains text for the warning)

2. The ability to modify previous revisions in pypi to get them to emit a warning when installed (or otherwise change their metadata to get the desired effect)

Does that make sense, did I miss any options, and which one is more realistic to do?

[uranusjr]

Not sure if I understand you options. What I’m thinking is pip should keep a list of discarded candidates, and when it selects a candidate, go through the list and emit a message about why candidates with higher versions in the list were discarded. This way we could even expand the functionality to cover other reasons than Requires-Python.

Also note that pip checks Requires-Python in two places; once when it fetches the list of files from the index (data-requires-python from PEP 503), and another when the file is downloaded and metadata is built (Requires-Python from Core Metadata).

[cj-darius-lapunas]

Additionally, warning users if none of the packages fit your requirements would be helpful. Recently spent quite a long time debugging why "Could not find a version that satisfies the requirement". Assumed I can't find the package entirely. Actually, it was discarding all of the versions, because my Python was too old. Pointing out that "9 packages discarded due to incompatibility" would have saved me hours of debugging.

[cyberw]

If you are a maintainer of said packages, there is a possible workaround: Publish a package that does support the lower python version, but fails on install if the version is too low. This will force your users to either explicitly select the old version or get a new Python.

I did something similar for when we renamed locustio to locust and wanted our uses to move. Try pip install locustio to see what I mean.

[ConnorStoneAstro]

Perhaps a simpler solution than working out why each dist was ignored/filtered, is to just look up what is the highest version number available after resolving which one to install. If pip didn't end up having the latest version out there, then a little warning saying: WARNING: installing version x.x.x because the higher version y.y.y is incompatible would already let people know to hunt for a reason why this might happen rather than silently giving unexpected behavior.

Obviously the error would not be thrown if the user requested a specific version.

A similar warning could be thrown if there is no compatible version like: WARNING: no compatible package found, the latest y.y.y is incompatible.

[RonnyPfannschmidt]

Pip is short for pip installs packages

It's not a lifecycle management system

[ConnorStoneAstro]

I'm not sure I understand. Pip already throws a [Notice] a new release of pip is available... when you aren't using the latest version of pip. I'm thinking of just doing something like that. Considering how often people just want to install the latest version of a package and they don't realize they are silently getting an outdated version, this seems to fall under the scope of pip installs packages. Also see PEP 20 In the face of ambiguity, refuse the temptation to guess..