A bug database and a bug signalling method in pip or withdrawing buggy versions from repos

[mirekphd]

What's the problem this feature will solve?
Packages versions known to be buggy and having a bugfix will be installed by pip without any warnings.

What is lacking is some bug signalling method in pip, so that the installer would warn or error when a buggy version of a package was about to be installed (e.g. when pinned or inherited from some obsolete requirements of a dependent package).

More context
Currently the developers of downstream packages have to add an exclusion in their requirements files manually, but this requires that they moniotor their requirements for issues on Github. If they have many users, with automated builds run upon every change of the upstream package, then those users will come to them and report the bug. In a week or so the issue will be solved (see #3100 as a recent illustration). The problems start when the downstream package is not so popular and no-one notices that it relies on a buggy dependency or two (which is likely, since unpopular, niche packages rarely have automated builds that automatically upgrade after every bugfix). Also if a bug is in a downstream package (e.g. a wrapper with a lot of possible optional dependencies), but is only raised when calling some function from an upstream one, it may be reported and investigated by the incorrect developers team.

Describe the solution you'd like
I think that pip should have a bug database of known bad package versions (ideally matched with "solutions", i.e. either uprade to a newer bugfixed version or downgrade to the latest known working one). This should be available in a machine-readable format similar to those CVE (vulnerabilities) databases used by linux systems security scanners (tools such as clair - see its data sources list).

I think open-source software quality could be improved that way.

Practical considerations
Querying bug databases will likely have impact on installer's performance, so this should be exposed only as a non-default option, to be used primarly by CI/CD automated build systems. The manually built "niche" packages would probably still suffer from the same problem given that they are unlikely to ever adopt this option (just like images in Docker Hub repos rarely use any CVE scanners). So probably the only solution that would apply to all packages would be if the bug database were integrated with an automated bug scanner (with the ability to withdraw buggy versions) and used regularly to monitor packages by PyPI and ideally also Anaconda,

[pradyunsg]

An important thing to note here is the logistics of managing and maintaining such a database. It would likely need thousands of hours of skilled work to create and maintain such a database in a central location, which is not possible for volunteer driven projects like pip. It's not practical to have a central database of all the bugs in all of the packages in all of PyPI.

Instead, Python Packaging tooling distributes the responsibility and workload for this to the publishers of the packages and the end users, giving them the ability to tweak the behaviours.

Python Package indexes like PyPI support "yanking" broken releases, which publishers can use to prevent versions with security vulnerabilities from being installed by pip "by default" and printing a warning when they are used. It is also possible for users to use constraint files to "block-out" installation of package versions that they wish to avoid.

[mirekphd]

Python Package indexes like PyPI support "yanking" broken releases, which publishers can use to prevent versions with security vulnerabilities from being installed by pip "by default" and printing a warning when they are used.

It's probably a good practice therefore to refuse to close a bug reporting issue until the broken package version is withdrawn from PyPI by the package publishers.

[pradyunsg]

@mirekphd how would pip determine how to do that? An issue is filed on a big tracker does not mean that the package is "broken". And, how would this mechanism be resistant to bad actors?

[mirekphd]

how would this mechanism be resistant to bad actors?

To be certain that withdrawing the package is not due to a malicious intent the bug would probably have to be acknowledged in the release notes of a bugfix. See for example such bug confirmation by the scikit-learn package developers in their release notes. In case of CVEs the issue is for obvious reasons largely unknown until fixed system package versions become available, so here with non-security related bugs in python packages, all reported but unfixed issues would have to be also probably ignored (and they are the part which is difficult to monitor, not release notes, which are normally tagged with "bugfix/fix").

[pradyunsg]

If it's acknowledged after a bugfix, there exists a mechanism for package maintainers to indicate this,and provide the appropriate behaviour - yanking the broken release.

[ichard26]

This is not something we would have the bandwidth for, and it's unclear how much value this would provide (especially compared to the amount of effort it would need to set up and maintain on a on-going basis). With this, I'll close this issue.

In terms of warning on published security vulnerabilities, see #5777 (which is related, although focuses more on automatic remediation)