Support including unsafe packages in exported requirements files

[shaib]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Hello, and thanks for Poetry.

In relatively recent versions of pip, there is a reqirement that In --require-hashes mode, all requirements must have their versions pinned (quoting the error message). That is not exactly accurate -- it is only dependencies that really need to be installed, which must be pinned; but when they do need to be installed, this applies to "unsafe" packages as well.

Poetry, at least as of 1.1.12, has no option to include unsafe packages in exported requirements files. This has recently led to breakage in a project I'm working on (which, unfortunately, I cannot share). The project is a web application which manages its dependencies through Poetry, and exports requirements for deployment on a PaaS.

The issue was triggered by an upgrade of Celery to version 5.2.3. In essence, we ran into celery/celery#7070 (the issue names pip-tools, but later users comment they experienced the same with Poetry):

• Celery introduced a dependency on specific, recent versions of setuptools, using a range of versions (setuptools>=59.1.1,<59.7.0);
• The pre-installed version of setuptools is typically not included in the range, so it must be upgraded;
• But setuptools is an unsafe package, so it is not included in the exported requirements,
• and so we get to the point where pip takes a requirements file where all entries have hashes -- implying --require-hashes -- and still has to install an unpinned dependency.

To reproduce:

$ poetry new fail && cd fail && poetry add celery==5.2.3 && poetry export -f requirements.txt -o requirements.txt && poetry run pip install -r requirements.txt

Of note, pip-compile (from pip-tools) has an --allow-unsafe option for this, and they plan to make that behavior the default -- see https://pypi.org/project/pip-tools/ (look for "Deprecations").

I suspect Poetry should do something similar.

Issues #3075 and #3472 seem to be related.

(FWIW, we considered two suboptimal-result workarounds -- Pinning Celery at 5.2.2, and doing away with hashes completely; the workaround chosen was a suboptimal-process one: We used pip-compile to generate a requirements-file with hashes for a suitable version of setuptools, and appended it to the requirements file exported by Poetry).

[dimbleby]

It seems that poetry currently considers that there are no unsafe packages:

poetry/src/poetry/puzzle/provider.py

Line 93 in ed26721

UNSAFE_PACKAGES: set[str] = set()

which is a slightly odd way to do it, but probably resolves this

[abn]

@dimbleby is correct. This was resolved in #2826.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.