Ship Poetry releases with pinned dependencies

[johnthagen]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

I propose that Poetry itself is distributed with locked/pinned dependencies for each release. This makes each Poetry release reproducible and consistent for all users. This treats Poetry more like an application than a library/package, which is how users interact with it.

Currently Poetry gets installed as a Python package with bounds on its dependencies rather than being pinned/locked. This means that if User A installs Poetry version X, and later User B installs version X, and User C upgrades to version X from version W, all three of these users can result in different Poetry installations with subtly different dependencies.

This can lead to some hard to undderstand/fix issues such as:

• poetry self update warning: The locked version 1.5.0 for shellingham is a yanked version. #7344

Within teams, it can also make it harder to debug why one developer's Poetry is acting in an odd way with the same version of Poetry than anothers, because their transitive dependencies Poetry is riding on top of are different.

I've noticed that if you simply run poetry update to update Poetry as described in the docs, then your dependencies tend to lag behind what you'd get if you did a fresh install or instead ran the more verbose poetry self lock && poetry self update.

I believe this could also help make triaging tickets better, as maintainers could know that if a user reports an issue with Poetry version X, the maintainer can reproduce the exact transitive dependencies that user has.

In this idea, we could remove poetry self lock, poetry self install, etc. and simply use poetry update to look for a new Poetry version, and install it and it's locked dependencies into the environment.

[finswimmer]

Hey @johnthagen,

this have been requested already earlier. While we agree this would be useful, we haven't found a working solution here. The main problems are poetry plugins. If we pin Poetry's own dependencies, these versions might be incompatible to the version a plugin needs.

Suggestions on how to solve this are welcome.

fin swimmer

[johnthagen]

Thanks for the feedback. That is a good point.

Would it help at all if there was a scheme for poetry plugins to target supported Poetry versions? Basically, put the responsibility on the plugin to have dependency compatibility with a specific version of Poetry.

There is also the binary distribution approach, publishing Poetry with something like PyInstaller or PyOxidizer, but that likely would cause even more issues for supporting plugins.

[johnthagen]

@finswimmer One short-term idea that would help mitigate this problem is if poetry self update also updated the transitive dependencies. The idea is that someone upgrading from Poetry version W -> version X using poetry self update would then get the same set of dependencies as someone who freshly installed Poetry version X at the same moment in time.

This would help alleviate dependency drift between uses updating Poetry and cleanly installing, while still only needing to learn/teach a single poetry self update command, without having to teach people poetry self lock && poetry self update.

Any thoughts on that as a shorter term first step?

[johnthagen]

Related, a proposal to remove poetry self all together

• deprecate poetry self commands #7872

[raratiru]

The commands used as a workaround for this issue:

poetry self lock
poetry self install --sync

Raise the following warning that will become an error in the future:

Installing dependencies from lock file

No dependencies to install or update

Installing the current project: poetry-instance (1.8.2)
Warning: The current project could not be installed: No file/folder found for package poetry-instance
If you do not want to install the current project use --no-root.
If you want to use Poetry only for dependency management but not for packaging, you can disable package mode by setting package-mode = false in your pyproject.toml file.
In a future version of Poetry this warning will become an error!

There is no poetry self install --sync --no-root command and no pyproject.toml file to avoid the error in the future.

[radoering]

@raratiru Actually, that's just a bug that will be fixed in #9302

[johnthagen]

As an example, filelock 3.15.0 broke Poetry install for all newly installed poetry instances (e.g., CI) for about 45 minutes until it was yanked:

• '_CountedFileLock' object has no attribute 'thread_safe' tox-dev/filelock#337

If Poetry had pinned dependencies, this would not have affected Poetry users.

[radoering]

@johnthagen What do you think about a pinned extra as prototyped in #9428?

[johnthagen]

@radoering interesting! So would the end use case for Poetry be

pipx install poetry[pinned]

To me it seems this could solve the issue? Would pipx be able to upgrade to the next locked release? I would think so?

[Secrus]

@radoering interesting approach, however I was thinking something different. My idea was to add a --pinned switch to the build command, that would overwrite the package metadata with pinned dependencies at build time. That way would be safer since we wouldn't need users to add [pinned] to their poetry installation setups everywhere.

[radoering]

Would pipx be able to upgrade to the next locked release? I would think so?

I did not test it but I would think so, too.

My idea was to add a --pinned switch to the build command, that would overwrite the package metadata with pinned dependencies at build time.

The big drawback is that users will have no chance to pick up bugfixes in dependencies without a new Poetry release in that case. Personally, I would prefer being able to update dependencies.

[johnthagen]

The big drawback is that users will have no chance to pick up bugfixes

Yeah, this is definitely one tradeoff, but I feel like in practice having the user community spread across many separate sets of dependencies also picks up new bugs that the Poetry devs have never been able to test locally or CI. I also wonder if we do go this route, perhaps the frequency of bug fix releases could be increased and along with those, new locked dependencies? So everyone is still in lock step, but getting somewhat regular patch updates.

Just some thoughts. Appreciate you all's work on Poetry!

[radoering]

I also wonder if we do go this route, perhaps the frequency of bug fix releases could be increased and along with those, new locked dependencies?

I would even say, we would have to release more often if we were to pin dependencies for all users. I'm thinking of cryptography security fixes for example. However, to be honest, I am afraid we cannot afford this considering the available time of maintainers at the moment.

If we chose the extra, probably, the pressure to do a new release would be smaller because users can update if they care. On the other side, users of the extra would probably want more frequent releases anyway; so not sure if the extra gives much value anyway if we do not release more often...