option to fail if poetry.lock doesn't match pyproject.toml

[DetachHead]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

steps

1: add dependencies and run poetry update
2: change dependencies and run poetry install

expected

The command fails as you will install dependencies that are not reflecting the project configuration

actual

All you get is a warning

Feature Request

currently when you run poetry install with an outdated lock file, you get this warning:

Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

it would be nice if there was an option to make this an error instead of a warning. in my case i want the CI to fail if poetry.lock is outdated.

the warning can be easily missed when running in CI

[glencairn]

In my opinion, this feature would be counterintuitive and it wouldn't add much value. Lock files are used as the ultimate source of truth when installing/updating your project's dependencies. The fact that a lock file exists in your project, and that it's contents contain older or outdated versions of dependencies is perfectly valid and should never be considered an error. I think adding a command line option to remove the assertions made by a lock file would be confusing.

The way I see it, poetry already gives you the tools to figure out if your locked environment is out of date. For example, you could update your CI to run poetry show --outdated and capture the standard output. If it's empty, you're good to go. If it's not, then you have outdated packages.

[DetachHead]

In my opinion, this feature would be counterintuitive and it wouldn't add much value.

I disagree. If I change/add a dependency in pyproject.toml and forget to run poetry update, I’ll experience unexpected behaviour where the cause won’t be immediately obvious. Other similar tools such as npm and gradle fail when the lock file is outdated

I think adding a command line option to remove the assertions made by a lock file would be confusing

I’m not asking for an option to remove the lock file’s assertions, only an option to fail if pyproject.toml contradicts those assertions

[glencairn]

No, if you choose to commit your lock file this is not unexpected behavior and it's certainly not an error. As per the documentation...

Either way, running install when a poetry.lock file is present resolves and installs all dependencies that you listed in pyproject.toml, but Poetry uses the exact versions listed in poetry.lock to ensure that the package versions are consistent for everyone working on your project. As a result you will have all dependencies requested by your pyproject.toml file, but they may not all be at the very latest available versions (some of the dependencies listed in the poetry.lock file may have released newer versions since the file was created). This is by design, it ensures that your project does not break because of unexpected changes in dependencies.

It looks like the authors are pretty clear about the intended behavior-- if you don't want this behavior, don't commit your lock file.

---

And this already happens...

... to fail if pyproject.toml contradicts those assertions

If the version of a dependency in the lock file does not satisfy the constraints specified in the pyproject.toml file, you will get an error when running poetry install (the error description is pretty bad though). Just because a locked dependency is out of date does not mean that it violates your project's constraints.

The example below shows that behavior...

1. Add a new dependency: requests <2.20.0. This generates a lock file with requests version 2.19.1.
2. Manually change the constraint in pyproject.toml to >2.25.0. The version in the lock file (2.19.1) no longer satisfies the constraint. Error. No change to lock file.
3. Manually change the constraint in pyproject.toml to ^2.0. The version in lock file (2.19.1) now satisfies the constraint. The install works. Poetry is kind enough to warn you that a dependency in your lock file is out of date based on the constraints. No change to lock file because we didn't run add/remove/update.

🐟 ➜ poetry new test
Created package test in test

🐟 ➜ cd test/

🐟 ➜ poetry add "requests@<2.20.0"
Creating virtualenv test-CZsUjjpm-py3.9 in XXX

Updating dependencies
Resolving dependencies... (0.4s)

Writing lock file

Package operations: 13 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.6)
  • Installing attrs (21.4.0)
  • Installing certifi (2021.10.8)
  • Installing chardet (3.0.4)
  • Installing idna (2.7)
  • Installing more-itertools (8.12.0)
  • Installing packaging (21.3)
  • Installing pluggy (0.13.1)
  • Installing py (1.11.0)
  • Installing urllib3 (1.23)
  • Installing wcwidth (0.2.5)
  • Installing pytest (5.4.3)
  • Installing requests (2.19.1)

🐟 ➜ sed -i '' "s/<2.20.0/>2.25.0/g" pyproject.toml

🐟 ➜ poetry install
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

  SolverProblemError

  Because test depends on requests (>2.25.0) which doesn't match any versions, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

🐟 ➜ sed -i '' "s/>2.25.0/^2.0/g" pyproject.toml

🐟 ➜ poetry install
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

No dependencies to install or update

Installing the current project: test (0.1.0)

[radoering]

The lock command of poetry 1.2 will have a --check option (see documentation). Not exactly the requested feature but quite similar and probably sufficient for CI use cases.

[glencairn]

The lock command of poetry 1.2 will have a --check option (see documentation). Not exactly the requested feature but quite similar and probably sufficient for CI use cases.

This is great, but you're right in that it doesn't address OP's original issue. The dependencies in a lock file can satisfy the constraints specified in the pyproject.toml file and not be the most up-to-date. Solely relying on the exit code of poetry lock --check wouldn't suffice.

To solve the original problem, I think the best approach is a combination of poetry lock --check (to see if the locked environment satisfies all constraints) and poetry show --outdated (to see if the locked environment is up-to-date and has no outdated packages).

[DetachHead]

If the version of a dependency in the lock file does not satisfy the constraints specified in the pyproject.toml file, you will get an error when running poetry install (the error description is pretty bad though). Just because a locked dependency is out of date does not mean that it violates your project's constraints.

i just ran poetry install where the pyproject.toml has new dependencies in it that are not present at all in the poetry.lock file. i would expect an error message when poetry install isn't going to install all of the dependencies specified in pyproject.toml because as you said it does not satisfy the constraints specified in the pyproject.toml file

i just tried poetry lock --check and it picks up this issue

[KotlinIsland]

> poetry install
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

  SolverProblemError

  Because test depends on flake8 (*) which doesn't match any versions, version solving failed.

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

That error message drove me insane!
I completely missed the warning and just saw "flake8 (*) which doesn't match any versions".

What a deeply confusing message, as it's using the version number from my pyproject file, but matching it against the lock file?
Why doesn't it fail straight away with "Lock file not up to date with pyproject.toml, run poetry udpate."

Or at the bare minimum change the current error message to say where it's looking:

- Because test depends on flake8 (*) which doesn't match any versions, version solving failed.
+ Because test depends on flake8 (*) which doesn't match any versions in the lock file, version solving failed.

[glencairn]

If the version of a dependency in the lock file does not satisfy the constraints specified in the pyproject.toml file, you will get an error when running poetry install (the error description is pretty bad though). Just because a locked dependency is out of date does not mean that it violates your project's constraints.

i just ran poetry install where the pyproject.toml has new dependencies in it that are not present at all in the poetry.lock file. i would expect an error message when poetry install isn't going to install all of the dependencies specified in pyproject.toml because as you said it does not satisfy the constraints specified in the pyproject.toml file

i just tried poetry lock --check and it picks up this issue

Ah, I see. My example was only addressing the case where the version of an existing dependency is changed manually. I had not considered the case where a new dependency is added manually and not via poetry add. But I agree with you, poetry should be able to identify when a dependency specified in the pyproject.toml file doesn't exist in a lock file when running poetry install.

I am interested in the decision to add dependencies manually vs using poetry add. It's definitely a valid workflow but could you elaborate on why you prefer that workflow?

---

@KotlinIsland agreed 😆

[DetachHead]

I am interested in the decision to add dependencies manually vs using poetry add. It's definitely a valid workflow but could you elaborate on why you prefer that workflow?

In general i find it easier to modify config files directly so I don’t have to go through the cli

[KotlinIsland]

I am interested in the decision to add dependencies manually vs using poetry add. It's definitely a valid workflow but could you elaborate on why you prefer that workflow?

I normally use the UI of PyCharm which doesn't have a poetry add, the workflow is to: manually add, then a UI element lets you trigger lock/update.
So I'm not used to ever using poetry add. If I only ever use the cli I would be more familiar I suppose.

[heimalne]

I'm with @DetachHead on this issue to have it as a flag on poetry install (or even make the install fail by default), because your dependency setup is simply wrong when pypoetry.toml, the file you look into and expect to use the consistent versions from, mentions one version (or range) but the poetry.lock contains another inconsistent one. The important part here is "inconsistent", which includes being outside of the version range you expect.

@ashokdelphia has made a good first step MR

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.