Support constraints file when resolving versions

[honnix]

Pull Request Check List

Resolves: #3225

• Added tests for changed code.
• Updated documentation for changed code.

This depends on python-poetry/poetry-core#172

This PR tries to address #3225 by pulling in all constraint dependencies as root dependencies when resolving version, and removing all unreachable ones afterwards.

[honnix]

@abn @sdispater What do you think of this? Not sure why all the CI actions are not moving but I unit test it locally. Thanks and please let me know whether this change makes sense.

[abn]

Hi @honnix the tests are not running due to https://docs.github.com/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks, I have approved the runs.

As for the change itself, I'll be honest, I am not really enthused about this. I cannot see this solving a real problem. This fundamentally breaks the determinsm poetry brings to your environment.

Further, the constraints file can be used with a pip install of the poetry projects in environments that requirement them anyway. If poetry is installing the package with the proposed constraints, this means you are triggerring (or will have to trigger) a dependency resolution leading to effectively a new locked versions.

I wonder if being able to do something like this is what is really required.

[testenv]
whitelist_externals = poetry
install_command =
    pip install -U -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
commands =
    poetry install -vv --dev-only --no-root
    poetry run pytest {posargs} tests/

Note that this right now will not work since --dev-only removes all packages that are not marked dev.

[honnix]

@abn Thanks for looking at this PR. Here are some thoughts I have in mind and please take a look whether they make sense.

This fundamentally breaks the determinsm poetry brings to your environment

The constraints files don't result installing any additional packages that are not declared in pyproject.toml, and the purpose of it is to enforce package version constraints globally. If I understood it correctly, running poetry install/lock etc. with the same content of pyproject.toml and constraints files, and the same set of packages available in repositories, will result the same poetry.lock. So it is still deterministic.

If poetry is installing the package with the proposed constraints, this means you are triggerring (or will have to trigger) a dependency resolution leading to effectively a new locked versions.

If I did it correctly, when a poetry.lock exists, poetry install should just install things declared in poetry.lock faithfully; constraints files will not be read and used to automatically regenerate poetry.lock and it's up to the user to re-lock. If we worry about constraints file content changes that users are not aware of, we could version a constraints file, so for any given constraints file once it's generated it becomes immutable.

I cannot see this solving a real problem

For a CLI package to be distributed to a public repository, this is true because very often a CLI package will pin all dependencies and users don't need to care what to be installed, so there is no reason for CLI developers to use a constraints file.

When coming to a CLI built and distributed to an organization using private repository, it is possible due to security reasons that a certain version range of a package or the package completely is banned globally in the organization, and anything developed there should respect the global constraints. Developers of the CLI can simply pull in the constraints files and they will be informed when a solution cannot be found so they know there are some versions/packages they cannot use.

For a library to be distributed to a public repository, this is true most of the time because downstream consumers of the lib will likely not be aware of constraints files used by the library, and they only rely on the metadata of the package. So the developers of the library don't have incentive or reason to use a constraints file.

However it could also be the case that the library is a part of a collection of libraries that are supposed to work together in a single runtime environment (where everything being constrained by a set of constraints files), then it will be useful for the library developers to use the same set of constraints file so they are confident that the library will never violate dependency constraints that may result runtime failure. For example if the library developers update pyproject.toml for dependency A to be A = ^3.0.0 while there exists a constraints file saying A = ^2.0.0, they will not be able to find a solution and they will be informed that their library cannot use a later version of A because it may fail at runtime and may also cause transitive dependency troubles for their downstream consumers. This scenario is much more likely to happen in an organization using private repositories, where libraries are maintained by different teams but are supposed to work together in the same backend service or data pipeline.

For a backend service or data pipeline, this is actually extremely useful because figuring out a set of dependencies that for sure can work at runtime is not trivial. With a set of constraints files that are manual crafted by a group of engineers in an organization, the backend service or pipeline developers will not need to care about dependency versions, so for many cases they could just declare A = * in pyproject.toml and let the constraints files decide what a version of A should be. Of course the constraints files cannot cover all possible dependencies, but they will gradually cover the majority of use cases in the organization, leaving only a few things for a backend service or pipeline developer to figure out. This idea is nothing new in literature and has been proven to be a good success. For example Maven and Gradle in Java land, and pip. pip set off to build this feature for exactly the same reason (I have a teammate who was part of the team built this feature in pip).

wonder if being able to do something like this is what is really required.

Yeah it might be possible to hack it to work with a pip constraints file and use pip directly, but to me that is sub-optimal and kinda defeats the purpose of using poetry as a single tool.

[honnix]

@abn What do you think of those thoughts? Should I keep my hope that this PR might get accepted? With the change throughput of this project, this PR could quickly go outdated, so if it is very unlikely to get it accepted, I will not spend additional effort resolving all the conflicts. Thanks.

[honnix]

@abn Any update on this? I'm totally fine withdrawing this proposal. Please kindly let me know. Thanks.

[abn]

@honnix I haven't really looked into it yet. Unfortunately haven't had the time to focus on poetry the last couple of weeks. Will look at your response as soon as I have some time.

[honnix]

I think it is time for me to close this. Thanks.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.