Externally defined installation constraints à la "pip install -c constraints.txt"

[anjos]

I've been experimenting with pixi in the past few days, coming from both the mamba and pip ecosystems. It is quite a compelling tool and we're excited about its potential.

We have a bunch of packages we maintain and, eventually, install together. One of the pain points in putting our packages together is dependency resolution when combining 2, 3 or sometimes 4 different packages together. Often package "A" is tested on the CI using a certain version set of dependencies, but when it gets installed with package "B" alongside, a different set of dependencies is resolved which, occasionally, may lead to untested behaviour.

To avoid this, we typically maintain a set of CI "constraints" - a text file containing a fixed (common) set of dependency versions from all packages we maintain, that we know (ie. test) work well together. When we test packages on the CI, we enforce these constraints via pip using the -c install flag, so we're sure that at least one (reproducible) combination of dependencies will correctly resolve in case of multi-package environments.

In this context, we wonder how to achieve a similar effect with pixi. Any experiences you could share w.r.t. this?

Thanks a lot!

[ruben-arts]

In pixi the idea is to always develop with a lockfile (pixi.lock) So If you define the dependencies in your manifest file it should by design never fail in CI.

If you want the opposite of that, you could make a CI stage that removes the pixi.lock and tries to reinstall to see if there aren't any packages upstream that would break the installation for your users.

As an extra feature pixi allows for defining multiple environments. So you could use that to specify different set of dependencies in different settings. For example:

[project]
name = "multi_env"
channels = ["conda-forge"]
platforms = ["linux-64"]

[feature.ci.dependencies]
python = "3.8.*"

[dependencies]
python = "*" # Latest greatest

[environments]
ci = ["ci"]

Now you can run a different version of python in CI.

pixi run --environment ci python main.py

[anjos]

Thanks for the mega-quick answer, @ruben-arts!

I really enjoy the multiple environments in pixi as well. We use it for adding documentation, debugging, qa or testing features to develop our packages.

I guess I'm looking for a solution more along the lines of your second suggestion - remove the pixi.lock and try to re-install with a given constraint set (e.g. python~=3.11.0; numpy~=1.26.0; scikit-learn~=0.20.0).

I think the difference would be that such a constraint set is not necessarily hard-coded in the manifest itself. Depending on the amount of constraints, the number of combinations may be relatively high. Were pixi features thought to be used like this?

There is another point for us as well - we'd like to apply the same constraints across several packages.

If I understand, as of now, the only way to enforce this would be by modifying the manifest file to add these constraints as features and rebuild the lock file (by issuing a pixi install, and possibly following this by pixi run test).

Do you think that a feature that would look like pixi install --constraint "python~=3.11.0" could be useful in such use-cases? Or would it break the current design (feature/environments) in some other way?

[ruben-arts]

In the current design the idea is that the same manifest file would give the same output (lock file). Regarding any input from the user through the CLI. We don't allow for separate file inclusion in such behavior. All of this is done to promise reproducible setups on multiple levels.

The constraints logic is not an unknown idea in pixi but its not an exposed field in the manifest. So there would definitely be room to introduce that next to dependencies.

Pixi is by/for users so if there is a big push on the inclusion of separate files or configuration affecting the environments we should revisit our vision. But currently I think there are other paths the we could take to improve your experience. Input is always welcome!

[anjos]

In the current design the idea is that the same manifest file would give the same output (lock file). Regarding any input from the user through the CLI. We don't allow for separate file inclusion in such behavior. All of this is done to promise reproducible setups on multiple levels.

I guess this makes sense. Adding such a feature would have to rely on a different mechanism for tracking what went into the lock, indeed.

The constraints logic is not an unknown idea in pixi but its not an exposed field in the manifest. So there would definitely be room to introduce that next to dependencies.

Makes total sense, of course.

Pixi is by/for users so if there is a big push on the inclusion of separate files or configuration affecting the environments we should revisit our vision.

Great, thanks for sharing these thoughts here. You have already done quite an amazing job!

But currently I think there are other paths the we could take to improve your experience.

Saucy! Is there anything in particular that I should be looking for in the future?

[ruben-arts]

Saucy! Is there anything in particular that I should be looking for in the future?
Sorry for the confusion, I was talking about those constraints.