-
-
Notifications
You must be signed in to change notification settings - Fork 274
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move CI setup to pixi #2323
Comments
We would need to accommodate for {% for pkg in remote_ci_setup %}
pixi add pkg
{% endfor %}
pixi install An argument for keeping Miniforge around is that we were installing a conda+Python distribution anyway so why not just use that, but that might not be as relevant these days. conda-smithy has |
Note that any of these changes will effect all folks who run |
I don't think this is a tool question but rather the approach to installing versions. In the end, the choice also will lead to the different tools as each is better tailored for both approaches. I think our choice is
Pinning brings in speed and reproducibility at the cost of maintaining lockfiles, especially in the case where you have |
micromamba also handles lock files, but preserves the more traditional conda env workflow many people are used to. |
An alternative could also be to create an installer that already contains the base installation and use that instead of Miniforge. This is closer to our current approach and also includes some locking. |
I've been thinking about this for a bit because right now Miniforge release cycle is coupled to the operational status of ALL feedstocks. In the past we've been blocked by boa not being compatible with the latest conda, etc. That said, at that point we can also put that effort to switch to a single-binary provider, be it Pixi or micromamba. We just need to write down which packages are needed in an The "traditional env management" point I don't get, though. We are only creating a new environment on each CI run, which happens to be |
My comment on the traditional env management is in a way related to how we as developers want to maintain smithy and what our mental model for it is precisely. This interacts with users when they want to build locally and also debug builds using conda debug IIUIC. |
From our core call today, the two items we seem to be tackling here are:
|
On staged-recipes, I took the logs from conda-forge/staged-recipes#27748
|
On a feedstock, I took the logs from
|
Some timings for a |
I'd expect windows to be much faster because of the parallel download and extraction. We would get the same benefit with mamba 2.0. |
For Pixi, both macOS and Windows take under 30s from scratch: |
When you said micromamba was about the same for osx, what was the number? I'm curious. |
On staged-recipes macOS, the Miniforge approach takes 1m10s to 1m30s. With micromamba, that goes down to 1min; with Pixi, 30s. The differences with Windows are striking: from ~4mins to under a minute (micromamba) or even <30s (pixi). |
On Windows, you get <1m with micromamba v1, <2m with micromamba v2, <30s with Pixi. |
@isuruf unfortunately, no. With pixi / rattler the linking is done in a completely parallelized / pipelined way using async Rust (e.g. the whole download -> extraction -> linking per package is done in one go). Clobber issues are resolved after the transaction has executed (vs. ordered installation as in mamba / conda). So it's not yet possible to reach the same speeds with mamba / conda. We could build something bespoke with |
Thanks @wolfv @jaimergp and nice work all around! It appears that micromamba would be an easy win now and we could basically drop it in. We'd need to ensure it uses the same cache as conda/mamba in the docker container. It also appears we should either move to pixi or a micromamba-like tool built on the same components. @wolfv Is it possible to use pixi to create an env that has a name (and isn't a global env)? That'd make it easier to work with inside of smithy now I think, though I don't think this is a blocker. The osx and Linux builds share the same env management commands and we mount the feedstock dir and recipe dir separately which could make deciding on a directory for the env a bit tricky. |
@beckermr Here is some information on your question for pixi: You can activated an environment with After activation all
To activate similarly to |
This is suspicious. micromamba v2 takes twice that of micromamba v1 ?
I was talking about the difference between mamba/conda and micromamba. I don't understand why you are trying to talk about pixi/rattler-build. |
Might be related to simdjson parsers not being so optimized on Windows (e.g. simdjson/simdjson#847). 1.x used libsolv parsers, IIRC. I think there's a flag for that, let me check 🤔 Edit: nope, didn't change much. |
Turns out that part of the slowdown in Windows is due to installing to C drive. Changing to D cuts it in half. See conda-forge/conda-smithy#2076 (comment). |
Let's summarize more or less what I've found out today (no lockfiles):
I think the key points we can enforce now without too much controversy is:
Pixi would be awesome too for a sub-30s deploy, but it will require a bigger overhaul of how the infra is set up. Footnotes
|
Does it actually work when building packages? I thought that since the caches aren't shared, this is only moving the cost to a later stage. |
I think we can set This can be checked with some carefully tuned deps in meta.yaml and then check the logs in debug mode. |
Is there a specific set of features you miss to move to
You could detach environments from their folder with |
Well @ruben-arts it'd be nice to have a drop-in replacement for conda based on the rattler tools. Then transitions would be very easy for us. |
Btw the latest version of pixi (0.32.1) (and py-rattler) should be a lot faster again. We landed some very significant solver improvements. :) |
That's only for Otherwise I guess we'd need to craft something with |
If you want to experiment with https://github.com/marketplace/actions/setup-dev-drive as well, you may be able to get even more speed increase (that's not my action, I don't think I actually know the creator of it, but I did help with the underlying functionality, and the action's implementation looks reasonable at a quick glance). Basically, the Windows OS drive does a lot of processing on every file access that any other drive will (probably) not do, and a Dev Drive is even more optimised for this kind of use. Hopefully, one day Actions will use a Dev Drive by default, but I don't think they've enabled that yet. |
Is the Dev Drive functionality available on Azure too? Maybe we can replicate the action there. |
It needs a recent enough Windows version, that's all. I don't know the exact build number (probably around 10.0.26000). Even without it, that action will give you a similar speed up. uv is using their own script, which will be more portable than the Action. Again, they're falling back to a VHDX for now, but it only needs a |
That's amazing, thanks, I'll add an issue to conda-smithy so this is tracked later! |
I double checked and we don't need to re-set CONDA_PKGS_DIRS. It's enough to move |
Correct, if a |
I can't promise that's "everything" we need to implement a provider based on Pixi, but it would definitely get us closer, imo. |
I've written some basic pixi integrations in smithy at conda-forge/conda-smithy#2099. You can see them in action at conda-forge/dav1d-feedstock#21. Where is the pixi pkgs cache located? Does it use the same format and hashing mechanism for files as conda and mamba do? I'd like to repurpose the |
I think rattler uses a very similar cache layout/format as mamba and conda. Although rattler does have a different mechanism to lock the cache in case of multiprocess action. I havent tested this in a while though. You can configure the cache location with the RATTLER_CACHE_DIR environment variable. See https://pixi.sh/latest/features/environment/#caching |
Oh nice, there's an env var. This is what I tried, but unfortunately it fails to lock:
Note that |
Currently, when the CI starts, it does a number of things:
On macOS / Windows it would start with setting up a Miniforge, and then installing the same:
macOS script
Windows script
This could all be done in one swift step with
pixi
. Pixi is a single binary, that can be dropped anywhere, and that can either resolve + install, or use a lockfile for even faster & more controlled installation.We could create / maintain the
pixi.toml
+ lockfile externally to the feedstocks.Pixi + rattler-build have the added benefit that they share the cache (repodata & packages) but that is a minor concern.
The text was updated successfully, but these errors were encountered: