Handling large workspaces

[TheNeikos]

This is a continuation of #31, but it goes further and I don't want to disturb a perfectly fine solution for the asked question.

However, it is now a year later and the monorepo has grown...

And with it, we have now integrated nixos vm tests. Which means that to run the tests, we now need to use nix to build the software. And since any code change triggers a full rebuild of the whole (rust) part of the monorepo, tests take way longer than they should.

I can't show you the actual repo, but the construction is somewhat like this:

• crates/{foo_core, foo_lib}
• plugins/{plugin_bar, plugin_baz}, each depend on foo_{core,lib}
• cli/

(we actually have something like 20+ crates, but that is the pattern)

Now, when we test one of the plugins, the 'theoretically best' build chain would be:

foo_{core,lib} (unchanged) -> plugin_bar (has changes) -> cli (plugin_bar changed)

But instead we rebuild every single crate, which is quite problematic as some of the plugins have a fairly large surface area and Rust is also not the fastest to compile for large programs.

I would say this is a fairly 'typical' setup for projects like these, but the twist is that the individual crates cannot be put on a registry. This leaves us with two options for potentially resolving this:

• Split everything into their own git/flake, this would allow us to have a 'natural' split and help crane/nix be more efficient.
• Teach or help crane on how to deal with workspaces

I don't think #1 is something we should be doing in our personal case (might work for others!), so I am very much interested in seeing #2 happen, if possible/wanted.

This is why I opened this Discussion, to talk about potential solutions to the problems that occur due to the cargo and nix paradigms.

I am also interested in implementing/experimenting with/for any solutions that might come up.

[TheNeikos]

@ipetkov you mention in #31 (comment):

it may be possible to avoid the rebuilds above but it will take more manual writing of build expressions to achieve (I hope to one day have better automatic support for monorepos but the functionality isn't there yet)

and

I have some ideas on having crane analyze the workspace layout and automatically construct the right derivations so that each crate can be built with only the sources it relies on so unrelated changes do not cause mass rebuilds, but so far that's all it has been, ideas

I would love to hear them, if you would like to lay them out?

[TheNeikos]

I have spent the past few days analyzing the situation myself, so here is what I have learned during this time:

1. cargo had plans to support emitting a 'build plan' to help external build tools in the past (Tracking issue for build plan generation (--build-plan) rust-lang/cargo#5579) but this was then changed a few years later (Should build-plans be deleted? rust-lang/cargo#7614).
   • From those issues it is also clear that this build plan feature also was wrong in subtle ways and could not be relied on. So nothing one should seriously use.
2. Rust has a set of features that make it harder to determine statically what gets built or not.
   • 'features' in Rust are global per build graph. This means that 'later' crates in the graph can influence the build step of earlier ones.
   • Features can enable/disable dependencies, making it harder to statically derive which dependencies are 'active'
   • I am sure I missed some other details of less-used features I did not come around yet
3. cargo does not allow abstracting parts of it inner working
   • In my 'ideal' world, cargo would abstract away 'inputs' and 'outputs' and allow them to be systematically overridden if needed.
     • so you could say "crate foo is at this location, no matter what it says in the Cargo.toml"
     • Or so you could say "you can find the already built crate at , don't try building it again"
     • "Put the output of the compilation of crate foo at "

Some problems I've encountered

2) makes it quite hard to find out what the actual build graph is.

Imagine you have a workspace with several crates, how do you even get to know what the current members are without having to import all of the sources. cargo metadata which would gladly give you that information, requires fully-formed rust projects to be able to determine anything. This is because it infers a lot of things from the project source (like binaries/libraries). Even if you are not interested in information about that.

Potential solutions:

• Replicate workspace detection from cargo
  • This is brittle and can subtly break in the future, since cargo is not bound by crane
  • It is the 'easiest' to implement though, but will be probably in the future once the rules change
• Work with the cargo project to either:
  • specify (i.e. stabilize) how workspace resolution works
  • A new subcommand that does 'less' than metadata and only uses the existing information in Cargo.{toml,locks}s

3) Makes it hard to guide cargo to a 'nix correct' situation

In an ideal world I would be able to do this:

• Find every crate ('easy' -> get all Cargo.tomls)
• Put the sources of each crate into their own derivation (also doable with nix filter sources etc...)
• Determine the 'final' build outputs (In my OP that would be the cli crate) and calculate the complete build graph only using the Cargo.tomls again
• Use crate sources + the 'workspace dependencies derivation' to build each individual crate in their natural build order (they can depend between each other)
• Tell cargo to build the final output by using a substitution table you calculated that uses the binaries of the crates in the previous step
• Get a final binary

Now, crane already does this for non-workspace crates, but in that situation its luckily fairly simple (but still tricky!), since one rarely use path dependencies allowing you to (mis-)use the patches feature to tell cargo "don't worry, the sources are here". + then copying in the already built external dependencies (the caching of crane).

But several points are problematic as far as I understand?

• You can't easily tell cargo:
  • "please tell me the build graph"
  • "Only build foobar as if part of the build of cli"
  • "Disregard the path/git/crates.io spec and use this path "

What can we do?

Now, my biggest problem is that I feel that this point to get closer to being able to handle this situation we'd need more support from the cargo project. But I don't want to dump this info on them, and ask of them 'to figure out the support' or anything like that.

So my primary goal here would be what the exact requirements are of cargo, so that we (or I) could then chat with the cargo project so that we can work out a potential solution.

Since you, @ipetkov, also have more experience with abstracting cargo in nix, I want to see what you think and/or any other ideas you might have?

[ipetkov]

This is a great analysis of the status quo so I'll try not to repeat much of what you've already said!

Replicate workspace detection from cargo

IM(H)O this is a non-starter, both from a maintenance perspective of re-implementing cargo's functionality, but also in becoming a bottle neck whenever new features are added to cargo. There are other projects which take this approach and since crane's goals are to embrace/delegate to cargo I'd rather not go down that path.

You can't easily tell cargo "please tell me the build graph"

You cannot easily get a build plan from cargo (e.g. stability/not fully fleshed out), but you can get a dependency graph (via cargo metadata). There is some wrangling that needs to happen to get an actual topological sort of the crate dependencies, though there are crates like guppy which can help.

Having to "ask cargo" about the workspace definitely decouples from having to know how to crawl a workspace and make sure .config/cargo.toml is properly applied etc., but it does end up requiring either IFD or having projects commit the computed build plan in their repository. It's not ideal, but IMO it's workable.

So my primary goal here would be what the exact requirements are of cargo,

Ignoring IFD-related headaches, I think the toughest part is answering "which set of files are truly necessary to build some specific crate" so that the derivations can properly filter out irrelevant files when building crates one a time. though I'm not entirely sure cargo itself can answer that question (really it's something rustc might know better, but even then it might not).

Most crates just have src/**/*.rs files, but sometimes they will include_bytes! from another path (which is sometimes in a parent directory). Then there's also build scripts which can look at arbitrary things as well.

My idea around handling this was to use a default heuristic (i.e. src/**/*.rs) but give the project maintainer some API hooks for including additional files (e.g. a callback method for each crate being built where they can provide a filter which could include that ../../README.md file they have included in their rustdocs). Sure we can get clever and try to parse the rust source and look for things like include_bytes! which would make the default heuristics much better, but at the end I can still see a need for having that escape hatch for when stuff like build scripts can't be fully analyzed

[ipetkov]

Also worth mentioning that I don't want to take over handling feature resolution and binary selection from cargo (there's too many edge cases and surprising behavior with how features can interact, I'd rather the project maintainer be explicit in specifying cargo build --features whatever --bin blah).

I was imagining buildWorkspace's API being more like "given this fixed cargo invocation, run it on a chain of derivations rooted at crate X where each derivation progressively adds more crates' sources and reuses the artifacts from the last". That would also give projects maximal flexibility in determining which crates/binaries are built with which features while also avoiding some of the pitfalls with unexpected feature flag unification

[TheNeikos]

Thanks for the reply!

IM(H)O this is a non-starter, both from a maintenance perspective of re-implementing cargo's functionality

I 100% agree. I wrote it down, so it could be crossed out haha. That way we can focus on other paths and stay focused.

I'd rather the project maintainer be explicit in specifying cargo build --features whatever --bin blah).

I’d say we’re aligned here. The core idea of this should be to work with cargo and marry it with the nix way of building things. So users should just use cargo „normally“ as much as possible. This also eliminates discrepancies when run during dev and then later in the sandbox.

———

If I understand correctly then, as a first step. We’d need a way to get cargo metadata without pulling in all source files.

As well as, optionally, a way that you can detect which files are needed to build crate X. Also without depending on unneeded files…

fun challenge! I’ll see if I can make some progress there.

[TheNeikos]

Alright, @matthiasbeyer and I have made some progress on this!

It actually works for a simple workspace!... except the final step where cargo falls over:

workspace-unknown> ++ command cargo build --release --message-format json-render-diagnostics -vvv
workspace-unknown>        Fresh unicode-ident v1.0.9
workspace-unknown>        Fresh autocfg v1.1.0
workspace-unknown>        Fresh pin-project-lite v0.2.9
workspace-unknown>        Fresh proc-macro2 v1.0.60
workspace-unknown>        Fresh quote v1.0.28
workspace-unknown>        Fresh syn v2.0.18
workspace-unknown>        Fresh serde_derive v1.0.164
workspace-unknown>        Fresh tokio-macros v2.1.0
workspace-unknown>        Fresh serde v1.0.164
workspace-unknown>        Fresh tokio v1.28.2
workspace-unknown>        Fresh another v0.1.0 (/build/final-workspace-source/others/another)
workspace-unknown>        Fresh bar v0.1.0 (/build/final-workspace-source/bar)
workspace-unknown>        Fresh foo v0.1.0 (/build/final-workspace-source/foo)
workspace-unknown>     Finished release [optimized] target(s) in 0.01s
workspace-unknown> error: failed to create file `/build/final-workspace-source/target/release/libanother.d`

The general idea of what we did was the following:

• Get the metadata of the workspace (using crane's dummySrc)
• Use the metadata to find the resolved features + dependencies of all workspace members
• For each workspace member do the following:
  • Create a src directory that only depends on their subtree of the project + all local workspace dependency subtrees
  • Create an artifact derivation consisting of ([all dependencies] + [local workspace dependencies])
  • Patch the Cargo.toml to have the combined direct dependencies of all workspace members and their features
  • Build it using the new src + artifact derivation
  • Keep the artifacts
• Finally use the original source, patch the Cargo.tomls of all crates
• Run the final 'cargo build' using buildPackage which should in theory then copy the outputs and voila

Sadly the final final step doesn't work yet. But we are working on why this happens here, as:

1. another is "fresh" so why is it creating files for it?
2. inheritCargoArtifactsHook does the right thing w.r.t. permissions

crane/lib/setupHooks/inheritCargoArtifactsHook.sh

Lines 35 to 43 in 7809d36

	cp -r "${preparedArtifacts}" \
	--dereference \
	--no-target-directory "${cargoTargetDir}" \
	--preserve=timestamps \
	--no-preserve=ownership
	# Keep existing permissions (e.g. exectuable), but also make things writable
	# since the store is read-only and cargo would otherwise choke
	chmod -R u+w "${cargoTargetDir}"

The investigation continues...

[TheNeikos]

Huzzah! We solved the last problems for a first succesful test. And are now fixing the kinks!

If you want to check it out, you can find it here:

https://github.com/TheNeikos/crane-workspaces

Once we feel its ready, I'll try to see if you are interested in integrating this solution.

[ipetkov]

Super exciting, I'll check it out! 👀

[therishidesai]

We do something similar to crane-workspaces with our monorepo and crane but we are running into an issue where the shared dependencies nix artifact is huge and when building with remote builders it spends a good amount of time nix copying that shared artifact. I think a better approach is to have each dependency be its own path in the nix store so you don't have a giant workspace-deps nix store path and you could also get better individual dependency caching. I'm not entirely sure the best approach to doing this with crane but I wonder if craneLib.buildDepsOnly could be used to just build one of the deps. We could split a Cargo.toml into many cargo toml's with just the one dependency and use the existing crane dummy strategy.

[ipetkov]

You could try compiling each crate in its own derivation but sooner or later you'll have crates which need the artifacts of other previously built crates and at some point the root of the workspace would need it all, unless I'm misunderstanding something.

It's also worth noting that the way cargo does feature unification means that even if you could build a bunch of dependencies in isolation, you would need to be dead sure the same features are always turned on (which is difficult to do). Here's a quick illustrative example:

• A depends on B feature=foo
• C depends on B feature=bar
• D depends on A and C

When you build D, B will be compiled with both features enabled. But if you were to only build A in isolation it would build B with only one feature enabled. Even if you merged the artifacts together they might not have built the correct thing...

[dpc]

Mainly due to feature unification, but not only, I don't think splitting deps build is worth it. It also misses opportunity to compress everything well.

Make sure you're using crane's use-zstd artifact mode, especially since recently it gained incremental build capacity.

At Fedimint we have a fairly huge project 66k LoC of Rust code, hundreds of deps, heavy CI (including cross-compilation to android targets, building docker containers, running e2e tests via Nix, wasm tests, you name it. Everything goes zoom and it's a Open Source project on a tiny budget.

Check out Flakebox, join our Matrix channel and I'd be happy to help you figure out a good crane-based setup.

[TheNeikos]

I’ve been Hacking at this problem and should soon have something to show for! I believe I got Feature unification working, and am working on not having to copy things around (too much).

[TheNeikos]

Alright, @dpc @therishidesai, @ipetkov. I've just pushed the latest version.

This manages to build a fairly large and complicated workspace with inter-dependencies and multiple binaries and outputs the built parts of it.

Check out the flake.nix for an example of how to use it. The resulting derivation has a passthru that exposes the individual workspace members. So you can do myBigProject.workspaceMembers.oneBinary and it will do the right thing™ (aka only build the workspace members that are needed to produce oneBinary.

I would love to hear some feedback and ideas on how this could be done cleaner!

Currently it is doing the following:

• Gather all the metadata and fix it up using the provided rust program. It could be technically replaced with another language that doesn't need compiling (not sure what stdenv provides?)
• Make a global dummy of the workspace. This is just mkDummySrc, but it also removes all dev-dependencies. This is needed for now because I couldn't get cargo to tell me what dependencies are 'dev' only and which aren't. Maybe this can be inferred through other means (like generating metadata twice? PRs welcome!)
• Define the 'workspaceMembers' attrset. This is lazy thanks to nix, so we can use nix itself to have it do the right ordering! Internally it is depending on both the 'global workspace dependencies' as well as the relevant crates from workspaceMembers. This being recursive is not a problem in practice, as for normal builds, cargo doesn't allow cyclic dependencies.
  • Each workspaceMember first patches in its own source + that of its complete workspace dependencies. (So direct + transitive)
  • crane and crane-workspaces then copies/links in the artifacts from its complete workspace deps (again, direct + transitive)
  • We build normally
  • It compares the target folder before/after and copies out only the files that have changed into a special output named 'artifacts'
  • It also installs any potential executables that this workspace member has built
• The output of 'buildWorkspaces' is workspaceMembers._workspace, which is the 'meta' derivation that shouldn't build anything at all. It does the same thing as above, and then only copies out executables. (It does not copy out artifacts)

I would love to hear ideas/suggestions and welcome any patches that make this cleaner, saves space/time and/or makes it more idiomatic nix. I am definitely not an expert 'packager' and was frankly amazed at how well crane was written (kudos @ipetkov!!).

Current tradeoffs:

• We are still copying artifacts a lot of the time
  • These do not propagate though, so if the builder directory gets cleaned up, these file should disappear as well!
• No tests, benchmarks, docs, etc...
  • I explained above why, but this seems like something that could be done!
• The copying is currently very slow, I am not exactly sure why

[dpc]

So this is kind of trying to achieve what https://crates.io/crates/cargo-hakari does, right? Just in crane settings (so extra source filtering)?

I don't think it should be named buildWorkspace. Because it doesn't build the whole workspace. :D

Seems very useful in certain setups, but I'm a bit intimidated by the amount of extra logic. In my projects so far I was able to either ride on building the whole workspace or a subgroup of it, but I can see this not scaling as well.

All in all I'd love to see it in crane, possibly with some warnings about being experimental. Then I could give it a chance in projects where it makes sense.