Add new feature resolver. #7820

ehuss · 2020-01-22T17:17:31Z

This adds a new resolver which handles feature unification independently of the main resolver. This can be enabled with the -Zfeatures flag which takes a comma-separated list of options to enable new behaviors. See unstable.md docs for details.

There are two significant behavior changes:

Ignore targets that are not enabled.
Do not unify features between build_deps, dev_deps, and normal deps.

The "forks" in the unit graph are handled by adding DepKind to UnitFor. The feature resolver tracks features independently for the different dependency kinds.

Unfortunately this currently does not support decoupling proc_macro dependencies. This is because at resolve time it does not know which dependencies are proc_macros. Moving feature resolution to after the packages are downloaded would require massive changes, and would make the unit computation much more complex. Nobody to my knowledge has requested this capability, presumably because proc_macros are relatively new, and they tend not to have very many dependencies, and those dependencies tend to be proc-macro specific (like syn/quote). I'd like to lean towards adding proc-macro to the index so that it can be known during resolve time, which would be much easier to implement, but with the downside of needing to add a new field to the index.

I did not update cargo metadata, yet. It's not really clear how it should behave. I think I'll need to investigate how people are currently using the feature information and figure out how it should work. Perhaps adding features to "dep_kinds" will be the solution, but I'm not sure.

The goal is to try to gather feedback about how well this new resolver works. There are two important things to check: whether it breaks a project, and how much does it increases compile time (since packages can be built multiple times with different features). I'd like to stabilize it one piece at a time assuming the disruption is not too great. If a project breaks or builds slower, the user can implement a backwards-compatible workaround of sprinkling additional features into Cargo.toml dependencies. I think itarget is a good candidate to try to stabilize first, since it is less likely to break things or change how things are built. If it does cause too much disruption, then I think we'll need to consider making it optional, enabled somehow.

There is an environment variable that can be set which forces Cargo to use the new feature resolver. This can be used in Cargo's own testsuite to explore which tests behave differently with the different features set.

rust-highfive · 2020-01-22T17:17:41Z

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

bors · 2020-01-22T23:08:34Z

☔ The latest upstream changes (presumably #7731) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton

Ok I've left a bunch of initial comments below, although I didn't get a chance to finish today unfortunately. Wanted to give you some time to respond though since I may be busy for a day or so. The major thing I didn't get finished reviewing is the internals of the new implementation of feature resolution, which I suspect would lift some confusion I have about DepKind, so feel free to defer to those internals if you'd like.

src/cargo/core/compiler/unit_dependencies.rs

src/cargo/core/profiles.rs

src/cargo/core/resolver/features.rs

src/cargo/ops/cargo_compile.rs

src/cargo/ops/cargo_clean.rs

src/cargo/core/resolver/features.rs

ehuss · 2020-01-24T17:57:06Z

While looking through some of your questions, I found a bug with cargo test and dev-dependency handling. It might require some non-trivial changes to fix. The problem is that in some cases the only difference between two lib.rs units is what it links against, and the current unit-deduplication logic can't handle that.

I'll try to fix that, and then respond to your questions (and try to add some comments to clarify confusing sections).

ehuss · 2020-01-24T23:51:39Z

So I think there may need to be significant changes, and I wanted to confirm that my assumptions make sense. I'll illustrate with an example project with these dependencies:

[dependencies]
dep = {version="1.0", features=["f1"]}

[dev-dependencies]
dep = {version="1.0", features=["f2"]}

With:

src/lib.rs
src/main.rs
tests/test.rs
examples/example.rs

cargo build will create a unit graph like the following:

(Hopefully that's pretty straightforward.)

The tricky part comes with cargo test. I would assume with decoupled dev-dependencies, we would want a unit graph like the following:

This causes lib.rs to be built 3 times: once as a test, once as a library linked with dev-dependencies (for other tests), and once as a library for examples and binaries (without dev-dependencies). This is done with the assumption that you want main.rs to be built the same as it does with cargo build.

If you agree with that, then the tricky bit comes with how to build that graph. The only difference for the lib.rs unit is what it links against. However, the Unit structure as it exists doesn't differentiate that. I'm thinking of adding DepKind to Unit to handle this properly. I'll try to avoid the extra build of lib.rs if you don't have any dev-dependencies, which will make it a little tricky (same with shared build-dependencies where there are no differences). I'm also wondering if I can change (or remove) UnitFor, which is essentially doing the same thing (the reason UnitFor exists is to encode these differences while still handling de-duplication). I'll try to put a fix together that does this, and see where it goes.

alexcrichton · 2020-01-24T23:56:34Z

So that makes sense to me if we take into account DepKind. I'm not entirely convinced though (for now at least) that we should be doing that. Our current feature resolution is to unify based entirely on just the structure of the crate graph, but I'm not sure how much further we need to go other than unifying only based on target. For example I'm not sure if it's a bad thing where, in your example, cargo build works the same but cargo test only builds dep with both features enabled.

ehuss · 2020-01-27T23:27:49Z

Yea, I'm not really clear on what the best behavior here is. How about this: have some sort of internal boolean (CompileFilter::need_dev_deps) that indicates whether or not dev-dependency features should be included. It would be enabled if any test/bench/example is being built. That means that cargo test would behave the same as it does today. It would also mean something like cargo build --all-targets would also unify (the same as it does today). But if you do a plain cargo build, it will build without unifying dev-dependencies.

I'm not sure if that might be too confusing or surprising. Perhaps with documentation it can be made clear?

I'm inclined to go that direction because it will be much simpler to implement. If we decide that dependencies should be "forked" and built multiple times with different features, we can add that in the future.

How does that sound?

alexcrichton · 2020-01-28T07:47:30Z

That sounds relatively reasonable to me yeah, the difference in --all-targets behavior is a little unfortunate but I think that basically mirrors what happens today (albeit probably in a slightly different place)

bors · 2020-01-31T20:49:46Z

☔ The latest upstream changes (presumably #7851) made this pull request unmergeable. Please resolve the merge conflicts.

ehuss · 2020-02-02T22:13:50Z

I did a fairly major rewrite to try to simplify things. Now that dev-dependency unification is a global setting, it all really boils down to whether or not something is a build dependency, so I just made that a bool. It's not as flexible, but I can always switch it back to something more complex if needed.

I also add -Zfeatures=all to try all options at once.

BTW, I'd like to squash this before merging, so don't merge as-is. This also now includes #7857 to fix some issues, but that PR should probably go first to simplify this.

shamatar · 2020-02-03T18:00:43Z

Does this PR also address #2589 and #2524 ? If not I'd try to look deeper into the 2589 myself as it would be quite convenient to have something like

[target.'cfg(not(target_os = "macos"))'.dependencies.a]
version = "0.5"

[target.'cfg(target_os = "macos")'.dependencies.a]
git = "https://github.com/foo/a_patched.git"

This example is actually much simpler than what you try to solve cause dependency graph just need a replacement of the node (actually just a path/version/git) based on target, not a feature, but may be it's also fixed as a by-product?

ehuss · 2020-02-03T18:10:12Z

@shamatar yes, it should close those issues (and many others). 2589 relates to features, not multiple sources. Multiple sources is #7753, and I believe that will be very difficult to fix.

shamatar · 2020-02-03T18:41:44Z

@ehuss

Ok, thanks for an answer. Another short question: if paths are not easy to fix, may be it would be able to introduce a syntax like

[target.'cfg(not(target_os = "macos"))'.features]
default = []

[target.'cfg(target_os = "macos")'.features]
default = ["macos_patch"]

or something similar. If it ever makes sense. At least it's a short-term fix for different paths problem.

ehuss · 2020-02-03T19:00:37Z

That is discussed in #1197.

bors · 2020-02-05T02:53:25Z

☔ The latest upstream changes (presumably #7855) made this pull request unmergeable. Please resolve the merge conflicts.

Fix BuildScriptOutput when a build script is run multiple times. When I implemented profile build overrides, I created a scenario where a build script could run more than once for different scenarios. See the test for an example. However, the `BuildScriptOutputs` map did not take this into consideration. This caused multiple build script runs to stomp on the output of previous runs. This is further exacerbated by the new feature resolver in #7820 where build scripts can run with different features enabled. The solution is to make the map key unique for the Unit it is running for. Since this map must be updated on different threads, `Unit` cannot be used as a key, so I chose just using the metadata hash which is hopefully unique. Most of this patch is involved with the fussiness of getting the correct metadata hash (we want the RunCustomBuild unit's hash). I also added some checks to avoid collisions and assert assumptions.

alexcrichton · 2020-02-07T19:36:19Z

Ok this is taking me longer to get to reviewing than I thought it might. I'll be gone most of next week as well. From my previous review and given the changes you've described implementing, I'd be fine landing this in the meantime and I can comment on this when I get a chance to. Or if you're ok waiting a week or so that's ok too!

matklad · 2020-02-12T10:34:25Z

I did not update cargo metadata, yet. It's not really clear how it should behave. I think I'll need to investigate how people are currently using the feature information and figure out how it should work.

I guess I can say a thing or two about IntelliJ and rust-analyzer, who are perhaps the two biggest users of cargo metadata at the moment. In short, the current Resolve in cargo metadata doesn't really make sense; it works good enough, but it is on the wrong level of abstraction.

Specifically, Resolve is on the level of packages, but what rust-analyzer & IntelliJ want is more close to units. For example, we would like to see "crate ./test/foo.rs depends on crate ./src/lib.rs with these cfg flags set".

I think we can make two steps to make sure that things "make sense".

Recast existing Resolve as an exact reflection of Cargo.lock. Ie, make it fully independent of the set of feature flags and the target, such that it is basically the flat list of packages from a lockfile + deps which explain why those packages are in lockfile. I am not sure if this would be actually useful, but at least that would be a clear defined thing which makes sense :-)
Add a separate thing (perhaps a new command even?) which outputs a format close to the UnitMap, which speaks not about Cargo packages, but rather crates in rust-reference sense, and dependencies between them. This would depend on the precise set of flags passed to Cargo (we'll get different outptus for cargo build vs cargo test) and would actually resemble a sort-of static build plan.

ehuss · 2020-02-12T18:16:21Z

@matklad I've been thinking the exact same thing. I've also been wondering if we could just extend cargo metadata or need some other command or flag (like how --build-plan works).

I'd really like a tool to visualize the unit graph, but that is difficult for any non-trivial project.

sunshowers · 2020-02-12T18:49:35Z

FYI cargo-guppy relies on metadata as well: https://github.com/calibra/cargo-guppy

alexcrichton

Ok apologies again for the delay in getting to review this.

This time around though I was able to process it a whole lot faster! The current logic makes sense to me, especially how dev-dependencies and their features are handled. I think what's implemented probably won't last until the end of time but it's certainly a helluva lot better than what we have now and is worth getting stabilized (IMO)

src/cargo/core/compiler/standard_lib.rs

src/cargo/core/resolver/features.rs

alexcrichton · 2020-02-19T20:35:02Z

src/cargo/core/resolver/features.rs

+        for fv in fvs {
+            self.activate_fv(pkg_id, fv, for_build)?;
+        }
+        if !self.processed_deps.insert((pkg_id, for_build)) {


This guard seems a bit suspicious to me. Just because we visited a package before here doesn't mean we didn't later add features which themselves add dependencies, right?

I'm thinking of something like cargo build -p deep-dependency -p root-dependency where it activates deep-dependency quickly but by activating root-dependency we transitively enable more features of deep-dependency

I think it is fine. After resolving deep-dependency once, other packages that add features to that deep-dependency will do so via FeatureValue::CrateFeature in activate_fv. This in turn checks if it is activating an optional dependency, and will recurse into that if necessary.

This is kinda fundamental to the way this is organized. If you notice, a few lines below it always skips optional dependencies. Whenever a dependency is encountered that enables an optional feature, it will enable it and recurse right away.

I believe this is structured somewhat differently from how the dependency resolver works. Presumably because it is figuring out the features as it goes along, and doesn't have the full graph built, yet.

Ah ok the skipping of optional deps below actually does make this sense yeah, sorry I missed that! It still feels a bit funky to me but because it's doing the skip I'm fine deferring to tests to verify the correctness of this. Mind adding a comment here clarifying what sort of recursion this is guarding against, and how more recursion is actually allowed via feature activation below in a limited way?

alexcrichton · 2020-02-19T20:39:41Z

src/cargo/core/profiles.rs

+    /// features of the non-host package (whereas `host` is true because the
+    /// build script is being built for the host). `build_dep` becomes `true`
+    /// for build-dependencies, or any of their dependencies.
+    build_dep: bool,


I'm thinking about this for a bit now. I understand (I think?) all the words in this comment, but I'm having trouble piecing together a situation about where this becomes relevant. Can you give a small example about where this subtle distinction is necessary?

Additionally, since host=false, build_dep=true probably doesn't make much sense, would it make more sense for this to be some sort of tri-value thing like:

enum Kind { Target, Host { build_dep: bool }, }

(please don't use Kind as I'm sure you're already thinking when reading that)

I added an example.

I'm reluctant to add an enum to represent the 3 states. All the code is oriented around simple booleans, and I think it would make it harder to follow if they were unified into one field. I usually agree that it is good to prevent impossible scenarios, but in this case I think it might be harder to understand.

Ok the example looks good to me, thanks! I'm also fine deferring to your thoughts about booleans!

One final question here, though. Technically we have two sets of fetures to deal with, one when the build script is compiled (the --feature flags) and one when it's executed (CARGO_FEATURE_*). Given that is_custom_build is conflating both executing and compiling the build script, I think we may want to be a little more nuanced/precise? I would expect that the compilation of the build script would use the host: true and build_dep: false set of features, but the execution of the build script would use host: false and build_dep: false as well, right? (ok maybe this doesn't have to do with build_dep?)

Also to clarify, in the example you listed here, shared_dep build.rs is only compiled once regardless of feature settings, right? And then it's executed twice maybe depending on feature settings/targets?

Hm, so it is working correctly, but you do point out that host isn't accurate for the RunCustomBuild unit. host only matters for the profile, and the RunCustomBuild unit goes through a separate code path for computing the profile. That is here where it manually fetches the profile (ignoring unit_for). All other units go through this path where it inspects unit_for to determine the profile. However, it is necessary for host to be true for RunCustomBuild so that all dependencies below it are marked as "host:true". Added some comments explaining this.

shared_dep build.rs is only compiled once regardless of feature settings, right? And then it's executed twice maybe depending on feature settings/targets?

It depends. With -Zfeatures=build_dep, and the features listed are different between the normal and build dep, then it will get built twice. The reason for this is that many scripts out there use #[cfg] and cfg! to detect features, instead of inspecting "CARGO_FEATURE_*". Only building it once would break a large number of build scripts. I don't think it would be feasible to avoid that.

Ah ok that makes sense, I know I've "used and abused" that CARGO_FEATURE_* is the "same" as #[cfg] in the past myself, so having a special case for this makes sense.

src/cargo/core/compiler/unit_dependencies.rs

alexcrichton · 2020-02-19T20:43:48Z

src/cargo/core/profiles.rs

+    ///
+    /// This is part of the machinery responsible for handling feature
+    /// decoupling for build dependencies in the new feature resolver.
+    pub fn with_build_dep(mut self, build_dep: bool) -> UnitFor {


As a minor nuance (and this was already present, just something I'm realizing now) the terminology of with_* doesn't quite fit here in my mind since I'd expect a function of that name to unconditionally configure build_dep with the argument provided, but here it's more of a unioning function. I'm not sure of a better name though!

Yea, I struggled a bit with the naming convention and wasn't happy with it. But I'm unable to think of anything particularly better. If you think of something, let me know!

I don't know if there is a common naming convention for this idiom (creating a copy and modifying a field in one step)? Maybe that's a sign it may be better to be explicit about creating a copy, and then call set_build_dep?

Eh it's a minor thing, I wouldn't stress out much over it. I definitely know of no naming convention for this, so it may be good to just highlight it in the documentation and move on. (the current call-sites all "look right" which is what matters)

Update cargo 11 commits in e02974078a692d7484f510eaec0e88d1b6cc0203..e57bd02999c9f40d52116e0beca7d1dccb0643de 2020-02-18 15:24:43 +0000 to 2020-02-21 20:20:10 +0000 - fix most remaining clippy findings (mostly redundant imports) (rust-lang/cargo#7912) - Add -Zfeatures tracking issues. (rust-lang/cargo#7917) - Use rust-lang/rust linkchecker on CI. (rust-lang/cargo#7913) - Clean up code mostly based on clippy suggestions (rust-lang/cargo#7911) - Add an option to include crate versions to the generated docs (rust-lang/cargo#7903) - Better support for license-file. (rust-lang/cargo#7905) - Add new feature resolver. (rust-lang/cargo#7820) - Switch azure to macOS 10.15. (rust-lang/cargo#7906) - Modified the help information of cargo-rustc (rust-lang/cargo#7892) - Update for nightly rustfmt. (rust-lang/cargo#7904) - Support `--config path_to_config.toml` cli syntax. (rust-lang/cargo#7901)

cbeck88 · 2020-02-24T21:13:44Z

Thank you so so much!!!

The [new cargo feature resolver](rust-lang/cargo#7820) won't unify features across build deps, dev deps, and targets. This solves our problem of needing to work around feature unification to avoid Clone implementations on private key material. This commit puts back our true dependency information into the Cargo.tomls. Specifically, it adds dev-dependencies that enable features that aren't enabled on normal dependencies. This will cause the feature unification to happen when using the old resolver, but the build will be correct under the new resolver. In order to maintain correct builds in CI, this commit also changes CI to use the nightly cargo with `-Z features=all` but still preserving the rustc toolchain specified in `rustc-toolchain`. Local developer builds will likely still use the `rustc-toolchain` version of cargo with the old resolver, but this shouldn't cause any problems for development.

rust-highfive assigned alexcrichton Jan 22, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 22, 2020

ehuss force-pushed the features2-split branch from 3eeda09 to 68304c2 Compare January 23, 2020 01:46

alexcrichton reviewed Jan 23, 2020

View reviewed changes

Eh2406 mentioned this pull request Jan 24, 2020

resolve build dependencies independently [WIP] #7761

Closed

4 tasks

sunshowers mentioned this pull request Jan 30, 2020

Add model for new feature resolver once it lands upstream facebookarchive/cargo-guppy#64

Closed

ehuss mentioned this pull request Feb 2, 2020

Fix BuildScriptOutput when a build script is run multiple times. #7857

Merged

ehuss force-pushed the features2-split branch from 68304c2 to 61f73a6 Compare February 2, 2020 21:38

ehuss force-pushed the features2-split branch from f07d254 to ae57a25 Compare February 5, 2020 03:11

ehuss force-pushed the features2-split branch from ae57a25 to 57867e7 Compare February 17, 2020 17:42

alexcrichton approved these changes Feb 19, 2020

View reviewed changes

This was referenced Feb 21, 2020

Tracking issue for -Z features=itarget #7914

Closed

Tracking issue for -Z features=host_dep #7915

Closed

Tracking issue for -Z features=dev_dep #7916

Closed

Update cargo rust-lang/rust#69358

Merged

valarauca mentioned this pull request Feb 24, 2020

build-dependencies and dependencies should not have features unified #4866

Closed

Dushistov mentioned this pull request Feb 25, 2020

refactoring: allow usage of UBX protocol related stuff only ublox-rs/ublox#3

Closed

bjorn3 mentioned this pull request Feb 29, 2020

[experiment] Cargo -Zfeatures=itarget crater run rust-lang/rust#69560

Closed

alexcrichton mentioned this pull request Mar 4, 2020

Regression in nightly handling features + debug assertions #7966

Closed

metajack mentioned this pull request Mar 25, 2020

Change to new, experimental cargo feature resolver diem/diem#3134

Closed

djc mentioned this pull request Mar 26, 2020

Allow specifying dependencies for individual artifacts rust-lang/rfcs#2887

Closed

macs-overdrv mentioned this pull request May 3, 2020

Update to latest cargo API encounters breaking changes cargo2nix/cargo2nix#110

Closed

matklad mentioned this pull request May 5, 2020

RA picking stable v nightly errors in dependencies rust-lang/rust-analyzer#4271

Closed

CameronNemo mentioned this pull request May 5, 2020

Add no_std feature capnproto/capnproto-rust#138

Closed

m-ou-se mentioned this pull request Sep 3, 2021

🎉 Rust 2021 celebration and thanks rust-lang/rust#88623

Closed

ehuss added this to the 1.43.0 milestone Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new feature resolver. #7820

Add new feature resolver. #7820

ehuss commented Jan 22, 2020

rust-highfive commented Jan 22, 2020

bors commented Jan 22, 2020

alexcrichton left a comment

ehuss commented Jan 24, 2020

ehuss commented Jan 24, 2020

alexcrichton commented Jan 24, 2020

ehuss commented Jan 27, 2020

alexcrichton commented Jan 28, 2020

bors commented Jan 31, 2020

ehuss commented Feb 2, 2020

shamatar commented Feb 3, 2020 •

edited

Loading

ehuss commented Feb 3, 2020

shamatar commented Feb 3, 2020

ehuss commented Feb 3, 2020

bors commented Feb 5, 2020

alexcrichton commented Feb 7, 2020

matklad commented Feb 12, 2020 •

edited

Loading

ehuss commented Feb 12, 2020

sunshowers commented Feb 12, 2020

alexcrichton left a comment

alexcrichton Feb 19, 2020

alexcrichton Feb 19, 2020

ehuss Feb 20, 2020

alexcrichton Feb 20, 2020

alexcrichton Feb 19, 2020

alexcrichton Feb 19, 2020

ehuss Feb 20, 2020

alexcrichton Feb 20, 2020

ehuss Feb 20, 2020

alexcrichton Feb 20, 2020

alexcrichton Feb 19, 2020

ehuss Feb 20, 2020

alexcrichton Feb 20, 2020

cbeck88 commented Feb 24, 2020

Add new feature resolver. #7820

Add new feature resolver. #7820

Conversation

ehuss commented Jan 22, 2020

rust-highfive commented Jan 22, 2020

bors commented Jan 22, 2020

alexcrichton left a comment

Choose a reason for hiding this comment

ehuss commented Jan 24, 2020

ehuss commented Jan 24, 2020

alexcrichton commented Jan 24, 2020

ehuss commented Jan 27, 2020

alexcrichton commented Jan 28, 2020

bors commented Jan 31, 2020

ehuss commented Feb 2, 2020

shamatar commented Feb 3, 2020 • edited Loading

ehuss commented Feb 3, 2020

shamatar commented Feb 3, 2020

ehuss commented Feb 3, 2020

bors commented Feb 5, 2020

alexcrichton commented Feb 7, 2020

matklad commented Feb 12, 2020 • edited Loading

ehuss commented Feb 12, 2020

sunshowers commented Feb 12, 2020

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbeck88 commented Feb 24, 2020

shamatar commented Feb 3, 2020 •

edited

Loading

matklad commented Feb 12, 2020 •

edited

Loading