MSRV resolver has surprising behaviour in workspaces with mixed MSRV

[VorpalBlade]

Problem

As state by @epage in #13873 (comment) , a new issue was preferred for this. The issue is that the new MSRV aware resolver leads to surprising and suboptimal behaviour in a workspace with mixed MSRV:

• Call for Testing: MSRV-aware resolver #13873 (comment)
• Call for Testing: MSRV-aware resolver #13873 (comment)
• Call for Testing: MSRV-aware resolver #13873 (comment)

The issue is that (for whatever reason) you have a workspace where one (or more) of you crates is on a much older MSRV than the rest. Then the MSRV aware resolver will attempt to create a lockfile for all of the workspace based on that oldest version even if the dependency in question isn't even used by that old-MSRV crate. In certain cases this can cause there to be no possible solutions to the dependency resolution (as detailed by the comment linked in the third bullet point above).

There are valid reasons to have mixed MSRV, such as the different crates having different MSRV policy. or just that some crates naturally have solver development and don't need to be bumped to a newer MSRV as often as other crates.

I will here describe my use case where I run into this (I cannot speak for the cases where others run into this as what @Darksonn described):

I have a workspace I use to develop a command line program. Having it all the related crates (both the main binary and supporting libraries) all in one workspace makes development much easier, especially during the early phases when everything is evolving rapidly. Some of these crates are generally useful to others as well (file format parsers in my case).

I also have an agressive MSRV policy: only N-2 is guaranteed (but I won't bump MSRV gratuitously just for the sake of it, so actual MSRV can be much further back).

Eventually some of these crates will start to stabilise, especially the "leaf node" supporting crates with simple and clearly defined scope. There is only so many things you need in a library that parses a file format. That means it won't get breaking changes nor get published to crates.io as often. And also that there will be no reason to bump MSRV.

Now the problem here is that this leaf node dependency with older MSRV will now drag down the versions of dependencies that get used for my main crate of the workspace (the "flagship" binary crate).

Steps

1. Have a (virtual?, I don't know if it affects non-virtual as well) workspace with mixed MSRV
2. CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback cargo +nightly -Zmsrv-policy generate-lockfile

Possible Solution(s)

Possible solutions (and why they are suboptimal):

• I could bump the MSRV and publish releases to crates.io that are nothing but "bump MSRV". This leads to churn and "pointless" releases from the point of view of everyone else.
• I could selectively publish single crates from the workspace (update the MSRV of the dependency crates but don't publish them). It would be error prone:
  • I would need to ensure that I didn't start depending on any "unreleased commits". Sounds like I need a "cargo-reverse-semver-checks"?
  • This would be complicated since I automate publishing from CI using release-plz. Partial publishing is as far as I understand not supported with the automated CI/PR workflow, only when publishing locally from your own computer (which is bad practise).

The better option would seem to me that cargo would consider the entire dependency graph when writing the lock file and look at what dependencies are reachable from what crates, and compute the minimum needed MSRV for each dependency.

Notes

No response

Version

❯ cargo +nightly version --verbose
cargo 1.82.0-nightly (2f738d617 2024-08-13)
release: 1.82.0-nightly
commit-hash: 2f738d617c6ead388f899802dd1a7fd66858a691
commit-date: 2024-08-13
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Arch Linux Rolling Release [64-bit]

[epage]

btw Cargo is a multi-MSRV workspace but we only have two and one of those is "N" so we avoid most of this problem.

[epage]

So another way to put this is that when you are in a mixed-MSRV workspace, you get proper handling of your lowest MSRV but for all other MSRVs you get one of

• Older dependencies than necessary
• No MSRV-aware resolver

Depending on the dependency and how "wide" of a version requirement you have.

For older dependencies, the answer is "just pick a higher version req" but then that puts you in the second situation unless you make your MSRV "N"

[epage]

As this puts this use caae in the "No MSRV-aware resolver" situation, this is no worse off than we are today except for a communication / expectations issue.

The MSRV-aware resolver not being path aware is a fundamental limitation of the "fallback" strategy. Quoting from #13873 (comment)

When we are evaluating a version requirement, we don't know what all version requirements might apply. We filter and sort to find a version we pick and move on. If a chosen version is filtered out for another version req, we backtrack. fallback applies to the sorting step, not the filtering step, and never gets re-evaluated unless something else causes enough of a backtrack. If we sorted per version requirement, what you got would be dependent on the order the dependency tree was walked.

Originally we had considered incompatible-rust-versions = "deny" as the policy. The quality of the error messages from the current resolver is a blocker to that policy and stopped us from looking any further into this. In discussing fallback, other workflows came up that wouldn't work with deny, like no-MSRV-dev-dependencies or per-feature MSRVs.

The fallback mechanism was a compromise to get something that works now.

Once we have the PugGrub resolver, we'll also need to figure out how to map package versions back to workspace members back to know which MSRVs apply.

Neither of these facts mean we shouldn't consider ways of improving things.

[epage]

So far, the only idea I have is that we collect all MSRVs in the workspace and sort the version candidates into MSRV buckets. If a version req disallows versions in the lowest MSRV bucket, consider the next MSRV bucket on up.

So long as your version requirements are MSRV compatible, you'll get an MSRV-aware resolve. You just might get older versions than you absolutely need for MSRVs sake but you did say you are compatible with it, so I'm not too worried without mitigating context being brought up.

I just checked cargo add and it checks the MSRV of the package being added to so that will play nicely with this scheme though it may need tweaks for #10608.

[epage]

Before I forget, it would be very easy for dependency unification to cause a package with a high MSRV to make it so you can't verify a package with a low MSRV, so the mixed-MSRV workflow is already a bit "here be dragons" (which is why we need to be careful with #10608).

[VorpalBlade]

A third possible workaround would be to split out "stabilised crates" from your workspace into their own separate git repos. That is slightly non-trivial though (you need to use some arcane git commands if you want to keep the history, you need to set up CI anew, generate new and separate publishing secrets, possibly add contributors again if there are other contributors, etc etc). Doable, but there is a definite barrier to entry. Plus if you go back to a active development phase later on again you won't have the benefits. Monorepos are in many ways a lot nicer to work with.

Frankly it is a bit more work than I have the energy to deal with right now, but I might consider it. Bonus: It would also simplify MSRV CI testing (that is a bit complex in a multi-MSRV repo),

So far, the only idea I have is that we collect all MSRVs in the workspace and sort the version candidates into MSRV buckets. If a version req disallows versions in the lowest MSRV bucket, consider the next MSRV bucket on up.

This sounds similar to my graph suggestion (I guess it depends on what your internal representation is). If it is a full blown graph you could traverse the graph from the workspace crates, and mark each crate with the oldest MSRV it needs to support. Then you have your answer there. Though a complication is that an older/newer version of a dependency might have different dependencies transitive dependencies I guess.

Before I forget, it would be very easy for dependency unification to cause a package with a high MSRV to make it so you can't verify a package with a low MSRV, so the mixed-MSRV workflow is already a bit "here be dragons" (which is why we need to be careful with #10608).

I do put all dependencies (and features that aren't tied to feature flags of my own crates) in the workspace currently. This seems to help a bit with feature unification (without going to a full blown workspace hack, which I have been considering, though the publishing story for that seems complex).

I think it is fine if you get a (somewhat reasonable) error message for these tricky situations though. Unexpected behaviour is worse, but a "Can't resolve tokio due to conflicting MSRV requirements from crate_a and crate_b" is perfectly fine.

[epage]

A third possible workaround would be to split out "stabilised crates" from your workspace into their own separate git repos. That is slightly non-trivial though (you need to use some arcane git commands if you want to keep the history, you need to set up CI anew, generate new and separate publishing secrets, possibly add contributors again if there are other contributors, etc etc). Doable, but there is a definite barrier to entry. Plus if you go back to a active development phase later on again you won't have the benefits. Monorepos are in many ways a lot nicer to work with.

You don't have to go so far, you can have multiple workspaces in one repo. Still not great.

Without worrying about transient state like "stabilization level" (unless you coupled MSRV to that), you could break up your workspace by MSRV. I've been tempted to do that for cargo.

This sounds similar to my graph suggestion (I guess it depends on what your internal representation is). If it is a full blown graph you could traverse the graph from the workspace crates, and mark each crate with the oldest MSRV it needs to support. Then you have your answer there. Though a complication is that an older/newer version of a dependency might have different dependencies transitive dependencies I guess.

If I'm understanding, you are suggesting we traverse the transitive dependency graph, marking packages with the MSRV that they need to uphold. The problem is we are in the process of building that graph as we go and there can be multiple paths to a single dependency and by the time we walk down the second path to it, it is already selected. We'd need a solution that allows us to then reject that dependency and try again, like when you resolve for foo = "1.0" and foo = "1.1" with -Zminimal-versions. We first select 1.0.0 but then find it doesn't work, so back track and try again with. Worse, we'd then need to re-traverse foos dependencies and make sure they also satisfy the new MSRV. This only works with incompatible-rust-versions = "deny" which is blocked at this time.

[VorpalBlade]

Oh right, you need each graph node to be a set of possible versions (with associated MSRV versions). Or do search with backtracking (which likely won't be performant, it seldom is). And yes indirect dependencies (that may contain subgraphs that are also reachable via other paths, this is a DAG not a tree after all) makes this really annoying.

[Darksonn]

I would find it really useful if there was a flag to tell cargo what version to use as MSRV when resolving packages. That way, I can at least use that in cases that are too complicated for the tool to handle automatically. I also have a case where I want to resolve using a different version than the MSRV of any crate in the workspace. Specifically, I'd want to resolve using whichever rustc was the latest release when I last used the branch.

[epage]

That way, I can at least use that in cases that are too complicated for the tool to handle automatically

Say we move forward with my earlier proposal, I assume these use cases would still exist? Could you describe what the use case if? Is this something that would need to block any MSRV-aware resolver from being released and, if so, why?

I also have a case where I want to resolve using a different version than the MSRV of any crate in the workspace. Specifically, I'd want to resolve using whichever rustc was the latest release when I last used the branch.

Could you describe the use case and the motivation behind it?

Most likely, we'll want to split the conversation about manually specifying the rust-version to resolve to as a separate issue and it will be important to understand the problems that are being solved with it to know how to prioritize it. My inclination at this moment would be to not adjust the status from "future possibility".

[VorpalBlade]

Manually specifying the MSRV to use for resolution could be a viable workaround for me, since the crates that become stable for me tend to have fairly few and slow moving dependencies.

It could however be risky, should one of those dependencies bump MSRV if I don't notice it. In my case CI would notice that, but that isn't a given for everyone.

[epage]

Manually specifying the MSRV to use for resolution could be a viable workaround for me, since the crates that become stable for me tend to have fairly few and slow moving dependencies.

Likely, it would also be the least generally applicable solution. It works for your mixed MSRV use case because of how its being maintained. Its unclear how much we can generalize that to other mixed MSRV use cases.

[Darksonn]

Say we move forward with #14414 (comment), I assume these use cases would still exist? Could you describe what the use case if? Is this something that would need to block any MSRV-aware resolver from being released and, if so, why?

No, no, there's no need to block anything. The solution of having several buckets sounds sensible enough. I suggested manually specifying the version because it seems like there are a lot of different ways you could set up your workspace, and it seems difficult to support all of them with a single system. Having an escape hatch in cases like this is often a good idea.

One use-case is this: I generally want to check that my project works with both with the MSRV rustc and the latest rustc. When checking with the latest rustc I also want to use the latest deps. However, if I go back to an old branch where CI doesn't pass with the newest compiler (e.g., due to new warnings), then I may want to generate a lockfile using the newest rustc that compiles the project warning-free. I can do this by manually telling the tool to resolve according to whichever version that is.

But again, the intent behind the suggestion is as an escape hatch for all of the things we aren't able to think of. I don't necessarily think it's the best solution to any specific case.

[epage]

One use-case is this: I generally want to check that my project works with both with the MSRV rustc and the latest rustc. When checking with the latest rustc I also want to use the latest deps. However, if I go back to an old branch where CI doesn't pass with the newest compiler (e.g., due to new warnings), then I may want to generate a lockfile using the newest rustc that compiles the project warning-free. I can do this by manually telling the tool to resolve according to whichever version that is.

Interesting use case. Thanks for sharing!

[epage]

To lay out the situation more explicitly.

Say you have a workspace:

• higher
  • MSRV: 1.75
  • depends on
    • only-higher-incompatible 1.75
    • only-higher-compatible 1.70
    • shared 1.70
• lower
  • MSRV: 1.70
  • depends on
    • only-lower-compatible 1.70
    • shared 1.70

The registry has at least

• only-higher-incompatible 1.80 with MSRV 1.80
• only-higher-incompatible 1.75 with MSRV 1.75
• only-higher-incompatible 1.70 with MSRV 1.70
• only-higher-compatible 1.80 with MSRV 1.80
• only-higher-compatible 1.75 with MSRV 1.75
• only-higher-compatible 1.70 with MSRV 1.70
• only-lower-compatible 1.80 with MSRV 1.80
• only-lower-compatible 1.75 with MSRV 1.75
• only-lower-compatible 1.70 with MSRV 1.70
• shared 1.80 with MSRV 1.80
• shared 1.75 with MSRV 1.75
• shared 1.70 with MSRV 1.70

Today, we resolve everything without MSRV and get

• higher
  • MSRV: 1.75
  • depends on
    • only-higher-incompatible 1.80 (MSRV incompatible)
    • only-higher-compatible 1.80 (MSRV incompatible)
    • shared 1.80 (MSRV incompatible)
• lower
  • MSRV: 1.70
  • depends on
    • only-lower-compatible 1.80 (MSRV incompatible)
    • shared 1.80 (MSRV incompatible)

With the MSRV-aware resolver, we'll resolve with MSRV 1.70 and get:

• higher
  • MSRV: 1.75
  • depends on
    • only-higher-incompatible 1.80 (MSRV incompatible)
    • only-higher-compatible 1.70 (lower than needed)
    • shared 1.70
• lower
  • MSRV: 1.70
  • depends on
    • only-lower-compatible 1.70
    • shared 1.70

With my proposal in this issue, you'd resolve for 1.70 and 1.75 and get:

• higher
  • MSRV: 1.75
  • depends on
    • only-higher-incompatible 1.75
    • only-higher-compatible 1.70 (lower than needed)
    • shared 1.70
• lower
  • MSRV: 1.70
  • depends on
    • only-lower-compatible 1.70
    • shared 1.70

[epage]

When discussing this, the question came up of whether this was worth fixing. The MSRV-aware resolver has corner cases. How much should we patch over and hide those corner cases vs teaching users how it works and giving them the resources they need to fix issues. One of those potential tools could be to slip in an as-yet designed (but discussed) resolve.rust-version / CARGO_RESOLVER_RUST_VERSION.

To analyze this, we need to first break down the corner cases users will be hitting

• If a version requirement is too high, we'll resolve anyways
  • Use of cargo add should reduce the chance of hitting this (feat(add): Stabilize MSRV-aware version req selection #13608)
  • We error during compilation if incompatible with rustc (fix(msrv): Report all incompatible packages, not just a random one #13514 improved this)
  • cargo update reports added/upgraded packages that are incompatible with
    package.rust-version (as of feat(update): Report when incompatible-rust-version packages are selected #14401, further improvements pending)
• If a package is missing package.rust-version, we select the latest
  • Not much we can do about that but make the MSRV experience nice enough people are more likely to set it
• If a package is missing package.rust-version but its dependencies have it, we still select the latest
  • Due to the fallback policy, we cannot backtrack to handle this. We would need a deny policy which is blocked on the resolver implementation
• In a workspace with a mix of packages with an MSRV and without (e.g. internal-only or libs mixed with bins), those packages without an MSRV will have lower-than-needed dependency versions unless the dependencies' version requirements are high enough to be above the MSRV
  • This is again due to the fallback policy
  • cargo update reports new versions are available (feat(update): Tell users when they are still behind #13372 and feat: Report some dependency changes on any command #13561) but could be expanded further to specifically report that new, MSRV-compatible versions are available
• In a workspace with packages with a mix of MSRVs and without (e.g. libs mixed with bins), package with higher MSRVs will either resolve to lower-than-needed dependency versions or resolve the latest version, likely incompatible (e.g. this issue)
  • This is again due to the fallback policy
  • The improvements for "version req too high" and "with and without MSRV" apply here as well
• With mixed MSRVs, higher MSRV packages cannot use cargo features that would bump the MSRV of packages with a lower one
• With mixed MSRVs, higher MSRV packages must set their dependency version reqs to be compatible with lower MSRV packages that share those dependencies
• The more hand-editing needing, the more likely people generating MSRV-compatible lockfiles in CI are to use -Zminimal-versions instead, workaround around
  • Bad resolving of multi-major version requirements
  • Deps with no MSRV
  • Mixed MSRV in the workspace

Say we change the fallback policy to resolve by prioritizing packages according to each MSRV in the workspace as buckets, only the resolving-mixed-MSRV case would be affected. I suspect people will more quickly run into each of the other corner cases before this one, so holding back this solution for educating users seems like it will have little benefit.

Say we do nothing. We've improved a lot of reporting but users will need to manually run play whack-a-mole with cargo update --precise to eventually zero-in on their dependencies being MSRV compatible. As demonstrated in the RFC, that is a big chore.

Say we provide CARGO_RESOLVER_RUST_VERSION.

• Users can set this when changing a single package to get the lockfile to resolve correctly
  • This requires a lot more forethought and figuring things out
    • This creates a trap for contributors who have to know of the extra steps that are needed
  • This breaks down for any dependencies with lower MSRV packages as we'll resolve to too high of a version
• Users could instead switch to generating an MSRV-compatible lockfile for each MSRV in a CI job
  • This runs counter to the MSRV RFC which optimized instead for the lockfile being maintained with the MSRV
  • This won't work for those projects that can't generate a lockfile from scratch for testing MSRV
  • For projects that can do it today and intentionally do it, it'd be possible. A bit of a pain unless they get cargo-hack to help them with it

With the multi-MSRV bucketing solution, we would greatly improve the quality of life for people in the resolving-mixed-MSRV case. The happy path would be MSRV compatible dependencies out of the box, which is a goal of the RFC. We'll inform them when they could manually intercede to get newer versions. MSRV-only-in-CI users would get the right behavior in more cases, reducing the need to rely on -Zminimal-versions for something its not promising to do.

[joshtriplett]

The solution of using each crate's MSRV to affect only the things needed by that crate would be obviously correct, but the problem with that would be the complexity of implementation. (Which is why that solution might not be viable, at least with the current resolver.)

The proposal to do MSRV bucketing and prioritization still seems to me like it'll produce unwanted surprises, where in a mixed-MSRV workspace, it's almost like the old MSRV in one crate affects others, but worse, because it sometimes affects others but gets bumped forward if it fails to resolve.

So a crate with a brand new MSRV will still get old dependencies as long as they resolve.

(Assuming I'm understanding the proposed solution correctly.)

I definitely agree that the current state is not ideal, but it is at least really predictable.

It seems like a cargo-hack-style solution of resolving deps for different MSRVs separately would be closer to what people may often want/expect in a mixed-MSRV workspace, such as for testing that the crates with old MSRVs still build with those MSRVs.

On the simpler side, even just forcing the specification of a workspace-wide MSRV for dependency resolution would be the easiest way to get an easily predictable outcome when doing a single workspace-wide dependency resolution.

[VorpalBlade]

What would work for me is if it tried to use a newer MSRV and reported an error that some crates in the workspace use too old an MSRV for that to be viable. My policy is aggressive (and I don't do LTS branches or fixes for older releases on general) so if that happens I could simply bump the MSRV of those crates.

This is what currently happens once CI runs (cargo hack style, verifying the MSRV for each crate). So the other option would be to be able to opt out of the new resolver and use the traditional one (and catch the need to bump in CI). This wouldn't work for the new resolver though (as I understand it).

The really big issue is silent downgrading of crates with no obvious warning that this is because of MSRV (there is a lot of noise about not using the latest version due to indirect dependencies depending on older semver).

[epage]

The really big issue is silent downgrading of crates with no obvious warning that this is because of MSRV (there is a lot of noise about not using the latest version due to indirect dependencies depending on older semver).

My current reporting plan is, in precedence order

1. Red: incompatible with MSRV (fix(resolve): Report incompatible packages with precise Rust version #14457)
2. Yellow: MSRV-compatible upgrades are available (path-aware, unlike the resolver)
3. Plain: upgrades are available but blocked (semver, MSRV)