cargo tree panics with "activated_features for invalid package: features did not find PackageId" when a binary dependency is specified with an explicit target key

[bstrie]

Problem

For convenience I have also uploaded the failing code here: https://github.com/bstrie/bindeperror4

For the following code:

# Cargo.toml
[package]
name = "mycrate"
version = "0.0.0"
edition = "2021"

[dependencies]
mybindep = { path = "mybindep", artifact = "bin", target = "x86_64-unknown-linux-gnu" }

// src/main.rs
fn main() {
    env!("CARGO_BIN_FILE_MYBINDEP");
}

# mybindep/Cargo.toml
[package]
name = "mybindep"
version = "0.0.0"
edition = "2021"

// mybindep/src/main.rs
fn main() {}

Attempting to use cargo tree produces the following:

thread 'main' panicked at 'activated_features for invalid package: features did not find PackageId { name: "mybindep", version: "0.0.0", source: "/home/ben/code/scrap/bindeperror4/mybindep" } NormalOrDevOrArtifactTarget(None)', src/tools/cargo/src/cargo/core/resolver/features.rs:318:14

Expand for full `cargo tree` backtrace.

thread 'main' panicked at 'activated_features for invalid package: features did not find PackageId { name: "mybindep", version: "0.0.0", source: "/home/ben/code/scrap/bindeperror4/mybindep" } NormalOrDevOrArtifactTarget(None)

Stack backtrace:
   0: <anyhow::Error>::msg::<alloc::string::String>
   1: <cargo::core::resolver::features::ResolvedFeatures>::activated_features_int
   2: cargo::ops::tree::graph::add_pkg
   3: cargo::ops::tree::graph::add_pkg
   4: cargo::ops::tree::graph::build
   5: cargo::ops::tree::build_and_print
   6: cargo::commands::tree::exec
   7: cargo::cli::main
   8: cargo::main
   9: std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>
  10: std::rt::lang_start::<()>::{closure#0}
  11: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/core/src/ops/function.rs:280:13
  12: std::panicking::try::do_call
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panicking.rs:492:40
  13: std::panicking::try
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panicking.rs:456:19
  14: std::panic::catch_unwind
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panic.rs:137:14
  15: std::rt::lang_start_internal::{{closure}}
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/rt.rs:128:48
  16: std::panicking::try::do_call
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panicking.rs:492:40
  17: std::panicking::try
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panicking.rs:456:19
  18: std::panic::catch_unwind
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panic.rs:137:14
  19: std::rt::lang_start_internal
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/rt.rs:128:20
  20: main
  21: <unknown>
  22: __libc_start_main
  23: <unknown>', src/tools/cargo/src/cargo/core/resolver/features.rs:318:14
stack backtrace:
   0: rust_begin_unwind
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/core/src/panicking.rs:142:14
   2: core::result::unwrap_failed
             at /rustc/de1bc0008be096cf7ed67b93402250d3b3e480d0/library/core/src/result.rs:1785:5
   3: cargo::ops::tree::graph::add_pkg
   4: cargo::ops::tree::graph::add_pkg
   5: cargo::ops::tree::graph::build
   6: cargo::ops::tree::build_and_print
   7: cargo::commands::tree::exec
   8: cargo::cli::main
   9: cargo::main

Curiously the panic message is very similar to the panic reported in #10431 . Since that issue has already been fixed, it's possible that the solution for this issue may be similar to the solution for that one.

Note that the following diff causes cargo tree to run as expected:

- mybindep = { path = "mybindep", artifact = "bin", target = "x86_64-unknown-linux-gnu" }
+ mybindep = { path = "mybindep", artifact = "bin" }

So the presence of the explicit target key is crucial here. This is similar to the circumstances of #10526 and #10525 , so it further suggests something interesting about how the target key is being handled. It may be that fixing one of these will fix all of these.

cc @Byron

Version

cargo 1.62.0-nightly (edffc4a 2022-04-19)
release: 1.62.0-nightly
commit-hash: edffc4ada3d77799e5a04eeafd9b2f843d29fc23
commit-date: 2022-04-19
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Pop!_OS 21.10 (impish) [64-bit]

[bstrie]

For now, note that the original cargo-tree tool still works and doesn't have this problem: https://github.com/sfackler/cargo-tree

[weihanglo]

This issue can be fixed by duplicating the logic of resolver to determine the FeatureFor artifact deps. The following patch demonstrates it:

Click to view the patch

diff --git a/src/cargo/ops/tree/graph.rs b/src/cargo/ops/tree/graph.rs
index 20a9ca0b6..c975e6fcc 100644
--- a/src/cargo/ops/tree/graph.rs
+++ b/src/cargo/ops/tree/graph.rs
@@ -387,10 +387,38 @@ fn add_pkg(
         let dep_pkg = graph.package_map[&dep_id];
 
         for dep in deps {
-            let dep_features_for = if dep.is_build() || dep_pkg.proc_macro() {
-                FeaturesFor::HostDep
-            } else {
-                features_for
+            // This is somehow similar to
+            // <https://github.com/rust-lang/cargo/blob/5181f995d08e72da457bf8a2e6c9c8d0bba09088/src/cargo/core/resolver/features.rs#L800-L852>
+            // without the consideration of `dep = { …, lib = true }`
+            //
+            // TODO: decide how to display artifact deps within cargo-tree,
+            // including handling artifact deps with `{ …, lib = true }`
+            let dep_features_for = match dep
+                .artifact()
+                .and_then(|a| a.target())
+                .and_then(|t| t.to_resolved_compile_target(requested_kind))
+            {
+                // Dependency has a `{ …, target = <triple> }`
+                Some(target) => FeaturesFor::ArtifactDep(target),
+                // Get the information of the dependent crate from `features_for`.
+                // If a dependent crate is
+                //
+                // * specified as an artifact dep with a `target`, or
+                // * a host dep,
+                //
+                // its transitive deps, including build-deps, need to be built on that target.
+                None if features_for != FeaturesFor::default() => features_for,
+                // Dependent crate is a normal dep, then back to old rules:
+                //
+                // * normal deps, dev-deps -> inherited target
+                // * build-deps -> host
+                None => {
+                    if dep.is_build() || dep_pkg.proc_macro() {
+                        FeaturesFor::HostDep
+                    } else {
+                        features_for
+                    }
+                }
             };
             let dep_index = add_pkg(
                 graph,

However, I found another panic in cargo tree when it meets all the followings conditions at once:

• a dependency of an artifact dependency is platform specified
• artfiact dep itself is { target = <specified> } with the same platform as its own dependency
• the platform is not activated.

Here is a reproducer in Cargo's own test dialect:

Click to view the reproducer

#[cargo_test]
fn artifact_dep_target_specified() {
    if cross_compile::disabled() {
        return;
    }
    let alt = cross_compile::alternate();
    let p = project()
        .file(
            "Cargo.toml",
            &format!(
                r#"
                    [package]
                    name = "foo"
                    version = "0.1.0"

                    [dependencies]
                    bar = {{ path = "bar", artifact = "bin", target = "{alt}" }}
                "#,
            ),
        )
        .file("src/lib.rs", "")
        .file(
            "bar/Cargo.toml",
            &format!(
                r#"
                    [package]
                    name = "bar"
                    version = "0.1.0"

                    [target.{alt}.dependencies]
                    baz = {{ path = "../baz" }}
                "#,
            ),
        )
        .file("bar/src/lib.rs", "")
        .file(
            "baz/Cargo.toml",
            r#"
                [package]
                name = "baz"
                version = "0.1.0"
            "#,
        )
        .file("baz/src/lib.rs", "")
        .build();

    p.cargo("tree -Z bindeps")
        .masquerade_as_nightly_cargo(&["bindeps"])
        .with_stdout("") 
        .run();
    // thread 'main' panicked at 'no entry found for key', src/cargo/ops/tree/graph.rs:387:23
}

Personally, I don't think this should panic, as cargo tree is used only for displaying dependency information. However, I won't fix it and instead pause here until a broader issue gets resolved: How does Cargo display artifact dependencies in cargo tree properly? Some questions might be:

• Should we annotate artifact dependency nodes with a special (artifact deps) suffix?
• Should we show the information about which target platform an artifact dependency will be built on?
• How do we want to display the node of an artifact dependency with { …, lib = true }?

I haven't thought about it much. Will be back when I get more time. If anyone comes up with great ideas to show artifact dependencies in cargo tree, I am all ears!

[bstrie]

@weihanglo , my opinions for each question:

Should we annotate artifact dependency nodes with a special (artifact deps) suffix?

There are multiple possible values for the artifact field, so I wouldn't mind having a (bin) or (staticlib) or (cdylib) suffix.

Should we show the information about which target platform an artifact dependency will be built on?

No, I think that's a detail that can be easily answered by checking Cargo.toml. But if you did decide to display this information, I would include it as part of the above annotation, e.g. (wasm32-wasi bin) or (x86_64-unknown-linux-musl staticlib).

How do we want to display the node of an artifact dependency with { …, lib = true }?

I think it should show up multiple times in the tree graph. The lib node should show up as an entirely normal dependency, whereas the artifact node should be annotated as mentioned above.

[bstrie]

To elaborate on the above, I think it would be nice if users could think of this:

bar = { version = "*", artifact = "bin", lib = true }

...as though it were this:

bar_1 = { package = "bar", version = "*" }
bar_2 = { package = "bar", version = "*", artifact = "bin" }

...with the only difference that using the lib = "true" form doesn't force you to use renames via package = to avoid a collision. And since it's natural that the second example would have two lines in the cargo tree output, it seems natural that the first example should as well, and the (bin) bit is a convenient disambiguator.

[bstrie]

(And to elaborate even further, I think that #10837 suggests that the lib and bin artifacts from the exact same line can actually have different dependencies as a result of per-target feature unification, which is something that we'd want to reflect in cargo tree's output.)

[benwis]

It looks like I am also experiencing this. With resolver=1 it gets past the panic, but tries to build all my optional dependencies. With the default resolver it panics with

❱ cargo -Zbindeps run
thread 'main' panicked at src/cargo/core/resolver/features.rs:322:14:
activated_features for invalid package: features did not find PackageId { name: "libc", version: "0.2.147", source: "registry `crates-io`" } ArtifactDep(CompileTarget { name: "wasm32-unknown-unknown" })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

As others have stated, it seems to happen when specifying a wasm32-unknown-unknown target like this in both a rust script and a regular lib crate

//! cargo
//! [dependencies]
//! benwis_leptos = { artifact = "cdylib", path="../",features=["hydrate"], target = "wasm32-unknown-unknown" }
//! 

I'll have to go back to std::process::Command for now.

[bstrie]

@benwis If this is the only problem that you're having, then you can also consider using sfackler's standalone cargo-tree tool, as linked above.

[benwis]

Unless I'm misunderstanding, this also breaks normal compilation with cargo build, and I'm not sure replacing cargo tree would fix ti

…

On Mon, Aug 7, 2023, at 9:39 AM, bstrie wrote: @benwis <https://github.com/benwis> If this is the only problem that you're having, then you can also consider using sfackler's standalone cargo-tree tool, as linked above. — Reply to this email directly, view it on GitHub <#10593 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVBTCI3YTPQDX4EMC7DKU3XUEK47ANCNFSM5UDISIAQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[hawkw]

I'm also hitting an issue similar to what @benwis described: the panic occurs any time I attempt to build a crate which has a bindep with a different target. The panic occurs with cargo build, not cargo tree.

Using resolver = "1" isn't an option, as the bindep crate is #![no_std] and has shared dependencies with the crate that depends on the bindep, so resolver = "1"'s feature unification results in those dependencies' std feature flags, breaking my build when building the bindep crate.

I'm happy to provide an additional repro, but I believe @bstrie's reproduction looks like it ought to be sufficient to reproduce the issue with cargo build, as well as with cargo tree?

[bstrie]

@benwis When I first filed this issue the bug only exhibited during cargo tree, not cargo build (at least, as far as I remember... I've filed plenty of bindeps issues that related to cargo build, so I feel like I would have mentioned that before mentioning cargo tree). If cargo build causes this same error to happen, that sounds like something must have changed since I filed this issue. @weihanglo has been doing lots of work resolving many of the other bindeps issues that I've filed, perhaps they can chime in with whether or not any recent changes spring to mind?

[hawkw]

@bstrie As an update, I've found that the issue I observed with cargo build actually only occurs when the bindep crate is in a separate workspace from the crate that has a bindep on it. When both crates are within the same workspace, cargo build completes without error.

[rvolosatovs]

@hawkw just in case it helps, we've moved away from (unstable) bindeps feature and now build our Wasm binaries in a build.rs script largely replicating the same functionality. It only works for crates available locally.

• build.rs: https://github.com/wasmCloud/wasmCloud/blob/7a84469dae07cd31185dbb0ad6cfd0af02d0e3a3/tests/actors/build.rs
• src/lib.rs: e.g. https://github.com/wasmCloud/wasmCloud/blob/7a84469dae07cd31185dbb0ad6cfd0af02d0e3a3/tests/actors/src/lib.rs#L3-L4

This way we are able to build Wasm binaries as part of the normal cargo build/cargo test, while staying on stable Rust and not breaking things like cargo tree, which, in turn, means also not breaking Dependabot (which is otherwise broken in most cases when using bindeps feature).
Do note that the approach outlined above require the crates-to-be-built to be outside the workspace building it, because otherwise an attempt to build those would cause a deadlock in cargo (due to cargo on-disk per-workspace lock)

[weihanglo]

Sorry I haven't got time taking another deep look for -Zbindeps recently. Things continue popping up and focus were shifted.

#12358 is the issue that cargo build panics. It may or may not share the same code path.

[elchukc]

However, I found another panic in cargo tree when it meets all the followings conditions at once:

* a dependency of an artifact dependency is platform specified

* artfiact dep itself is `{ target = <specified> }` with the same platform as its own dependency

* the platform is not activated.

This happens because baz is not included in the package_map that is passed when building the graph. Instead it is a None value, because the cell for baz has not been initialized. This is why running cargo tree -Z bindeps --target all doesn't trigger this bug.

Click to view the code

// inside src/cargo/ops/tree/mod.rs

let package_map: HashMap<PackageId, &Package> = ws_resolve
        .pkg_set
        .packages()    // .packages() filters out packages of None value
        .map(|pkg| (pkg.package_id(), pkg))
        .collect();


// inside graph::add_pkg

deps.sort_unstable_by_key(|dep| dep.name_in_toml());
let dep_pkg = graph.package_map[&dep_id];      // stderr no entry found for key

for dep in deps {...}

[weihanglo]

@elchukc so #12554 is very likely a fix for this.