Activating features for tests/benchmarks

[ctz]

Given a library crate, I want to have optional features which are available to integration-level tests and examples so that 'cargo test' tests them.

Here's what I have:

[package]
name = "rustls"
version = "0.1.0"
(etc)

[features]
default = []
clientauth = []

[dependencies]
untrusted = "0.2.0"
(etc)

[dev-dependencies]
env_logger = "0.3.3"
(etc)

Here's what I've tried so far. Attempt one: add the crate into dev-dependencies with the feature, ie:

[dev-dependencies.rustls]
path = "."
version = "0.1.0"
features = ["clientauth"]

I kind of expected that to work, but it panics:

jbp@debian:~/rustls$ RUST_BACKTRACE=1 cargo test
thread '<main>' panicked at 'assertion failed: my_dependencies.insert(dep.clone())', src/cargo/util/dependency_queue.rs:86
stack backtrace:
   1:     0x558d17c3a2a0 - std::sys::backtrace::tracing::imp::write::h9fb600083204ae7f
   2:     0x558d17c4383b - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hca543c34f11229ac
   3:     0x558d17c434c3 - std::panicking::default_hook::hc2c969e7453d080c
   4:     0x558d17c28048 - std::panicking::rust_panic_with_hook::hfe203e3083c2b544
   5:     0x558d17701fef - std::panicking::begin_panic::h4ebf9fe884b2415f
   6:     0x558d17802d27 - cargo::ops::cargo_rustc::job_queue::JobQueue::enqueue::h98b20a33b8744ebe
   7:     0x558d177f4faf - cargo::ops::cargo_rustc::compile::h93336dd21aba29d0
   8:     0x558d1777a513 - cargo::ops::cargo_rustc::compile_targets::h7223dffae01d6b9d
   9:     0x558d1776eec9 - cargo::ops::cargo_compile::compile_pkg::h48656ffa9b1541f3
  10:     0x558d1776bd09 - cargo::ops::cargo_compile::compile::h1b43b20047c53d10
  11:     0x558d1785fb0d - cargo::ops::cargo_test::compile_tests::h028a22d3131e0d65
  12:     0x558d1785f379 - cargo::ops::cargo_test::run_tests::h0ad98c54a6f40c9d
  13:     0x558d176de256 - cargo::call_main_without_stdin::hd484f08b17c419d1
  14:     0x558d1768afe7 - cargo::execute::hab7accd6bf9b5c64
  15:     0x558d17683f8b - cargo::call_main_without_stdin::hf099bd36acb849a3
  16:     0x558d17680da1 - cargo::main::h2b219a79f378c1ef
  17:     0x558d17c430d8 - std::panicking::try::call::hc5e1f5b484ec7f0e
  18:     0x558d17c4e0eb - __rust_try
  19:     0x558d17c4e08e - __rust_maybe_catch_panic
  20:     0x558d17c42b03 - std::rt::lang_start::h61f4934e780b4dfc
  21:     0x7f79ea9bf86f - __libc_start_main
  22:     0x558d17680728 - <unknown>

(version: "cargo 0.11.0-nightly (259324c 2016-05-20)")

Attempt two: see if features.* works like profile.*:

[features.test]
default = ["clientauth"]

alas

$ cargo test
error: failed to parse manifest at `/home/jbp/rustls/Cargo.toml`

Caused by:
  expected a value of type `array`, but found a value of type `table` for the key `features.test`

Is this possible?

See also #4663

[alexcrichton]

Unfortunately, no, it's not possible for cargo test to automatically enable features currently. The panic though definitely seems bad!

[jjpe]

I just ran into this issue because I'm currently using a feature to decide whether or not to do some rather verbose tracing. I'd like to be able to set this during testing so that when tests fail I can look at the trace.
The reason I made it a feature is that the code itself is in a library, but at the same time it seems more appropriate to have the consuming crate decide whether or not to perform the trace.

To be clear, an actual consuming crate has no problem setting the feature and getting corresponding results. It is just for testing the library crate itself that it doesn't seem possible at the moment.

Update: There is a workaround I just found:
If the feature is called foo, then this will test with the feature activated:

cargo test --features "foo"

The only thing is that this is activated manually, and being able to specify it in the library's Cargo.toml would activate the feature by default, which is desirable in my use case.

[sanmai-NL]

@alexcrichton:

As of yet, the Cargo panic is gone.

  cargo --version
cargo 0.19.0-nightly (fa7584c14 2017-04-26)

Excerpt from Cargo.toml

[features]
default = [] # "clippy"

[profile.test]
default = ["clippy"]

The output is now as follows:

cargo test --color=always --package my_app --bin my_app test_1 -- --nocapture

warning: unused manifest key: profile.test.default
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/structure_web_server-3dc5f86e9b4fb259

running 1 test
test tests::test_1 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

You wrote this isn't possible. It appears there are no immediate objections to adding such a feature. Could you explain how fundamental the limitation is? And, if possible, sketch a solution? Just so a PR becomes more likely to happen.

[alexcrichton]

Ah I believe this was fixed in #3921 as #3902 was a dupe of this issue. I guess I was wrong before!

[sanmai-NL]

@alexcrichton: the crashes have been resolved yes, but the original feature request still stands AFAIK. Please reopen.

[alexcrichton]

Oh sorry, missed that aspect.

[sunjay]

Is there any way to add features per profile now? That is, if I want to enable certain features during tests, can I do that yet?

[alexcrichton]

@sunjay unfortunately no, you'll still need to pass --features on the command line

[sunjay]

Is this change something that would require an RFC or could it just be implemented? It sounds like being able to specify features in the profile configuration is something that could benefit a lot of people.

[alexcrichton]

@sunjay I think due to the subtle possible ramifications of auto-enabling features I'd personally want to see an RFC, but it may be worth canvassing others from @rust-lang/cargo first as well

[sunjay]

I would love to work on an RFC for this. Very interested to hear what the cargo team thinks.

Right now, I'm having to constantly remember to pass features in when I run cargo test. I would really prefer to not have to do that. It's an extra point of friction for anyone who uses my code.

Though it would be great to even just be able to configure features in tests, I would really like to see cargo support a features/default features option in each of the [profile.*] sections. This would make it so you could configure default features for your release, debug and benchmarking profiles too. My RFC would be proposing something like that.

[matklad]

I think we have an RFC by @withoutboats for this already: rust-lang/rfcs#1956, which is postponed because of profiles.

[sunjay]

@matklad Ah yup that's exactly what I would have proposed. Thanks for looking that up! I read the discussion and it seems like that's pending on some work with how multiple profiles are applied together (e.g. test --release). It seems that the current behavior is more complicated than what it looks like at first glance.

Is there any way to do this in a backwards compatible way? It would be great if there was some way to just add another configuration option that is in effect the same as the command line argument while keeping everything the same as it is now. Would that make it too hard to change to something better (like profiles) later?

The main issue I'd like to solve is that right now, introducing features into your codebase makes it so that you have to pass features in as command line arguments every time you run a certain commands. If these aren't default features that you need for all builds, it's not possible to simply configure the ones you want in Cargo.toml. This makes it harder for people new to your codebase or to cargo set things up. It is very repetitive and easy to forget. Is there an easier way to just address that issue?

[brson]

I've also run into the need to have features automatically enabled in test builds. In my case I want the python bindings "extension-module" feature on for production builds, but off for test builds so that I can link directly to python during testing.

[Object905]

Also it's very useful for #![no_std] ergonomics. Since you need to test with libstd linked, otherwise you get
error: no #[default_lib_allocator] found but one is required; is libstd not linked?. So you constantly running cargo test --features std

[ebfull]

Note that you can do this with required-features if you specify your test targets explicitly, aka:

[[test]]
name = "whatever"
path = "tests/whatever.rs"
required-features = ["mycoolfeature"]

[RReverser]

@ebfull That only skips building/running tests if feature is not enabled instead of force-enabling the feature during build.

[synek317]

Some ugly workaround, at least for some cases, is to extend all your #[cfg(feature = ...)] with test (e.g. #[cfg(any(test, feature = "..."))] and copy all optional dependencies in [dev-dependencies] and remove optional = true.

It works nice for me but for sure there are cases where it is not enough.

[nixpulvis]

Yea I feel like an RFC here would make sense if one doesn't already exist. I'm using rust again, and finding myself really wanting something like cargo test --feature-matrix for example.

[epage]

Most of the above details aren't relevant: what matters is just that we have a custom test, not using the standard harness, which requires some feature, and which we want to always be run.

Actually it is relevant, particularly the combination of (1) tests that need access to private APIs or behaviors and (2) not being able to write these as unit tests (with enough context to understand the "why")

This kind of information is the core of whats needed to decide how important a use case is and how to handle it.

Is the logged information access through a feature-gated API or is it exposed through stdio / GUI?

[epage]

@ilyagr another potential workaround is to have the test bins auto-discovered b
but to ensure they aren't included in the published artifact (.crate) via package.include or package.exclude.

[ilyagr]

@epage

another potential workaround is to have the test bins auto-discovered b but to ensure they aren't included in the published artifact (.crate) via package.include or package.exclude.

Could you elaborate? I'm far from an expert on Cargo.toml. I'm not sure what "auto-discovered" means.

Naively, I was wondering whether there could be a dev-dependency on a separate crate for the fake binaries, but I'm not sure whether the tests will be able to find the binaries from another crate.

[martinling]

Actually it is relevant, particularly the combination of (1) tests that need access to private APIs or behaviors and (2) not being able to write these as unit tests (with enough context to understand the "why")

I take your point; I was just trying to pre-emptively avoid the discussion getting dragged into the weeds of why GTK/macOS has that particular annoying limitation, when there are any number of other reasons why someone might want to have tests that don't use the standard test harness.

Is the logged information access through a feature-gated API or is it exposed through stdio / GUI?

The test creates a UI in the same way the normal program would, but the feature causes the UI to be created off-screen, with logging code. The test then uses feature-gated APIs on the UI to set where to direct the logging, and also to inject a sequence of pre-recorded user actions into the UI. The log output is then compared against a reference file, and if the two match, the test passes.

As I understand it, the reason that #[cfg(test)] doesn't work for us is that the UI code is in a library target, and that library target is built without cfg(test) set, because a standalone test is presumed to be an integration test that only uses public APIs. In fact, we don't have any truly public API at all - we don't publish a library crate. The library target only exists so that we can share UI code between the main program and the replay test.

[martinling]

For our particular use case, if there were a way to tag a unit test as "must be run in the main thread of a process", and have Cargo honour that somehow (either by executing from its main thread or spawning a new process), then that would be a better solution for us. Originally test_replay was just a unit test in main.rs; it's only because that wouldn't work on macOS that I had to break up main.rs into lib.rs, ui.rs, and test_replay.rs so that it could run as a standalone test.

[epage]

@ilyagr

another potential workaround is to have the test bins auto-discovered b but to ensure they aren't included in the published artifact (.crate) via package.include or package.exclude.

Could you elaborate? I'm far from an expert on Cargo.toml. I'm not sure what "auto-discovered" means.

By default (controlled by package.autobins), cargo will look for main.rss in specific locations and automatically create [[bin]] tables for them. This auto-discovery happens on every load. If you specify that some main.rs should not be included in the .crate file, then when a user does cargo install jj-cli, cargo won't find those additional main.rs files and won't install them.

Naively, I was wondering whether there could be a dev-dependency on a separate crate for the fake binaries, but I'm not sure whether the tests will be able to find the binaries from another crate.

That is what the artifact dependencies feature is about that I linked to. Its not just about depending on them but needing to tell cargo to build the [[bin]]s and not just the [lib].

[epage]

@martinling The reason for that question I asked was to understand whether there would be problems by the fact that cargo test could only ever run your code in a non-production mode.

btw some other potential workarounds for your use case

• Put most of your code in a lib and have another package depend on it, activating the feature that is needed, and have the tests live there.
• Adding accessibility support (which would likely be a lot of work) as that tends to have enough overlap for testing

[martinling]

Put most of your code in a lib and have another package depend on it, activating the feature that is needed, and have the tests live there.

I have just verified that yes, it is possible to make cargo test work as desired by further splitting what was originally a simple, single-binary project into three different packages, with four Cargo.toml files. But what was the point of this suggestion?

Yes, it is possible to work around the lack of the feature that people are asking for in this issue. That doesn't make the feature request invalid. The complexity of the workarounds you're suggesting illustrate exactly why it would be helpful.

Adding accessibility support (which would likely be a lot of work) as that tends to have enough overlap for testing

Even with a lot of work, I do not believe that it would be either practical or performant for us to implement the same test through the GTK accessibility APIs. We're not just clicking on some buttons and looking for dialog messages. We're dealing with a list model with what may be millions of rows that the UI loads from on demand. And the test also needs to control the inflow of packets into the decoder engine so that it can verify how descriptions of items are updated by new packets. There's no way to do all that through the normal application UI, we have to hook into it specially, and that requires gating some code.

But even if we could, the fact that this may a suitable solution for some GUI applications doesn't invalidate the use cases where it isn't.

Why is there so much desire to close this? I don't really see anyone asking for anything complicated here. A solution could be as simple as a Cargo.toml key that makes cargo test behave as if you'd typed cargo test --features=foo,bar. As far as I can see, it doesn't need to affect how feature unification works, or any other Cargo internals. It's just about setting some defaults per-profile on the frontend side.

[epage]

I have just verified that yes, it is possible to make cargo test work as desired by further splitting what was originally a simple, single-binary project into three different packages, with four Cargo.toml files. But what was the point of this suggestion?

Yes, it is possible to work around the lack of the feature that people are asking for in this issue. That doesn't make the feature request invalid. The complexity of the workarounds you're suggesting illustrate exactly why it would be helpful.

I would assume you could get away with just two packages and two Cargo.toml files (workspace + lib + main bin, test bin)

As for the point, it was to offer another alternative solution for people running into it. It might work for some people; it might not work for others. It does offer the benefit of having the minimal affect on feature activations which is likely needed in some other cases.

I do feel it important to call out that what is important in an issue is the use cases. We shouldn't get attached to any given solution.

Why is there so much desire to close this? I don't really see anyone asking for anything complicated here. A solution could be as simple as a Cargo.toml key that makes cargo test behave as if you'd typed cargo test --features=foo,bar. As far as I can see, it doesn't need to affect how feature unification works, or any other Cargo internals. It's just about setting some defaults per-profile on the frontend side.

I think there might be a misunderstanding. None of this is "arguing back to force through a closure". I am trying to understand use cases and explore the problem space. There are valid workflow problems people are having and they weren't all communicated earlier. The question is whether this is still the right solution. That, I'm still not sure on.

And yes, we can add a lot of little features like this but

• I've found the more features you have, the less value you get out of them because you overwhelm users and they can't find what they need to solve problems. Its best to look for patterns in case you can find wider ways of solving problems. There might even be times when its appropriate to say "this is too narrow of case for built-in support so long as there are valid workarounds"
• We need to consider the whole workflow and how the feature is used to ensure we are helping drive the right behavior, while still allowing the exception cases. Otherwise, a "simple" issue like this could turn into 10 more issues that want to extend whatever solution we come up with in 100 different ways because it doesn't quite work for them.

[martinling]

I think there might be a misunderstanding. None of this is "arguing back to force through a closure". I am trying to understand use cases and explore the problem space.

That intent would have been clearer had you not first entered the discussion with a proposal to close, and then proceeded to respond to use cases by listing workarounds.

But anyway. In our specific case, as I already said, I don't think this feature is actually the best solution to our problem. What we have is really a unit test, we're just lacking a way to specify that this unit test must be executed on the main thread of a program. I guess I should go raise that as a feature request for libtest.

[weihanglo]

A solution could be as simple as a Cargo.toml key that makes cargo test behave as if you'd typed cargo test --features=foo,bar. As far as I can see, it doesn't need to affect how feature unification works, or any other Cargo internals.

Are you proposing something like #4942? It doesn't seem too hard in terms of supporting that. However but I've outlined some unresolved questions around that needing to be addressed.

If we look into RFC 3374 a bit deeper, there is a paragraph calling out some difficulties and open questions on the implementation side. I know it looks not too hard but things are under the surface 😞.

Anyway, thank you for bringing more info on the table.