From 81c3b3d92cacfbd04fe5c54b601cdcfab4bbd9c1 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 25 Jan 2018 15:39:45 +0530 Subject: [PATCH 01/22] eRFC: Post Build Contexts --- text/0000-erfc-post-build-contexts.md | 358 ++++++++++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 text/0000-erfc-post-build-contexts.md diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md new file mode 100644 index 00000000000..06ff5348999 --- /dev/null +++ b/text/0000-erfc-post-build-contexts.md @@ -0,0 +1,358 @@ +- Feature Name: post_build_contexts +- Start Date: 2018-01-25 +- RFC PR: (leave this empty) +- Rust Issue: (leave this empty) + +# Summary +[summary]: #summary + +This is an *experimental RFC* for adding the ability to integrate custom test/bench/etc frameworks ("post-build frameworks") in Rust. + +# Motivation +[motivation]: #motivation + +Currently, Rust lets you write unit tests with a `#[test]` attribute. We also have an unstable `#[bench]` attribute which lets one write benchmarks. + +In general it's not easy to use your own testing strategy. Implementing something that can work +within a `#[test]` attribute is fine (`quickcheck` does this with a macro), but changing the overall +strategy is hard. For example, `quickcheck` would work even better if it could be done as: + +```rust +#[quickcheck] +fn test(input1: u8, input2: &str) { + // ... +} +``` + +If you're trying to do something other than testing, you're out of luck -- only tests, benches, and examples +get the integration from `cargo` for building auxiliary binaries the correct way. [cargo-fuzz] has to +work around this by creating a special fuzzing crate that's hooked up the right way, and operating inside +of that. Ideally, one would be able to just write fuzz targets under `fuzz/`. + +[Compiletest] (rustc's test framework) would be another kind of thing that would be nice to +implement this way. Currently it compiles the test cases by manually running `rustc`, but it has the +same problem as cargo-fuzz where getting these flags right is hard. This too could be implemented as +a custom test framework. + +A profiling framework may want to use this mode to instrument the binary in a certain way. We +can already do this via proc macros, but having it hook through `cargo test` would be neat. + +Overall, it would be good to have a generic framework for post-build steps that can support use +cases like `#[test]` (both the built-in one and quickcheck), `#[bench]` (both built in and custom +ones like [criterion]), `examples`, and things like fuzzing. While we may not necessarily rewrite +the built in test/bench/example infra in terms of the new framework, it should be possible to do so. + +The main two problems that we need to solve are: + + - Having a nice API for generating custom post-build binaries + - Having good `cargo` integration so that custom tests are at the same level of integration as regular tests as far as build processes are concerned + + [cargo-fuzz]: https://github.com/rust-fuzz/cargo-fuzz + [criterion]: https://github.com/japaric/criterion.rs + [Compiletest]: http://github.com/laumann/compiletest-rs + +# Detailed proposal +[detailed-proposal]: #detailed-proposal + +(As an eRFC I'm merging the "guide-level/reference-level" split for now; when we have more concrete +ideas we can figure out how to frame it and then the split will make more sense) + +## Procedural macro for a new post-build context + +A custom post-build context is essentially a whole-crate procedural +macro that is evaluated after all other macros in the target crate have +been evaluated. It is passed the `TokenStream` for every element in the +target crate that has a set of attributes the post-build context has +registered interest in. Essentially: + +```rust +extern crate proc_macro; +use proc_macro::TokenStream; + +// attributes() is optional +#[post_build_context(test, attributes(foo, bar))] +pub fn like_todays_test(items: &[AnnotatedItem]) -> TokenStream { + // ... +} +``` + +where + +```rust +struct AnnotatedItem + tokens: TokenStream, + span: Span, + attributes: TokenStream, + path: SomeTypeThatRepresentsPathToItem +} +``` + +`items` here contains an `AnnotatedItem` for every element in the +target crate that has one of the attributes declared in `attributes` +along with attributes sharing the name of the context (`test`, here). + +A post-build context could declare that it reacts to multiple different +attributes, in which case it would get all items with any of the +listed attributes. These items be modules, functions, structs, +statics, or whatever else the post-build context wants to support. Note +that the post-build context function can only see all the annotated +items, not modify them; modification would have to happen with regular +procedural macros The returned `TokenStream` will become the `main()` +when this post-build context is used. + +Because this procedural macro is only loaded when it is used as the +post-build context, the `#[test]` annotation should probably be kept +behind `#[cfg(test)]` so that you don't get unknown attribute warnings +whilst loading. (We could change this by asking attributes to be +registered in Cargo.toml, but we don't find this necessary) + +## Cargo integration + +Alternative post-build contexts need to integrate with cargo. +In particular, when crate `a` uses a crate `b` which provides an +post-build context, `a` needs to be able to specify when `b`'s post-build +context should be used. Furthermore, cargo needs to understand that when +`b`'s post-build context is used, `b`'s dependencies must also be linked. +Note that `b` could potentially provide multiple post-build contexts --- +these are named according to the name of their `#[post_build_context]` +function. + +Crates which define a post-build context must have an `post-build-context = true` +key. + +For crates that wish to *use* a custom post-build context, they do so by +defining a new post-build context under a new `post-build` section in +their `Cargo.toml`: + +```toml +[post-build.context.fuzz] +provider = { rust-fuzz = "1.0" } +folder = "fuzz/" +specify-single-target = true # false by default +``` + +This defines a post-build context named `fuzz`, which uses the +implementation provided by the `rust-fuzz` crate. When run, it will be +applies to all files in the `fuzz` directory. `specify-single-target` +addresses whether it must be run with a single target. If true, you will +be forced to run `cargo post-build foobar --test foo`. This is useful for cases +like `cargo-fuzz` where running tests on everything isn't possible. + +By default, the following contexts are defined: + +```toml +[post-build.context.test] +provider = { test = "1.0", context = "test" } +folder = "tests/" + +[post-build.context.bench] +provider = { test = "1.0", context = "bench" } +folder = ["benchmarks/", "morebenchmarks/"] +``` + +These can be overridden by a crate's `Cargo.toml`. The `context` +property is used to disambiguate when a single crate has multiple +functions tagged `#[post_build_context]` (if we were using the example +post-build provider further up, we'd give `like_todays_test` here). +`test` here is `libtest`, though note that it could be maintained +out-of-tree, and shipped with rustup. + +To invoke a particular post-build context, a user invokes `cargo post-build +`. `cargo test` and `cargo bench` are aliases for `cargo +post-build test` and `cargo post-build bench` respectively. Any additional +arguments are passed to the post-build context binary. By convention, the +first position argument should allow filtering which +test/benchmarks/etc. are run. + + +By default, the crate has an implicit "test", "bench", and "example" context that use the default libtest stuff. +(example is a no-op context that just runs stuff). However, declaring a context with the name `test` +will replace the existing `test` context. In case you wish to supplement the context, use a different +name. + +By default, `cargo test` will run doctests and the `test` and `examples` context. This can be customized: + +```toml +[post-build.set.test] +contexts = [test, quickcheck, examples] +``` + +This means that `cargo test` will, aside from doctests, run `cargo post-build test`, `cargo post-build quickcheck`, +and `cargo post-build examples` (and similar stuff for `cargo bench`). It is not possible to make `cargo test` +_not_ run doctests. + +There are currently only two custom post-build sets (test and bench). + +Custom test targets can be declared via `[[post-build.target]]` + +```toml +[[post-build.target]] +context = fuzz +path = "foo.rs" +name = "foo" +``` + +`[[test]]` is an alias for `[[post-build.target]] context = test` (same goes for `[[bench]]` and `[[example]]`). + + +The generated test binary should be able to take one identifier argument, used for narrowing down what tests to run. +I.e. `cargo test --kind quickcheck my_test_fn` will build the test(s) and call them with `./testbinary my_test_fn`. +Typically, this argument is used to filter tests further; test harnesses should try to use it for the same purpose. + + +## To be designed + +This contains things which we should attempt to solve in the course of this experiment, for which this eRFC +does not currently provide a concrete proposal. + +### Standardizing the output + +We should probably provide a crate with useful output formatters and stuff so that if test harnesses desire, they can +use the same output formatting as a regular test. This also provides a centralized location to standardize things +like json output and whatnot. + +@killercup is working on a proposal for this which I will try to work in. + +### Configuration + +Currently we have `cfg(test)` and `cfg(bench)`. Should `cfg(test)` be applied to all? Should `cfg(nameofharness)` +be used instead? Ideally we'd have a way when declaring a framework to declare what cfgs it should be built with. + +# Drawbacks +[drawbacks]: #drawbacks + + - This adds more sections to `Cargo.toml`. + - This complicates the execution path for cargo, in that it now needs + to know about post-build contexts and sets. + - Flags and command-line parameters for test and bench will now vary + between post-build contexts, which may confuse users as they move + between crates. + +# Rationale and alternatives +[alternatives]: #alternatives + +We should either do this or stabilize the existing bencher. + +## Alternative procedural macro + +An alternative proposal was to expose an extremely general whole-crate proc macro: + +```rust +#[post_build_context(test, attributes(foo, bar))] +pub fn context(crate: TokenStream) -> TokenStream { + // ... +} +``` + +and then we can maintain a helper crate, out of tree, that uses `syn` to provide a nicer +API, perhaps something like: + +```rust +fn clean_entry_point(tree: syn::ItemMod) -> syn::ItemMod; + +trait TestCollector { + fn fold_function(&mut self, path: syn::Path, func: syn::ItemFn) -> syn::ItemFn; +} + +fn collect_tests(collector: &mut T, tree: syn::ItemMod) -> ItemMod; +``` + +This lets us continue to develop things outside of tree without perma-stabilizing an API; +and it also lets us provide a friendlier API via the helper crate. + +It also lets crates like `cargo-fuzz` introduce things like a `#![no_main]` attribute or do +other antics. + +Finally, it handles the "profiling framework" case as mentioned in the motivation. On the other hand, +these tools usually operate at a differeny layer of abstraction so it might not be necessary. + +A major drawback of this proposal is that it is very general, and perhaps too powerful. We're currently using the +more focused API in the eRFC, and may switch to this during experimentation if a pressing need crops up. + +# Unresolved questions +[unresolved]: #unresolved-questions + +These are mostly intended to be resolved during the experimental +feature. + +## Integration with doctests + +Documentation tests are somewhat special, in that they cannot easily be +expressed as `TokenStream` manipulations. In the first instance, the +right thing to do is probably to have an implicitly defined execution +context called `doctest` which is included in the execution context set +`test` by default. + +Another argument for punting on doctests is that they are intended to +demonstrate code that the user of a library would write. They're there +to document *how* something should be used, and it then makes somewhat +less sense to have different "ways" of running them. + +## Translating existing cargo test flags + +Today, `cargo test` takes a number of flags such as `--lib`, `--test +foo`, and `--doc`. As it would be a breaking change to change these, +cargo should recognize them and map to the appropriate execution +contexts. + +Currently, `cargo test` lets you pick a single testing target via `--test`, +and `cargo bench` via `--bench`. We'll need to create an agnostic flag +for `cargo post-build` (we cannot use `--target` because it is already used for +the target architecture, and `--test` is too specific for tests). `--post-build-target` +is one rather verbose suggestion. + +## Standardizing the output + +We should probably provide a crate with useful output formatters and +stuff so that if test harnesses desire, they can use the same output +formatting as a regular test. This also provides a centralized location +to standardize things like json output and whatnot. + +## Configuration + +Currently we have `cfg(test)` and `cfg(bench)`. Should `cfg(test)` be +applied to all? Should `cfg(post_build_context)` be used instead? +Ideally we'd have a way when declaring a post-build context to declare +what cfgs it should be built with. + +## Runtime dependencies and flags + +The generated harness itself may have some dependencies. Currently there's +no way for the post-build context to specify this. One proposal is for the crate +to specify _runtime_ dependencies of the post-build context via: + +```toml +[post-build.dependencies] +libfuzzer-sys = ... +``` + +If a crate is currently running this post-build context, its dev-dependencies +will be semver-merged with the post-build-context.dependencies. + +However, this may not be strictly necessary. Custom derives have +a similar problem and they solve it by just asking users to import the correct +crate and keep it in their dev-dependencies. + +## Naming + +The general syntax and toml stuff should be approximately settled on before this eRFC merges, but +iterated on later. Naming the feature is hard, some candidates are: + + - test framework + - post-build context + - execution context + +None of these are particularly great, ideas would be nice. + +## Other questions + + - Should a post-build context be able to declare "defaults" for what + folders and post-build sets it should be added to? This might save + users from some boilerplate in a large number of situations. + - Should we be shipping a bencher by default at all (i.e., in libtest)? + Could we instead default `cargo bench` to a `rust-lang-nursery` + crate? + - `specify-single-target = true` probably should be specified by the execution context itself, + not the consumer. It's also questionable if it's necessary -- cargo-fuzz is going to need + a wrapper script anyway, so it's fine if the CLI isn't as ergonomic for that use case. + \ No newline at end of file From 53ba19d267244275c98a871ce894411db530bca7 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 1 Feb 2018 15:28:00 +0530 Subject: [PATCH 02/22] Add alternate proposal with minimal impact on the compiler --- text/0000-erfc-post-build-contexts.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 06ff5348999..26cde9195a1 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -269,6 +269,29 @@ these tools usually operate at a differeny layer of abstraction so it might not A major drawback of this proposal is that it is very general, and perhaps too powerful. We're currently using the more focused API in the eRFC, and may switch to this during experimentation if a pressing need crops up. +## Alternative procedural macro with minimal compiler changes + +The above proposal can be made even more general, minimizing the impact on the compiler. + +This assumes that `#![foo]` ("inner attribute") macros work on modules and on crates. + +The idea is that the compiler defines no new proc macro surface, and instead simply exposes +a `--attribute` flag. This flag, like `-Zextra-plugins`, lets you attach a proc macro attribute +to the whole crate before compiling. (This flag actually generalizes a bunch of flags that the +compiler already has) + +Test crates are now simply proc macro attributes: + +```rust +#[proc_macro_attr(attributes(test, foo, bar))] +pub fn harness(crate: TokenStream) -> TokenStream { + // ... +} +``` + +The cargo functionality will basically compile the file with the right dependencies +and `--attribute=your_crate::harness`. + # Unresolved questions [unresolved]: #unresolved-questions From 55860d5b111ed9f2caa8d841762ad665229709a7 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 1 Feb 2018 16:33:00 +0530 Subject: [PATCH 03/22] Expand all 'other questions' into their own section --- text/0000-erfc-post-build-contexts.md | 56 +++++++++++++++++++++------ 1 file changed, 44 insertions(+), 12 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 26cde9195a1..3649d4b67b6 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -296,7 +296,9 @@ and `--attribute=your_crate::harness`. [unresolved]: #unresolved-questions These are mostly intended to be resolved during the experimental -feature. +feature. Many of these have strawman proposals -- unlike the rest of this RFC, +these proposals have not been discussed as thoroughly. If folks feel like +there's consensus on some of these we can move them into the main RFC. ## Integration with doctests @@ -367,15 +369,45 @@ iterated on later. Naming the feature is hard, some candidates are: None of these are particularly great, ideas would be nice. -## Other questions - - - Should a post-build context be able to declare "defaults" for what - folders and post-build sets it should be added to? This might save - users from some boilerplate in a large number of situations. - - Should we be shipping a bencher by default at all (i.e., in libtest)? - Could we instead default `cargo bench` to a `rust-lang-nursery` - crate? - - `specify-single-target = true` probably should be specified by the execution context itself, - not the consumer. It's also questionable if it's necessary -- cargo-fuzz is going to need - a wrapper script anyway, so it's fine if the CLI isn't as ergonomic for that use case. +## Default folders and sets + +Should a post-build context be able to declare "defaults" for what folders and post-build sets it +should be added to? This might save users from some boilerplate in a large number of situations. + +This could be done in the Cargo.toml as: + +```toml +[post-build.defaults] +folder = "tests/" +set = "test" # will automatically be added to the `test` set +``` + +This is useful if a crate wishes to standardize things. + +## Bencher + +Should we be shipping a bencher by default at all (i.e., in libtest)? Could we instead default +`cargo bench` to a `rust-lang-nursery` `bench` crate? + +If this RFC lands and [RFC 2287] is rejected, we should probably try to stabilize +`test::black_box` in some form (maybe `mem::black_box` and `mem::clobber` as detailed +in [this amendment]). + + + + [RFC 2287]: https://github.com/rust-lang/rfcs/pull/2287 + [this amendment]: https://github.com/Manishearth/rfcs/pull/1 + +## Specify-single-target + +`specify-single-target = true` probably should be specified by the execution context itself, not the +consumer. It's also questionable if it's necessary -- cargo-fuzz is going to need a wrapper script +anyway, so it's fine if the CLI isn't as ergonomic for that use case. + +If we do `post-build.defaults` it would just make sense to include that there. + + + + + \ No newline at end of file From 68695a6aa2873bbb037fd436366b0285fdcfba12 Mon Sep 17 00:00:00 2001 From: Jon Gjengset Date: Thu, 1 Feb 2018 09:34:46 -0500 Subject: [PATCH 04/22] Various changes --- text/0000-erfc-post-build-contexts.md | 198 ++++++++++++-------------- 1 file changed, 89 insertions(+), 109 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 3649d4b67b6..13bb73bf995 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -42,10 +42,13 @@ cases like `#[test]` (both the built-in one and quickcheck), `#[bench]` (both bu ones like [criterion]), `examples`, and things like fuzzing. While we may not necessarily rewrite the built in test/bench/example infra in terms of the new framework, it should be possible to do so. -The main two problems that we need to solve are: +The main two features proposed are: - - Having a nice API for generating custom post-build binaries - - Having good `cargo` integration so that custom tests are at the same level of integration as regular tests as far as build processes are concerned + - An API for crates that generate custom binaries, including + introspection into the target crate. + - A mechanism for `cargo` integration so that custom post-build + contexts are at the same level of integration as `test` or `bench` as + far as build processes are concerned. [cargo-fuzz]: https://github.com/rust-fuzz/cargo-fuzz [criterion]: https://github.com/japaric/criterion.rs @@ -63,15 +66,16 @@ A custom post-build context is essentially a whole-crate procedural macro that is evaluated after all other macros in the target crate have been evaluated. It is passed the `TokenStream` for every element in the target crate that has a set of attributes the post-build context has -registered interest in. Essentially: +registered interest in. For example, to declare a post-build context +called `mytest`: ```rust extern crate proc_macro; use proc_macro::TokenStream; // attributes() is optional -#[post_build_context(test, attributes(foo, bar))] -pub fn like_todays_test(items: &[AnnotatedItem]) -> TokenStream { +#[post_build_context(attributes(foo, bar))] +pub fn mytest(items: &[AnnotatedItem]) -> TokenStream { // ... } ``` @@ -87,9 +91,9 @@ struct AnnotatedItem } ``` -`items` here contains an `AnnotatedItem` for every element in the +`items` here contains an `AnnotatedItem` for every item in the target crate that has one of the attributes declared in `attributes` -along with attributes sharing the name of the context (`test`, here). +along with attributes sharing the name of the context (`mytest`, here). A post-build context could declare that it reacts to multiple different attributes, in which case it would get all items with any of the @@ -97,14 +101,18 @@ listed attributes. These items be modules, functions, structs, statics, or whatever else the post-build context wants to support. Note that the post-build context function can only see all the annotated items, not modify them; modification would have to happen with regular -procedural macros The returned `TokenStream` will become the `main()` -when this post-build context is used. +procedural macros. The returned `TokenStream` must declare the `main()` +that is to become the entry-point for the binary produced when this +post-build context is used. Because this procedural macro is only loaded when it is used as the -post-build context, the `#[test]` annotation should probably be kept -behind `#[cfg(test)]` so that you don't get unknown attribute warnings -whilst loading. (We could change this by asking attributes to be -registered in Cargo.toml, but we don't find this necessary) +post-build context, the `#[mytest]` annotation should probably be kept +behind `#[cfg(mytest)]` (which is automatically set when the `mytest` +context is used) so that you don't get unknown attribute warnings +whilst loading, and to avoid conflicts with other post-build contexts +that may use the same attributes. (We could change this by asking +attributes to be registered in Cargo.toml, but we don't find this +necessary) ## Cargo integration @@ -118,87 +126,76 @@ these are named according to the name of their `#[post_build_context]` function. Crates which define a post-build context must have an `post-build-context = true` -key. +key in their `Cargo.toml`. It can also specify -For crates that wish to *use* a custom post-build context, they do so by -defining a new post-build context under a new `post-build` section in -their `Cargo.toml`: +```rust +single-target = true # false by default +``` + +`single-target` indicates that only a single target can be run with this +context at once (some tools, like cargo-fuzz, run forever, and so it +does not make sense to specify multiple targets). + +Crates that wish to *use* a custom post-build context, do so by defining +a new post-build context under a new `build-context` section in their +`Cargo.toml`: ```toml [post-build.context.fuzz] provider = { rust-fuzz = "1.0" } -folder = "fuzz/" -specify-single-target = true # false by default +folders = ["fuzz/"] ``` -This defines a post-build context named `fuzz`, which uses the +This defines an post-build context named `fuzz`, which uses the implementation provided by the `rust-fuzz` crate. When run, it will be -applies to all files in the `fuzz` directory. `specify-single-target` -addresses whether it must be run with a single target. If true, you will -be forced to run `cargo post-build foobar --test foo`. This is useful for cases -like `cargo-fuzz` where running tests on everything isn't possible. - -By default, the following contexts are defined: +applied to all files in the `fuzz` directory. By default, the following +contexts are defined: ```toml [post-build.context.test] provider = { test = "1.0", context = "test" } -folder = "tests/" +folders = ["tests/"] [post-build.context.bench] provider = { test = "1.0", context = "bench" } -folder = ["benchmarks/", "morebenchmarks/"] +folders = ["benchmarks/"] ``` +There's also an `example` context defined that just runs the `main()` of +any files given. + These can be overridden by a crate's `Cargo.toml`. The `context` property is used to disambiguate when a single crate has multiple functions tagged `#[post_build_context]` (if we were using the example -post-build provider further up, we'd give `like_todays_test` here). +post-build context further up as a provider, we'd give `mytest` here). `test` here is `libtest`, though note that it could be maintained out-of-tree, and shipped with rustup. -To invoke a particular post-build context, a user invokes `cargo post-build +To invoke a particular post-build context, a user invokes `cargo context `. `cargo test` and `cargo bench` are aliases for `cargo -post-build test` and `cargo post-build bench` respectively. Any additional +context test` and `cargo context bench` respectively. Any additional arguments are passed to the post-build context binary. By convention, the -first position argument should allow filtering which -test/benchmarks/etc. are run. - - -By default, the crate has an implicit "test", "bench", and "example" context that use the default libtest stuff. -(example is a no-op context that just runs stuff). However, declaring a context with the name `test` -will replace the existing `test` context. In case you wish to supplement the context, use a different -name. +first position argument should allow filtering which targets +(tests/benchmarks/etc.) are run. -By default, `cargo test` will run doctests and the `test` and `examples` context. This can be customized: +To run multiple contexts at once, a crate can declare post-build context +*sets*. One such example is the `test` post-build context set, which +will run doctests and the `test` and `examples` context. This can be +customized: ```toml [post-build.set.test] contexts = [test, quickcheck, examples] ``` -This means that `cargo test` will, aside from doctests, run `cargo post-build test`, `cargo post-build quickcheck`, -and `cargo post-build examples` (and similar stuff for `cargo bench`). It is not possible to make `cargo test` -_not_ run doctests. - -There are currently only two custom post-build sets (test and bench). - -Custom test targets can be declared via `[[post-build.target]]` - -```toml -[[post-build.target]] -context = fuzz -path = "foo.rs" -name = "foo" -``` - -`[[test]]` is an alias for `[[post-build.target]] context = test` (same goes for `[[bench]]` and `[[example]]`). - - -The generated test binary should be able to take one identifier argument, used for narrowing down what tests to run. -I.e. `cargo test --kind quickcheck my_test_fn` will build the test(s) and call them with `./testbinary my_test_fn`. -Typically, this argument is used to filter tests further; test harnesses should try to use it for the same purpose. +This means that `cargo test` will, aside from doctests, run `cargo +context test`, `cargo context quickcheck`, and `cargo context examples` +(and similar stuff for `cargo bench`). It is not possible to make `cargo +test` _not_ run doctests. If both a context and a set exists with a +given name, the set takes precedence. +`[[test]]` and `[[example]]` in a crate's `Cargo.toml` add files to the +`test` and `example` contexts respectively. ## To be designed @@ -213,11 +210,6 @@ like json output and whatnot. @killercup is working on a proposal for this which I will try to work in. -### Configuration - -Currently we have `cfg(test)` and `cfg(bench)`. Should `cfg(test)` be applied to all? Should `cfg(nameofharness)` -be used instead? Ideally we'd have a way when declaring a framework to declare what cfgs it should be built with. - # Drawbacks [drawbacks]: #drawbacks @@ -231,15 +223,18 @@ be used instead? Ideally we'd have a way when declaring a framework to declare w # Rationale and alternatives [alternatives]: #alternatives -We should either do this or stabilize the existing bencher. +We could stabilize `#[bench]` and extend libtest with setup/teardown and +other requested features. This would complicate the in-tree libtest, +introduce a barrier for community contributions, and discourage other +forms of testing or benchmarking. ## Alternative procedural macro An alternative proposal was to expose an extremely general whole-crate proc macro: ```rust -#[post_build_context(test, attributes(foo, bar))] -pub fn context(crate: TokenStream) -> TokenStream { +#[post_build_context(attributes(foo, bar))] +pub fn mytest(crate: TokenStream) -> TokenStream { // ... } ``` @@ -264,7 +259,7 @@ It also lets crates like `cargo-fuzz` introduce things like a `#![no_main]` attr other antics. Finally, it handles the "profiling framework" case as mentioned in the motivation. On the other hand, -these tools usually operate at a differeny layer of abstraction so it might not be necessary. +these tools usually operate at a different layer of abstraction so it might not be necessary. A major drawback of this proposal is that it is very general, and perhaps too powerful. We're currently using the more focused API in the eRFC, and may switch to this during experimentation if a pressing need crops up. @@ -304,9 +299,9 @@ there's consensus on some of these we can move them into the main RFC. Documentation tests are somewhat special, in that they cannot easily be expressed as `TokenStream` manipulations. In the first instance, the -right thing to do is probably to have an implicitly defined execution -context called `doctest` which is included in the execution context set -`test` by default. +right thing to do is probably to have an implicitly defined post-build +context called `doctest` which is included in the post-build context set +`test` by default (as proposed above). Another argument for punting on doctests is that they are intended to demonstrate code that the user of a library would write. They're there @@ -317,12 +312,12 @@ less sense to have different "ways" of running them. Today, `cargo test` takes a number of flags such as `--lib`, `--test foo`, and `--doc`. As it would be a breaking change to change these, -cargo should recognize them and map to the appropriate execution +cargo should recognize them and map to the appropriate post-build contexts. -Currently, `cargo test` lets you pick a single testing target via `--test`, +Furthermore, `cargo test` lets you pick a single testing target via `--test`, and `cargo bench` via `--bench`. We'll need to create an agnostic flag -for `cargo post-build` (we cannot use `--target` because it is already used for +for `cargo context` (we cannot use `--target` because it is already used for the target architecture, and `--test` is too specific for tests). `--post-build-target` is one rather verbose suggestion. @@ -333,30 +328,31 @@ stuff so that if test harnesses desire, they can use the same output formatting as a regular test. This also provides a centralized location to standardize things like json output and whatnot. -## Configuration +## Namespacing -Currently we have `cfg(test)` and `cfg(bench)`. Should `cfg(test)` be -applied to all? Should `cfg(post_build_context)` be used instead? -Ideally we'd have a way when declaring a post-build context to declare -what cfgs it should be built with. +Currently, two post-build contexts can both declare interest in the same +attributes. How do we deal with collisions (e.g., most test crates will +want the attribute `#[test]`). Do we namespace the attributes by the +context name (e.g., `#[mytest::test]`)? Do we require them to be behind +`#[cfg(mytest)]`? ## Runtime dependencies and flags -The generated harness itself may have some dependencies. Currently there's -no way for the post-build context to specify this. One proposal is for the crate -to specify _runtime_ dependencies of the post-build context via: +The code generated by the post-build context may itself have dependencies. +Currently there's no way for the post-build context to specify this. One +proposal is for the crate to specify _runtime_ dependencies of the +post-build context via: ```toml -[post-build.dependencies] +[context-dependencies] libfuzzer-sys = ... ``` -If a crate is currently running this post-build context, its dev-dependencies -will be semver-merged with the post-build-context.dependencies. - -However, this may not be strictly necessary. Custom derives have -a similar problem and they solve it by just asking users to import the correct -crate and keep it in their dev-dependencies. +If a crate is currently running this post-build context, its +dev-dependencies will be semver-merged with the post-build context's +`context-dependencies`. However, this may not be strictly necessary. +Custom derives have a similar problem and they solve it by just asking +users to import the correct crate. ## Naming @@ -378,7 +374,7 @@ This could be done in the Cargo.toml as: ```toml [post-build.defaults] -folder = "tests/" +folders = ["tests/"] set = "test" # will automatically be added to the `test` set ``` @@ -393,21 +389,5 @@ If this RFC lands and [RFC 2287] is rejected, we should probably try to stabiliz `test::black_box` in some form (maybe `mem::black_box` and `mem::clobber` as detailed in [this amendment]). - - [RFC 2287]: https://github.com/rust-lang/rfcs/pull/2287 [this amendment]: https://github.com/Manishearth/rfcs/pull/1 - -## Specify-single-target - -`specify-single-target = true` probably should be specified by the execution context itself, not the -consumer. It's also questionable if it's necessary -- cargo-fuzz is going to need a wrapper script -anyway, so it's fine if the CLI isn't as ergonomic for that use case. - -If we do `post-build.defaults` it would just make sense to include that there. - - - - - - \ No newline at end of file From 152f4ad0777827ba2efbec65082fa423d14e5398 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 1 Feb 2018 20:19:32 +0530 Subject: [PATCH 05/22] Fixups --- text/0000-erfc-post-build-contexts.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 13bb73bf995..f09ffb8a094 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -146,7 +146,7 @@ provider = { rust-fuzz = "1.0" } folders = ["fuzz/"] ``` -This defines an post-build context named `fuzz`, which uses the +This defines a post-build context named `fuzz`, which uses the implementation provided by the `rust-fuzz` crate. When run, it will be applied to all files in the `fuzz` directory. By default, the following contexts are defined: @@ -321,6 +321,9 @@ for `cargo context` (we cannot use `--target` because it is already used for the target architecture, and `--test` is too specific for tests). `--post-build-target` is one rather verbose suggestion. +We also need to settle on a command name, `cargo context` and `cargo post-build` +don't quite capture what's going on. + ## Standardizing the output We should probably provide a crate with useful output formatters and @@ -360,7 +363,7 @@ The general syntax and toml stuff should be approximately settled on before this iterated on later. Naming the feature is hard, some candidates are: - test framework - - post-build context + - build context - execution context None of these are particularly great, ideas would be nice. From fa99ee450e3264a34a1d018764f1b4d92545fba7 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 09:40:15 +0530 Subject: [PATCH 06/22] Summary --- text/0000-erfc-post-build-contexts.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index f09ffb8a094..4af1740803c 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -60,6 +60,13 @@ The main two features proposed are: (As an eRFC I'm merging the "guide-level/reference-level" split for now; when we have more concrete ideas we can figure out how to frame it and then the split will make more sense) +The basic idea is that crates can define post-build contexts, which specify +how to transform collected test functions and construct a `main()` function, +and then crates using these can declare them in their Cargo.toml, which will let +crate developers invoke various test-like post-build steps using the post-build +context. + + ## Procedural macro for a new post-build context A custom post-build context is essentially a whole-crate procedural From 00f439f9c0a3ef1a5d24da3c5204bdc020633822 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 09:47:32 +0530 Subject: [PATCH 07/22] example for main() --- text/0000-erfc-post-build-contexts.md | 32 ++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 4af1740803c..0b430cf4219 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -69,7 +69,7 @@ context. ## Procedural macro for a new post-build context -A custom post-build context is essentially a whole-crate procedural +A custom post-build context is like a whole-crate procedural macro that is evaluated after all other macros in the target crate have been evaluated. It is passed the `TokenStream` for every element in the target crate that has a set of attributes the post-build context has @@ -112,6 +112,36 @@ procedural macros. The returned `TokenStream` must declare the `main()` that is to become the entry-point for the binary produced when this post-build context is used. +So an example transformation would be to take something like this: + +```rust +#[quickcheck] +fn foo(x: u8) { + // ... +} + +mod bar { + #[quickcheck] + fn bar(x: String, y: u8) { + // ... + } +} +``` + +and output a `main()` that does something like: + +```rust +fn main() { + // handles showing failures, etc + let mut runner = quickcheck::Runner(); + + runner.iter("foo", |random_source| foo(random_source.next().into())); + runner.iter("bar::bar", |random_source| bar::bar(random_source.next().into(), + random_source.next().into())); + runner.finish(); +} +``` + Because this procedural macro is only loaded when it is used as the post-build context, the `#[mytest]` annotation should probably be kept behind `#[cfg(mytest)]` (which is automatically set when the `mytest` From dd240662e098503f0aebb7f0e3b760748de586de Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 09:48:41 +0530 Subject: [PATCH 08/22] be more precise about 'when the mytest context is used' --- text/0000-erfc-post-build-contexts.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 0b430cf4219..5d81fc6f4c1 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -144,8 +144,9 @@ fn main() { Because this procedural macro is only loaded when it is used as the post-build context, the `#[mytest]` annotation should probably be kept -behind `#[cfg(mytest)]` (which is automatically set when the `mytest` -context is used) so that you don't get unknown attribute warnings +behind `#[cfg(mytest)]` (which is automatically set when we are currently running +with the `mytest` context, i.e. `cargo post-build mytest` is being run) +so that you don't get unknown attribute warnings whilst loading, and to avoid conflicts with other post-build contexts that may use the same attributes. (We could change this by asking attributes to be registered in Cargo.toml, but we don't find this From 6097d077d3cebdc8a184dfafa4eb833c346f78bf Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 09:51:23 +0530 Subject: [PATCH 09/22] context crates can't be used as a regular dependency --- text/0000-erfc-post-build-contexts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 5d81fc6f4c1..4e0059e0d4b 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -164,7 +164,7 @@ these are named according to the name of their `#[post_build_context]` function. Crates which define a post-build context must have an `post-build-context = true` -key in their `Cargo.toml`. It can also specify +key in their `Cargo.toml`. They cannot be used as regular dependencies. It can also specify ```rust single-target = true # false by default From 37a7ed834361a87ee563cd75e40dd399528201dd Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 09:59:52 +0530 Subject: [PATCH 10/22] Only define a single context --- text/0000-erfc-post-build-contexts.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 4e0059e0d4b..f777f228899 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -152,6 +152,9 @@ that may use the same attributes. (We could change this by asking attributes to be registered in Cargo.toml, but we don't find this necessary) + +A crate may only define a single post-build context. + ## Cargo integration Alternative post-build contexts need to integrate with cargo. @@ -191,23 +194,22 @@ contexts are defined: ```toml [post-build.context.test] -provider = { test = "1.0", context = "test" } +provider = { test = "1.0" } folders = ["tests/"] [post-build.context.bench] -provider = { test = "1.0", context = "bench" } -folders = ["benchmarks/"] +provider = { ?? = "1.0" } +folders = ["benches/"] + +[post-build.context.examples] +provider = { ?? = "1.0" } +folders = ["examples/"] ``` There's also an `example` context defined that just runs the `main()` of any files given. -These can be overridden by a crate's `Cargo.toml`. The `context` -property is used to disambiguate when a single crate has multiple -functions tagged `#[post_build_context]` (if we were using the example -post-build context further up as a provider, we'd give `mytest` here). -`test` here is `libtest`, though note that it could be maintained -out-of-tree, and shipped with rustup. +These can be overridden by a crate's `Cargo.toml`. To invoke a particular post-build context, a user invokes `cargo context `. `cargo test` and `cargo bench` are aliases for `cargo From 268cdb1da727c8fa8ad6092d8ad7b45d80cdb41f Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 10:46:26 +0530 Subject: [PATCH 11/22] PBC -> test framework, move folder specification to framework Cargo.toml --- text/0000-erfc-post-build-contexts.md | 187 ++++++++++++-------------- 1 file changed, 83 insertions(+), 104 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index f777f228899..546ad61772e 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -1,4 +1,4 @@ -- Feature Name: post_build_contexts +- Feature Name: custom_test_frameworks - Start Date: 2018-01-25 - RFC PR: (leave this empty) - Rust Issue: (leave this empty) @@ -6,7 +6,7 @@ # Summary [summary]: #summary -This is an *experimental RFC* for adding the ability to integrate custom test/bench/etc frameworks ("post-build frameworks") in Rust. +This is an *experimental RFC* for adding the ability to integrate custom test/bench/etc frameworks ("test frameworks") in Rust. # Motivation [motivation]: #motivation @@ -46,8 +46,8 @@ The main two features proposed are: - An API for crates that generate custom binaries, including introspection into the target crate. - - A mechanism for `cargo` integration so that custom post-build - contexts are at the same level of integration as `test` or `bench` as + - A mechanism for `cargo` integration so that custom test frameworks +are at the same level of integration as `test` or `bench` as far as build processes are concerned. [cargo-fuzz]: https://github.com/rust-fuzz/cargo-fuzz @@ -60,20 +60,19 @@ The main two features proposed are: (As an eRFC I'm merging the "guide-level/reference-level" split for now; when we have more concrete ideas we can figure out how to frame it and then the split will make more sense) -The basic idea is that crates can define post-build contexts, which specify +The basic idea is that crates can define test frameworks, which specify how to transform collected test functions and construct a `main()` function, and then crates using these can declare them in their Cargo.toml, which will let -crate developers invoke various test-like post-build steps using the post-build -context. +crate developers invoke various test-like steps using the framework. -## Procedural macro for a new post-build context +## Procedural macro for a new test framework -A custom post-build context is like a whole-crate procedural +A test framework is like a whole-crate procedural macro that is evaluated after all other macros in the target crate have been evaluated. It is passed the `TokenStream` for every element in the -target crate that has a set of attributes the post-build context has -registered interest in. For example, to declare a post-build context +target crate that has a set of attributes the test framework has +registered interest in. For example, to declare a test framework called `mytest`: ```rust @@ -81,7 +80,7 @@ extern crate proc_macro; use proc_macro::TokenStream; // attributes() is optional -#[post_build_context(attributes(foo, bar))] +#[test_framework(attributes(foo, bar))] pub fn mytest(items: &[AnnotatedItem]) -> TokenStream { // ... } @@ -100,17 +99,17 @@ struct AnnotatedItem `items` here contains an `AnnotatedItem` for every item in the target crate that has one of the attributes declared in `attributes` -along with attributes sharing the name of the context (`mytest`, here). +along with attributes sharing the name of the framework (`mytest`, here). -A post-build context could declare that it reacts to multiple different +A test framework could declare that it reacts to multiple different attributes, in which case it would get all items with any of the listed attributes. These items be modules, functions, structs, -statics, or whatever else the post-build context wants to support. Note -that the post-build context function can only see all the annotated +statics, or whatever else the test framework wants to support. Note +that the test framework function can only see all the annotated items, not modify them; modification would have to happen with regular procedural macros. The returned `TokenStream` must declare the `main()` that is to become the entry-point for the binary produced when this -post-build context is used. +test framework is used. So an example transformation would be to take something like this: @@ -143,99 +142,109 @@ fn main() { ``` Because this procedural macro is only loaded when it is used as the -post-build context, the `#[mytest]` annotation should probably be kept +test framework, the `#[mytest]` annotation should probably be kept behind `#[cfg(mytest)]` (which is automatically set when we are currently running -with the `mytest` context, i.e. `cargo post-build mytest` is being run) +with the `mytest` framework, i.e. `cargo test --framework mytest` is being run) so that you don't get unknown attribute warnings -whilst loading, and to avoid conflicts with other post-build contexts +whilst loading, and to avoid conflicts with other frameworks that may use the same attributes. (We could change this by asking attributes to be registered in Cargo.toml, but we don't find this necessary) -A crate may only define a single post-build context. +A crate may only define a single framework. ## Cargo integration -Alternative post-build contexts need to integrate with cargo. +Alternative frameworks need to integrate with cargo. In particular, when crate `a` uses a crate `b` which provides an -post-build context, `a` needs to be able to specify when `b`'s post-build -context should be used. Furthermore, cargo needs to understand that when -`b`'s post-build context is used, `b`'s dependencies must also be linked. -Note that `b` could potentially provide multiple post-build contexts --- -these are named according to the name of their `#[post_build_context]` -function. +framework, `a` needs to be able to specify when `b`'s framework +should be used. Furthermore, cargo needs to understand that when +`b`'s framework is used, `b`'s dependencies must also be linked. -Crates which define a post-build context must have an `post-build-context = true` -key in their `Cargo.toml`. They cannot be used as regular dependencies. It can also specify +Crates which define a test framework must have a `[testing.framework]` +key in their `Cargo.toml`. They cannot be used as regular dependencies. +This section works like this: ```rust +[testing.framework] +name = "bench" # mandatory +folders = ["bench/"] +lib = true # true by default single-target = true # false by default ``` +`lib` specifies if the `--lib` mode exists for this framework by default, +and `folders` specifies which folders the framework applies to. Both can be overridden +by consumers. + `single-target` indicates that only a single target can be run with this -context at once (some tools, like cargo-fuzz, run forever, and so it +framework at once (some tools, like cargo-fuzz, run forever, and so it does not make sense to specify multiple targets). -Crates that wish to *use* a custom post-build context, do so by defining -a new post-build context under a new `build-context` section in their +Crates that wish to *use* a custom test framework, do so by including a framework +under a new `[[testing.frameworks]]` section in their `Cargo.toml`: ```toml -[post-build.context.fuzz] +[[testing.frameworks]] provider = { rust-fuzz = "1.0" } -folders = ["fuzz/"] +name = "fuzz" # optional, overrides `name` on framework crate +folders = ["fuzz/"] # optional, overrides `folders` on framework crate +lib = false # optional, overrides `lib` on framework crate ``` -This defines a post-build context named `fuzz`, which uses the +This pulls in the framework named "fuzz", which uses the implementation provided by the `rust-fuzz` crate. When run, it will be applied to all files in the `fuzz` directory. By default, the following -contexts are defined: +frameworks are defined: ```toml -[post-build.context.test] +[[testing.frameworks]] +name = "test" provider = { test = "1.0" } folders = ["tests/"] -[post-build.context.bench] +[[testing.frameworks]] +name = "bench" provider = { ?? = "1.0" } folders = ["benches/"] -[post-build.context.examples] +[[testing.frameworks]] +name = "example" provider = { ?? = "1.0" } folders = ["examples/"] ``` -There's also an `example` context defined that just runs the `main()` of -any files given. - -These can be overridden by a crate's `Cargo.toml`. +Whereas having two frameworks of the same name is an error, if you define +a framework named "test", "bench", or "example", it will override the implicitly +defined builtin frameworks. -To invoke a particular post-build context, a user invokes `cargo context -`. `cargo test` and `cargo bench` are aliases for `cargo -context test` and `cargo context bench` respectively. Any additional -arguments are passed to the post-build context binary. By convention, the +To invoke a particular framework, a user invokes `cargo test --framework +`. `cargo bench` is an alias for `cargo +test --framework bench`. Any additional +arguments are passed to the testing binary. By convention, the first position argument should allow filtering which targets (tests/benchmarks/etc.) are run. -To run multiple contexts at once, a crate can declare post-build context -*sets*. One such example is the `test` post-build context set, which -will run doctests and the `test` and `examples` context. This can be +To run multiple frameworks at once, a crate can declare testing +*sets*. One such example is the `test` testing set, which +will run doctests and the `test` and `examples` framework. This can be customized: ```toml -[post-build.set.test] -contexts = [test, quickcheck, examples] +[testing.set.test] +frameworks = [test, quickcheck, examples] ``` This means that `cargo test` will, aside from doctests, run `cargo -context test`, `cargo context quickcheck`, and `cargo context examples` +test --framework test`, `cargo test --framework quickcheck`, and `cargo test --framework examples` (and similar stuff for `cargo bench`). It is not possible to make `cargo -test` _not_ run doctests. If both a context and a set exists with a +test` _not_ run doctests. If both a framework and a set exists with a given name, the set takes precedence. -`[[test]]` and `[[example]]` in a crate's `Cargo.toml` add files to the -`test` and `example` contexts respectively. +`[[test]]` and `[[example]]` in a crate's `Cargo.toml` add targets to the +`test` and `example` frameworks respectively. ## To be designed @@ -255,9 +264,9 @@ like json output and whatnot. - This adds more sections to `Cargo.toml`. - This complicates the execution path for cargo, in that it now needs - to know about post-build contexts and sets. + to know about testing frameworks and sets. - Flags and command-line parameters for test and bench will now vary - between post-build contexts, which may confuse users as they move + between testing frameworks, which may confuse users as they move between crates. # Rationale and alternatives @@ -273,7 +282,7 @@ forms of testing or benchmarking. An alternative proposal was to expose an extremely general whole-crate proc macro: ```rust -#[post_build_context(attributes(foo, bar))] +#[test_framework(attributes(foo, bar))] pub fn mytest(crate: TokenStream) -> TokenStream { // ... } @@ -318,7 +327,7 @@ compiler already has) Test crates are now simply proc macro attributes: ```rust -#[proc_macro_attr(attributes(test, foo, bar))] +#[test_framework(attributes(test, foo, bar))] pub fn harness(crate: TokenStream) -> TokenStream { // ... } @@ -339,8 +348,8 @@ there's consensus on some of these we can move them into the main RFC. Documentation tests are somewhat special, in that they cannot easily be expressed as `TokenStream` manipulations. In the first instance, the -right thing to do is probably to have an implicitly defined post-build -context called `doctest` which is included in the post-build context set +right thing to do is probably to have an implicitly defined framework + called `doctest` which is included in the testing set `test` by default (as proposed above). Another argument for punting on doctests is that they are intended to @@ -348,22 +357,6 @@ demonstrate code that the user of a library would write. They're there to document *how* something should be used, and it then makes somewhat less sense to have different "ways" of running them. -## Translating existing cargo test flags - -Today, `cargo test` takes a number of flags such as `--lib`, `--test -foo`, and `--doc`. As it would be a breaking change to change these, -cargo should recognize them and map to the appropriate post-build -contexts. - -Furthermore, `cargo test` lets you pick a single testing target via `--test`, -and `cargo bench` via `--bench`. We'll need to create an agnostic flag -for `cargo context` (we cannot use `--target` because it is already used for -the target architecture, and `--test` is too specific for tests). `--post-build-target` -is one rather verbose suggestion. - -We also need to settle on a command name, `cargo context` and `cargo post-build` -don't quite capture what's going on. - ## Standardizing the output We should probably provide a crate with useful output formatters and @@ -373,27 +366,27 @@ to standardize things like json output and whatnot. ## Namespacing -Currently, two post-build contexts can both declare interest in the same +Currently, two frameworks can both declare interest in the same attributes. How do we deal with collisions (e.g., most test crates will want the attribute `#[test]`). Do we namespace the attributes by the -context name (e.g., `#[mytest::test]`)? Do we require them to be behind +framework name (e.g., `#[mytest::test]`)? Do we require them to be behind `#[cfg(mytest)]`? ## Runtime dependencies and flags -The code generated by the post-build context may itself have dependencies. -Currently there's no way for the post-build context to specify this. One +The code generated by the framework may itself have dependencies. +Currently there's no way for the framework to specify this. One proposal is for the crate to specify _runtime_ dependencies of the -post-build context via: +framework via: ```toml -[context-dependencies] +[testing.framework.dependencies] libfuzzer-sys = ... ``` -If a crate is currently running this post-build context, its -dev-dependencies will be semver-merged with the post-build context's -`context-dependencies`. However, this may not be strictly necessary. +If a crate is currently running this framework, its +dev-dependencies will be semver-merged with the frameworks's +`framework.dependencies`. However, this may not be strictly necessary. Custom derives have a similar problem and they solve it by just asking users to import the correct crate. @@ -402,27 +395,13 @@ users to import the correct crate. The general syntax and toml stuff should be approximately settled on before this eRFC merges, but iterated on later. Naming the feature is hard, some candidates are: - - test framework + - testing framework + - post-build context - build context - execution context None of these are particularly great, ideas would be nice. -## Default folders and sets - -Should a post-build context be able to declare "defaults" for what folders and post-build sets it -should be added to? This might save users from some boilerplate in a large number of situations. - -This could be done in the Cargo.toml as: - -```toml -[post-build.defaults] -folders = ["tests/"] -set = "test" # will automatically be added to the `test` set -``` - -This is useful if a crate wishes to standardize things. - ## Bencher Should we be shipping a bencher by default at all (i.e., in libtest)? Could we instead default From 4fb9d5f58a867e2e5059164ff61b17698b0f1d18 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 10:48:29 +0530 Subject: [PATCH 12/22] Fill in missing pieces about targets --- text/0000-erfc-post-build-contexts.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 546ad61772e..8a0767d10fc 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -243,8 +243,19 @@ test --framework test`, `cargo test --framework quickcheck`, and `cargo test --f test` _not_ run doctests. If both a framework and a set exists with a given name, the set takes precedence. -`[[test]]` and `[[example]]` in a crate's `Cargo.toml` add targets to the -`test` and `example` frameworks respectively. + +You can also add targets to a framework a la `[[test]]` and `[[example]]` via `[[testing.target]]` + +```toml +[[testing.target]] +framework = fuzz +path = "foo.rs" +name = "foo" +``` + +`[[test]]` and `[[example]]` in a crate's `Cargo.toml` are aliases of +`[[testing.target]] framework = test` and `[[testing.target]] framework = example` +respectively. This also goes for `[[bench]]` if we decide to keep that around. ## To be designed From 4ab6c407dc27a811112164139c7c4cefef96c130 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 10:50:37 +0530 Subject: [PATCH 13/22] folders doesn't need a slash --- text/0000-erfc-post-build-contexts.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index 8a0767d10fc..a2bf573a970 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -169,7 +169,7 @@ This section works like this: ```rust [testing.framework] name = "bench" # mandatory -folders = ["bench/"] +folders = ["bench"] lib = true # true by default single-target = true # false by default ``` @@ -190,7 +190,7 @@ under a new `[[testing.frameworks]]` section in their [[testing.frameworks]] provider = { rust-fuzz = "1.0" } name = "fuzz" # optional, overrides `name` on framework crate -folders = ["fuzz/"] # optional, overrides `folders` on framework crate +folders = ["fuzz"] # optional, overrides `folders` on framework crate lib = false # optional, overrides `lib` on framework crate ``` @@ -203,17 +203,17 @@ frameworks are defined: [[testing.frameworks]] name = "test" provider = { test = "1.0" } -folders = ["tests/"] +folders = ["tests"] [[testing.frameworks]] name = "bench" provider = { ?? = "1.0" } -folders = ["benches/"] +folders = ["benches"] [[testing.frameworks]] name = "example" provider = { ?? = "1.0" } -folders = ["examples/"] +folders = ["examples"] ``` Whereas having two frameworks of the same name is an error, if you define From 416a071ecb075e5507ca9dd817a8704734c5c351 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 10:52:33 +0530 Subject: [PATCH 14/22] Split test into test and test-unit --- text/0000-erfc-post-build-contexts.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index a2bf573a970..f62cc1e952a 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -204,16 +204,25 @@ frameworks are defined: name = "test" provider = { test = "1.0" } folders = ["tests"] +lib = false + +[[testing.frameworks]] +name = "test-unit" +provider = { test = "1.0" } +folders = [] +lib = true [[testing.frameworks]] name = "bench" provider = { ?? = "1.0" } folders = ["benches"] +lib = true # could also be split into two [[testing.frameworks]] name = "example" provider = { ?? = "1.0" } folders = ["examples"] +lib = false ``` Whereas having two frameworks of the same name is an error, if you define @@ -234,11 +243,14 @@ customized: ```toml [testing.set.test] -frameworks = [test, quickcheck, examples] +frameworks = [test, test-unit, quickcheck, examples] + +# Default set is [test, test-unit] ``` This means that `cargo test` will, aside from doctests, run `cargo -test --framework test`, `cargo test --framework quickcheck`, and `cargo test --framework examples` +test --framework test`, `cargo test --framework test-unit`, +`cargo test --framework quickcheck`, and `cargo test --framework examples` (and similar stuff for `cargo bench`). It is not possible to make `cargo test` _not_ run doctests. If both a framework and a set exists with a given name, the set takes precedence. From 2ad377877699ae85cf944bb5d93d677ca92df22d Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 11:19:13 +0530 Subject: [PATCH 15/22] Add context --- text/0000-erfc-post-build-contexts.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index f62cc1e952a..dad72bbb37d 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -77,11 +77,11 @@ called `mytest`: ```rust extern crate proc_macro; -use proc_macro::TokenStream; +use proc_macro::{TestFrameworkContext, TokenStream}; // attributes() is optional #[test_framework(attributes(foo, bar))] -pub fn mytest(items: &[AnnotatedItem]) -> TokenStream { +pub fn mytest(context: &TestFrameworkContext) -> TokenStream { // ... } ``` @@ -89,6 +89,11 @@ pub fn mytest(items: &[AnnotatedItem]) -> TokenStream { where ```rust +struct TestFrameworkContext<'a> { + items: &'a [AnnotatedItem], + // ... (may be added in the future) +} + struct AnnotatedItem tokens: TokenStream, span: Span, From 1d9eaeb50761fa4a6a684347c346d3dbc3021895 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 4 Feb 2018 12:51:29 +0530 Subject: [PATCH 16/22] Remove test sets, be clearer on the test names --- text/0000-erfc-post-build-contexts.md | 31 ++------------------------- 1 file changed, 2 insertions(+), 29 deletions(-) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-erfc-post-build-contexts.md index dad72bbb37d..01ee43a9344 100644 --- a/text/0000-erfc-post-build-contexts.md +++ b/text/0000-erfc-post-build-contexts.md @@ -173,7 +173,6 @@ This section works like this: ```rust [testing.framework] -name = "bench" # mandatory folders = ["bench"] lib = true # true by default single-target = true # false by default @@ -194,7 +193,7 @@ under a new `[[testing.frameworks]]` section in their ```toml [[testing.frameworks]] provider = { rust-fuzz = "1.0" } -name = "fuzz" # optional, overrides `name` on framework crate +name = "fuzz" # optional, overrides the crate name folders = ["fuzz"] # optional, overrides `folders` on framework crate lib = false # optional, overrides `lib` on framework crate ``` @@ -209,12 +208,6 @@ frameworks are defined: name = "test" provider = { test = "1.0" } folders = ["tests"] -lib = false - -[[testing.frameworks]] -name = "test-unit" -provider = { test = "1.0" } -folders = [] lib = true [[testing.frameworks]] @@ -241,26 +234,6 @@ arguments are passed to the testing binary. By convention, the first position argument should allow filtering which targets (tests/benchmarks/etc.) are run. -To run multiple frameworks at once, a crate can declare testing -*sets*. One such example is the `test` testing set, which -will run doctests and the `test` and `examples` framework. This can be -customized: - -```toml -[testing.set.test] -frameworks = [test, test-unit, quickcheck, examples] - -# Default set is [test, test-unit] -``` - -This means that `cargo test` will, aside from doctests, run `cargo -test --framework test`, `cargo test --framework test-unit`, -`cargo test --framework quickcheck`, and `cargo test --framework examples` -(and similar stuff for `cargo bench`). It is not possible to make `cargo -test` _not_ run doctests. If both a framework and a set exists with a -given name, the set takes precedence. - - You can also add targets to a framework a la `[[test]]` and `[[example]]` via `[[testing.target]]` ```toml @@ -292,7 +265,7 @@ like json output and whatnot. - This adds more sections to `Cargo.toml`. - This complicates the execution path for cargo, in that it now needs - to know about testing frameworks and sets. + to know about testing frameworks. - Flags and command-line parameters for test and bench will now vary between testing frameworks, which may confuse users as they move between crates. From 3b35bc0993c1687d772bcedcf7263b015400b353 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 15 Feb 2018 16:05:43 -0800 Subject: [PATCH 17/22] rename --- ...erfc-post-build-contexts.md => 0000-custom-test-frameworks.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-erfc-post-build-contexts.md => 0000-custom-test-frameworks.md} (100%) diff --git a/text/0000-erfc-post-build-contexts.md b/text/0000-custom-test-frameworks.md similarity index 100% rename from text/0000-erfc-post-build-contexts.md rename to text/0000-custom-test-frameworks.md From a6f0074fc23b956e202cfe49186d587505add29e Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 15 Feb 2018 16:12:24 -0800 Subject: [PATCH 18/22] pare down cargo parts --- text/0000-custom-test-frameworks.md | 83 +++++++++-------------------- 1 file changed, 25 insertions(+), 58 deletions(-) diff --git a/text/0000-custom-test-frameworks.md b/text/0000-custom-test-frameworks.md index 01ee43a9344..0d0b6a4fbb8 100644 --- a/text/0000-custom-test-frameworks.md +++ b/text/0000-custom-test-frameworks.md @@ -80,8 +80,8 @@ extern crate proc_macro; use proc_macro::{TestFrameworkContext, TokenStream}; // attributes() is optional -#[test_framework(attributes(foo, bar))] -pub fn mytest(context: &TestFrameworkContext) -> TokenStream { +#[test_framework] +pub fn test(context: &TestFrameworkContext) -> TokenStream { // ... } ``` @@ -104,28 +104,22 @@ struct AnnotatedItem `items` here contains an `AnnotatedItem` for every item in the target crate that has one of the attributes declared in `attributes` -along with attributes sharing the name of the framework (`mytest`, here). - -A test framework could declare that it reacts to multiple different -attributes, in which case it would get all items with any of the -listed attributes. These items be modules, functions, structs, -statics, or whatever else the test framework wants to support. Note -that the test framework function can only see all the annotated -items, not modify them; modification would have to happen with regular -procedural macros. The returned `TokenStream` must declare the `main()` -that is to become the entry-point for the binary produced when this -test framework is used. +along with attributes sharing the name of the framework (`test`, here). + +The annotated function _must_ be named "test" for a test framework and +"bench" for a bench framework. We currently do not support +any other kind of framework, but we may in the future. So an example transformation would be to take something like this: ```rust -#[quickcheck] +#[test] fn foo(x: u8) { // ... } mod bar { - #[quickcheck] + #[test] fn bar(x: String, y: u8) { // ... } @@ -173,9 +167,7 @@ This section works like this: ```rust [testing.framework] -folders = ["bench"] -lib = true # true by default -single-target = true # false by default +kind = "test" # or bench ``` `lib` specifies if the `--lib` mode exists for this framework by default, @@ -193,59 +185,26 @@ under a new `[[testing.frameworks]]` section in their ```toml [[testing.frameworks]] provider = { rust-fuzz = "1.0" } -name = "fuzz" # optional, overrides the crate name -folders = ["fuzz"] # optional, overrides `folders` on framework crate -lib = false # optional, overrides `lib` on framework crate ``` This pulls in the framework named "fuzz", which uses the implementation provided by the `rust-fuzz` crate. When run, it will be applied to all files in the `fuzz` directory. By default, the following -frameworks are defined: +framework is defined: ```toml [[testing.frameworks]] -name = "test" provider = { test = "1.0" } -folders = ["tests"] -lib = true - -[[testing.frameworks]] -name = "bench" -provider = { ?? = "1.0" } -folders = ["benches"] -lib = true # could also be split into two - -[[testing.frameworks]] -name = "example" -provider = { ?? = "1.0" } -folders = ["examples"] -lib = false ``` -Whereas having two frameworks of the same name is an error, if you define -a framework named "test", "bench", or "example", it will override the implicitly -defined builtin frameworks. +(We may define a default framework for bench in the future) -To invoke a particular framework, a user invokes `cargo test --framework -`. `cargo bench` is an alias for `cargo -test --framework bench`. Any additional -arguments are passed to the testing binary. By convention, the -first position argument should allow filtering which targets -(tests/benchmarks/etc.) are run. +Declaring a test framework will replace the existing default one. You cannot declare +more than one test or bench framework. -You can also add targets to a framework a la `[[test]]` and `[[example]]` via `[[testing.target]]` - -```toml -[[testing.target]] -framework = fuzz -path = "foo.rs" -name = "foo" -``` - -`[[test]]` and `[[example]]` in a crate's `Cargo.toml` are aliases of -`[[testing.target]] framework = test` and `[[testing.target]] framework = example` -respectively. This also goes for `[[bench]]` if we decide to keep that around. +To invoke a particular framework, a user invokes `cargo test` or `cargo bench`. Any additional +arguments are passed to the testing binary. By convention, the first position argument should allow +filtering which targets (tests/benchmarks/etc.) are run. ## To be designed @@ -412,5 +371,13 @@ If this RFC lands and [RFC 2287] is rejected, we should probably try to stabiliz `test::black_box` in some form (maybe `mem::black_box` and `mem::clobber` as detailed in [this amendment]). +## Cargo integration + +A previous iteration of this RFC allowed for test frameworks to declare new attributes +and folders, so you would have `cargo test --kind quickcheck` look for tests in the +`quickcheck/` folder that were annotated with `#[quickcheck]`. + +This is no longer the case, but we may wish to add this again. + [RFC 2287]: https://github.com/rust-lang/rfcs/pull/2287 [this amendment]: https://github.com/Manishearth/rfcs/pull/1 From c6a3982052218ba127cf02e63f524b550f859ab9 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Thu, 15 Feb 2018 16:21:12 -0800 Subject: [PATCH 19/22] rm fuzz --- text/0000-custom-test-frameworks.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/text/0000-custom-test-frameworks.md b/text/0000-custom-test-frameworks.md index 0d0b6a4fbb8..41552bda97c 100644 --- a/text/0000-custom-test-frameworks.md +++ b/text/0000-custom-test-frameworks.md @@ -104,7 +104,8 @@ struct AnnotatedItem `items` here contains an `AnnotatedItem` for every item in the target crate that has one of the attributes declared in `attributes` -along with attributes sharing the name of the framework (`test`, here). +along with attributes sharing the name of the framework (`test`, here -- +the function must be named either `test` or `bench`). The annotated function _must_ be named "test" for a test framework and "bench" for a bench framework. We currently do not support @@ -184,12 +185,10 @@ under a new `[[testing.frameworks]]` section in their ```toml [[testing.frameworks]] -provider = { rust-fuzz = "1.0" } +provider = { quickcheck = "1.0" } ``` -This pulls in the framework named "fuzz", which uses the -implementation provided by the `rust-fuzz` crate. When run, it will be -applied to all files in the `fuzz` directory. By default, the following +This pulls in the framework from the "quickcheck" crate. By default, the following framework is defined: ```toml From 3307ad7fb48790aee7611a08d407ff00b8318446 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Sun, 18 Mar 2018 21:00:11 -0700 Subject: [PATCH 20/22] handle public --- text/0000-custom-test-frameworks.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/text/0000-custom-test-frameworks.md b/text/0000-custom-test-frameworks.md index 41552bda97c..9364998c742 100644 --- a/text/0000-custom-test-frameworks.md +++ b/text/0000-custom-test-frameworks.md @@ -141,15 +141,9 @@ fn main() { } ``` -Because this procedural macro is only loaded when it is used as the -test framework, the `#[mytest]` annotation should probably be kept -behind `#[cfg(mytest)]` (which is automatically set when we are currently running -with the `mytest` framework, i.e. `cargo test --framework mytest` is being run) -so that you don't get unknown attribute warnings -whilst loading, and to avoid conflicts with other frameworks -that may use the same attributes. (We could change this by asking -attributes to be registered in Cargo.toml, but we don't find this -necessary) +The compiler will make marked items `pub(crate)` (i.e. by making +all their parent modules public). `#[test]` and `#[bench]` items will only exist +with `--cfg test` (or bench), which is automatically set when running tests. A crate may only define a single framework. From 5362a8671cbc1b0df3bec0c5ef5ca680d08c4ad4 Mon Sep 17 00:00:00 2001 From: Manish Goregaokar Date: Mon, 26 Mar 2018 10:56:44 +0200 Subject: [PATCH 21/22] Make the proc macro an experimentation thing --- text/0000-custom-test-frameworks.md | 191 +++++++++++++++------------- 1 file changed, 102 insertions(+), 89 deletions(-) diff --git a/text/0000-custom-test-frameworks.md b/text/0000-custom-test-frameworks.md index 9364998c742..73e2a30acac 100644 --- a/text/0000-custom-test-frameworks.md +++ b/text/0000-custom-test-frameworks.md @@ -68,9 +68,79 @@ crate developers invoke various test-like steps using the framework. ## Procedural macro for a new test framework -A test framework is like a whole-crate procedural -macro that is evaluated after all other macros in the target crate have -been evaluated. It is passed the `TokenStream` for every element in the +A test framework is like a procedural macro that is evaluated after all other macros in the target +crate have been evaluated. The exact mechanism is left up to the experimentation phase, however we +have some proposals at the end of this RFC. + + +A crate may only define a single framework. + +## Cargo integration + +Alternative frameworks need to integrate with cargo. +In particular, when crate `a` uses a crate `b` which provides an +framework, `a` needs to be able to specify when `b`'s framework +should be used. Furthermore, cargo needs to understand that when +`b`'s framework is used, `b`'s dependencies must also be linked. + +Crates which define a test framework must have a `[testing.framework]` +key in their `Cargo.toml`. They cannot be used as regular dependencies. +This section works like this: + +```rust +[testing.framework] +kind = "test" # or bench +``` + +`lib` specifies if the `--lib` mode exists for this framework by default, +and `folders` specifies which folders the framework applies to. Both can be overridden +by consumers. + +`single-target` indicates that only a single target can be run with this +framework at once (some tools, like cargo-fuzz, run forever, and so it +does not make sense to specify multiple targets). + +Crates that wish to *use* a custom test framework, do so by including a framework +under a new `[[testing.frameworks]]` section in their +`Cargo.toml`: + +```toml +[[testing.frameworks]] +provider = { quickcheck = "1.0" } +``` + +This pulls in the framework from the "quickcheck" crate. By default, the following +framework is defined: + +```toml +[[testing.frameworks]] +provider = { test = "1.0" } +``` + +(We may define a default framework for bench in the future) + +Declaring a test framework will replace the existing default one. You cannot declare +more than one test or bench framework. + +To invoke a particular framework, a user invokes `cargo test` or `cargo bench`. Any additional +arguments are passed to the testing binary. By convention, the first position argument should allow +filtering which targets (tests/benchmarks/etc.) are run. + +## To be designed + +This contains things which we should attempt to solve in the course of this experiment, for which this eRFC +does not currently provide a concrete proposal. + +## Procedural macro design + + +We have a bunch of concrete proposals here, but haven't yet chosen one. + +### main() function generation with test collector + +One possible design is to have a proc macro that simply generates `main()` + +It is passed the `TokenStream` for every element in the target crate that has a set of attributes the test framework has registered interest in. For example, to declare a test framework called `mytest`: @@ -146,91 +216,7 @@ all their parent modules public). `#[test]` and `#[bench]` items will only exist with `--cfg test` (or bench), which is automatically set when running tests. -A crate may only define a single framework. - -## Cargo integration - -Alternative frameworks need to integrate with cargo. -In particular, when crate `a` uses a crate `b` which provides an -framework, `a` needs to be able to specify when `b`'s framework -should be used. Furthermore, cargo needs to understand that when -`b`'s framework is used, `b`'s dependencies must also be linked. - -Crates which define a test framework must have a `[testing.framework]` -key in their `Cargo.toml`. They cannot be used as regular dependencies. -This section works like this: - -```rust -[testing.framework] -kind = "test" # or bench -``` - -`lib` specifies if the `--lib` mode exists for this framework by default, -and `folders` specifies which folders the framework applies to. Both can be overridden -by consumers. - -`single-target` indicates that only a single target can be run with this -framework at once (some tools, like cargo-fuzz, run forever, and so it -does not make sense to specify multiple targets). - -Crates that wish to *use* a custom test framework, do so by including a framework -under a new `[[testing.frameworks]]` section in their -`Cargo.toml`: - -```toml -[[testing.frameworks]] -provider = { quickcheck = "1.0" } -``` - -This pulls in the framework from the "quickcheck" crate. By default, the following -framework is defined: - -```toml -[[testing.frameworks]] -provider = { test = "1.0" } -``` - -(We may define a default framework for bench in the future) - -Declaring a test framework will replace the existing default one. You cannot declare -more than one test or bench framework. - -To invoke a particular framework, a user invokes `cargo test` or `cargo bench`. Any additional -arguments are passed to the testing binary. By convention, the first position argument should allow -filtering which targets (tests/benchmarks/etc.) are run. - -## To be designed - -This contains things which we should attempt to solve in the course of this experiment, for which this eRFC -does not currently provide a concrete proposal. - -### Standardizing the output - -We should probably provide a crate with useful output formatters and stuff so that if test harnesses desire, they can -use the same output formatting as a regular test. This also provides a centralized location to standardize things -like json output and whatnot. - -@killercup is working on a proposal for this which I will try to work in. - -# Drawbacks -[drawbacks]: #drawbacks - - - This adds more sections to `Cargo.toml`. - - This complicates the execution path for cargo, in that it now needs - to know about testing frameworks. - - Flags and command-line parameters for test and bench will now vary - between testing frameworks, which may confuse users as they move - between crates. - -# Rationale and alternatives -[alternatives]: #alternatives - -We could stabilize `#[bench]` and extend libtest with setup/teardown and -other requested features. This would complicate the in-tree libtest, -introduce a barrier for community contributions, and discourage other -forms of testing or benchmarking. - -## Alternative procedural macro +### Whole-crate procedural macro An alternative proposal was to expose an extremely general whole-crate proc macro: @@ -266,7 +252,7 @@ these tools usually operate at a different layer of abstraction so it might not A major drawback of this proposal is that it is very general, and perhaps too powerful. We're currently using the more focused API in the eRFC, and may switch to this during experimentation if a pressing need crops up. -## Alternative procedural macro with minimal compiler changes +### Alternative procedural macro with minimal compiler changes The above proposal can be made even more general, minimizing the impact on the compiler. @@ -289,6 +275,33 @@ pub fn harness(crate: TokenStream) -> TokenStream { The cargo functionality will basically compile the file with the right dependencies and `--attribute=your_crate::harness`. + +### Standardizing the output + +We should probably provide a crate with useful output formatters and stuff so that if test harnesses desire, they can +use the same output formatting as a regular test. This also provides a centralized location to standardize things +like json output and whatnot. + +@killercup is working on a proposal for this which I will try to work in. + +# Drawbacks +[drawbacks]: #drawbacks + + - This adds more sections to `Cargo.toml`. + - This complicates the execution path for cargo, in that it now needs + to know about testing frameworks. + - Flags and command-line parameters for test and bench will now vary + between testing frameworks, which may confuse users as they move + between crates. + +# Rationale and alternatives +[alternatives]: #alternatives + +We could stabilize `#[bench]` and extend libtest with setup/teardown and +other requested features. This would complicate the in-tree libtest, +introduce a barrier for community contributions, and discourage other +forms of testing or benchmarking. + # Unresolved questions [unresolved]: #unresolved-questions From a7e3169fb428f6fdcc2ecd8def84eb2424f99e22 Mon Sep 17 00:00:00 2001 From: Mazdak Farrokhzad Date: Sat, 28 Apr 2018 08:44:55 +0200 Subject: [PATCH 22/22] RFC 2318 --- ...om-test-frameworks.md => 2318-custom-test-frameworks.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-custom-test-frameworks.md => 2318-custom-test-frameworks.md} (98%) diff --git a/text/0000-custom-test-frameworks.md b/text/2318-custom-test-frameworks.md similarity index 98% rename from text/0000-custom-test-frameworks.md rename to text/2318-custom-test-frameworks.md index 73e2a30acac..f3cd5aa7663 100644 --- a/text/0000-custom-test-frameworks.md +++ b/text/2318-custom-test-frameworks.md @@ -1,7 +1,7 @@ -- Feature Name: custom_test_frameworks +- Feature Name: `custom_test_frameworks` - Start Date: 2018-01-25 -- RFC PR: (leave this empty) -- Rust Issue: (leave this empty) +- RFC PR: [rust-lang/rfcs#2318](https://github.com/rust-lang/rfcs/pull/2318) +- Rust Issue: [rust-lang/rust#50297](https://github.com/rust-lang/rust/issues/50297) # Summary [summary]: #summary