diff --git a/text/2318-custom-test-frameworks.md b/text/2318-custom-test-frameworks.md new file mode 100644 index 00000000000..f3cd5aa7663 --- /dev/null +++ b/text/2318-custom-test-frameworks.md @@ -0,0 +1,389 @@ +- Feature Name: `custom_test_frameworks` +- Start Date: 2018-01-25 +- RFC PR: [rust-lang/rfcs#2318](https://github.com/rust-lang/rfcs/pull/2318) +- Rust Issue: [rust-lang/rust#50297](https://github.com/rust-lang/rust/issues/50297) + +# Summary +[summary]: #summary + +This is an *experimental RFC* for adding the ability to integrate custom test/bench/etc frameworks ("test frameworks") in Rust. + +# Motivation +[motivation]: #motivation + +Currently, Rust lets you write unit tests with a `#[test]` attribute. We also have an unstable `#[bench]` attribute which lets one write benchmarks. + +In general it's not easy to use your own testing strategy. Implementing something that can work +within a `#[test]` attribute is fine (`quickcheck` does this with a macro), but changing the overall +strategy is hard. For example, `quickcheck` would work even better if it could be done as: + +```rust +#[quickcheck] +fn test(input1: u8, input2: &str) { + // ... +} +``` + +If you're trying to do something other than testing, you're out of luck -- only tests, benches, and examples +get the integration from `cargo` for building auxiliary binaries the correct way. [cargo-fuzz] has to +work around this by creating a special fuzzing crate that's hooked up the right way, and operating inside +of that. Ideally, one would be able to just write fuzz targets under `fuzz/`. + +[Compiletest] (rustc's test framework) would be another kind of thing that would be nice to +implement this way. Currently it compiles the test cases by manually running `rustc`, but it has the +same problem as cargo-fuzz where getting these flags right is hard. This too could be implemented as +a custom test framework. + +A profiling framework may want to use this mode to instrument the binary in a certain way. We +can already do this via proc macros, but having it hook through `cargo test` would be neat. + +Overall, it would be good to have a generic framework for post-build steps that can support use +cases like `#[test]` (both the built-in one and quickcheck), `#[bench]` (both built in and custom +ones like [criterion]), `examples`, and things like fuzzing. While we may not necessarily rewrite +the built in test/bench/example infra in terms of the new framework, it should be possible to do so. + +The main two features proposed are: + + - An API for crates that generate custom binaries, including + introspection into the target crate. + - A mechanism for `cargo` integration so that custom test frameworks +are at the same level of integration as `test` or `bench` as + far as build processes are concerned. + + [cargo-fuzz]: https://github.com/rust-fuzz/cargo-fuzz + [criterion]: https://github.com/japaric/criterion.rs + [Compiletest]: http://github.com/laumann/compiletest-rs + +# Detailed proposal +[detailed-proposal]: #detailed-proposal + +(As an eRFC I'm merging the "guide-level/reference-level" split for now; when we have more concrete +ideas we can figure out how to frame it and then the split will make more sense) + +The basic idea is that crates can define test frameworks, which specify +how to transform collected test functions and construct a `main()` function, +and then crates using these can declare them in their Cargo.toml, which will let +crate developers invoke various test-like steps using the framework. + + +## Procedural macro for a new test framework + +A test framework is like a procedural macro that is evaluated after all other macros in the target +crate have been evaluated. The exact mechanism is left up to the experimentation phase, however we +have some proposals at the end of this RFC. + + +A crate may only define a single framework. + +## Cargo integration + +Alternative frameworks need to integrate with cargo. +In particular, when crate `a` uses a crate `b` which provides an +framework, `a` needs to be able to specify when `b`'s framework +should be used. Furthermore, cargo needs to understand that when +`b`'s framework is used, `b`'s dependencies must also be linked. + +Crates which define a test framework must have a `[testing.framework]` +key in their `Cargo.toml`. They cannot be used as regular dependencies. +This section works like this: + +```rust +[testing.framework] +kind = "test" # or bench +``` + +`lib` specifies if the `--lib` mode exists for this framework by default, +and `folders` specifies which folders the framework applies to. Both can be overridden +by consumers. + +`single-target` indicates that only a single target can be run with this +framework at once (some tools, like cargo-fuzz, run forever, and so it +does not make sense to specify multiple targets). + +Crates that wish to *use* a custom test framework, do so by including a framework +under a new `[[testing.frameworks]]` section in their +`Cargo.toml`: + +```toml +[[testing.frameworks]] +provider = { quickcheck = "1.0" } +``` + +This pulls in the framework from the "quickcheck" crate. By default, the following +framework is defined: + +```toml +[[testing.frameworks]] +provider = { test = "1.0" } +``` + +(We may define a default framework for bench in the future) + +Declaring a test framework will replace the existing default one. You cannot declare +more than one test or bench framework. + +To invoke a particular framework, a user invokes `cargo test` or `cargo bench`. Any additional +arguments are passed to the testing binary. By convention, the first position argument should allow +filtering which targets (tests/benchmarks/etc.) are run. + +## To be designed + +This contains things which we should attempt to solve in the course of this experiment, for which this eRFC +does not currently provide a concrete proposal. + +## Procedural macro design + + +We have a bunch of concrete proposals here, but haven't yet chosen one. + +### main() function generation with test collector + +One possible design is to have a proc macro that simply generates `main()` + +It is passed the `TokenStream` for every element in the +target crate that has a set of attributes the test framework has +registered interest in. For example, to declare a test framework +called `mytest`: + +```rust +extern crate proc_macro; +use proc_macro::{TestFrameworkContext, TokenStream}; + +// attributes() is optional +#[test_framework] +pub fn test(context: &TestFrameworkContext) -> TokenStream { + // ... +} +``` + +where + +```rust +struct TestFrameworkContext<'a> { + items: &'a [AnnotatedItem], + // ... (may be added in the future) +} + +struct AnnotatedItem + tokens: TokenStream, + span: Span, + attributes: TokenStream, + path: SomeTypeThatRepresentsPathToItem +} +``` + +`items` here contains an `AnnotatedItem` for every item in the +target crate that has one of the attributes declared in `attributes` +along with attributes sharing the name of the framework (`test`, here -- +the function must be named either `test` or `bench`). + +The annotated function _must_ be named "test" for a test framework and +"bench" for a bench framework. We currently do not support +any other kind of framework, but we may in the future. + +So an example transformation would be to take something like this: + +```rust +#[test] +fn foo(x: u8) { + // ... +} + +mod bar { + #[test] + fn bar(x: String, y: u8) { + // ... + } +} +``` + +and output a `main()` that does something like: + +```rust +fn main() { + // handles showing failures, etc + let mut runner = quickcheck::Runner(); + + runner.iter("foo", |random_source| foo(random_source.next().into())); + runner.iter("bar::bar", |random_source| bar::bar(random_source.next().into(), + random_source.next().into())); + runner.finish(); +} +``` + +The compiler will make marked items `pub(crate)` (i.e. by making +all their parent modules public). `#[test]` and `#[bench]` items will only exist +with `--cfg test` (or bench), which is automatically set when running tests. + + +### Whole-crate procedural macro + +An alternative proposal was to expose an extremely general whole-crate proc macro: + +```rust +#[test_framework(attributes(foo, bar))] +pub fn mytest(crate: TokenStream) -> TokenStream { + // ... +} +``` + +and then we can maintain a helper crate, out of tree, that uses `syn` to provide a nicer +API, perhaps something like: + +```rust +fn clean_entry_point(tree: syn::ItemMod) -> syn::ItemMod; + +trait TestCollector { + fn fold_function(&mut self, path: syn::Path, func: syn::ItemFn) -> syn::ItemFn; +} + +fn collect_tests(collector: &mut T, tree: syn::ItemMod) -> ItemMod; +``` + +This lets us continue to develop things outside of tree without perma-stabilizing an API; +and it also lets us provide a friendlier API via the helper crate. + +It also lets crates like `cargo-fuzz` introduce things like a `#![no_main]` attribute or do +other antics. + +Finally, it handles the "profiling framework" case as mentioned in the motivation. On the other hand, +these tools usually operate at a different layer of abstraction so it might not be necessary. + +A major drawback of this proposal is that it is very general, and perhaps too powerful. We're currently using the +more focused API in the eRFC, and may switch to this during experimentation if a pressing need crops up. + +### Alternative procedural macro with minimal compiler changes + +The above proposal can be made even more general, minimizing the impact on the compiler. + +This assumes that `#![foo]` ("inner attribute") macros work on modules and on crates. + +The idea is that the compiler defines no new proc macro surface, and instead simply exposes +a `--attribute` flag. This flag, like `-Zextra-plugins`, lets you attach a proc macro attribute +to the whole crate before compiling. (This flag actually generalizes a bunch of flags that the +compiler already has) + +Test crates are now simply proc macro attributes: + +```rust +#[test_framework(attributes(test, foo, bar))] +pub fn harness(crate: TokenStream) -> TokenStream { + // ... +} +``` + +The cargo functionality will basically compile the file with the right dependencies +and `--attribute=your_crate::harness`. + + +### Standardizing the output + +We should probably provide a crate with useful output formatters and stuff so that if test harnesses desire, they can +use the same output formatting as a regular test. This also provides a centralized location to standardize things +like json output and whatnot. + +@killercup is working on a proposal for this which I will try to work in. + +# Drawbacks +[drawbacks]: #drawbacks + + - This adds more sections to `Cargo.toml`. + - This complicates the execution path for cargo, in that it now needs + to know about testing frameworks. + - Flags and command-line parameters for test and bench will now vary + between testing frameworks, which may confuse users as they move + between crates. + +# Rationale and alternatives +[alternatives]: #alternatives + +We could stabilize `#[bench]` and extend libtest with setup/teardown and +other requested features. This would complicate the in-tree libtest, +introduce a barrier for community contributions, and discourage other +forms of testing or benchmarking. + +# Unresolved questions +[unresolved]: #unresolved-questions + +These are mostly intended to be resolved during the experimental +feature. Many of these have strawman proposals -- unlike the rest of this RFC, +these proposals have not been discussed as thoroughly. If folks feel like +there's consensus on some of these we can move them into the main RFC. + +## Integration with doctests + +Documentation tests are somewhat special, in that they cannot easily be +expressed as `TokenStream` manipulations. In the first instance, the +right thing to do is probably to have an implicitly defined framework + called `doctest` which is included in the testing set +`test` by default (as proposed above). + +Another argument for punting on doctests is that they are intended to +demonstrate code that the user of a library would write. They're there +to document *how* something should be used, and it then makes somewhat +less sense to have different "ways" of running them. + +## Standardizing the output + +We should probably provide a crate with useful output formatters and +stuff so that if test harnesses desire, they can use the same output +formatting as a regular test. This also provides a centralized location +to standardize things like json output and whatnot. + +## Namespacing + +Currently, two frameworks can both declare interest in the same +attributes. How do we deal with collisions (e.g., most test crates will +want the attribute `#[test]`). Do we namespace the attributes by the +framework name (e.g., `#[mytest::test]`)? Do we require them to be behind +`#[cfg(mytest)]`? + +## Runtime dependencies and flags + +The code generated by the framework may itself have dependencies. +Currently there's no way for the framework to specify this. One +proposal is for the crate to specify _runtime_ dependencies of the +framework via: + +```toml +[testing.framework.dependencies] +libfuzzer-sys = ... +``` + +If a crate is currently running this framework, its +dev-dependencies will be semver-merged with the frameworks's +`framework.dependencies`. However, this may not be strictly necessary. +Custom derives have a similar problem and they solve it by just asking +users to import the correct crate. + +## Naming + +The general syntax and toml stuff should be approximately settled on before this eRFC merges, but +iterated on later. Naming the feature is hard, some candidates are: + + - testing framework + - post-build context + - build context + - execution context + +None of these are particularly great, ideas would be nice. + +## Bencher + +Should we be shipping a bencher by default at all (i.e., in libtest)? Could we instead default +`cargo bench` to a `rust-lang-nursery` `bench` crate? + +If this RFC lands and [RFC 2287] is rejected, we should probably try to stabilize +`test::black_box` in some form (maybe `mem::black_box` and `mem::clobber` as detailed +in [this amendment]). + +## Cargo integration + +A previous iteration of this RFC allowed for test frameworks to declare new attributes +and folders, so you would have `cargo test --kind quickcheck` look for tests in the +`quickcheck/` folder that were annotated with `#[quickcheck]`. + +This is no longer the case, but we may wish to add this again. + + [RFC 2287]: https://github.com/rust-lang/rfcs/pull/2287 + [this amendment]: https://github.com/Manishearth/rfcs/pull/1