Rustdoc should run all doctests in one binary

[jyn514]

Raised by @Lokathor on discord:

Doctests are slower to run than integration tests. Each individual doctest block runs as a unique binary/process when testing, but each entire integration test file runs as a single process with integration tests. So the test themselves aren't different speeds, but because you're building less bins and spawning less child processes it's faster to go through the integration tests.

I don't think there's any intrinsic reason the tests have to be their own process? The only issue is that many of them have fn main, which would have to be rewritten somehow. This could speed up CI times across the Rust ecosystem, including for the rust compiler itself.

Somewhat related to #51228.

[Nemo157]

fn main() doesn't seem to hard to rewrite, but a lot of nightly-only crates have #![feature(...)] and other crate-level attributes which will be more difficult.

[cuviper]

There are a lot of things a test may do with the assumption that it's an isolated process, like configuring global resources.

[jyn514]

Hmm so this has backwards-compatibility issues, I see. Maybe we could do this in a new edition?

[cuviper]

I could see this in an edition, but I think you'd need an RFC. I think there should be an opt-out too -- maybe tests that contain fn main would remain in their own process, side-stepping the rewrite questions.

[Mark-Simulacrum]

FWIW I am somewhat doubtful that this is a good idea. We likely would have a hard time managing the distinct complexity merging these into a single compilation unit without causing a long stream of issues, not only at run-time but also at compile time (e.g., remapping warning spans and such).

I also don't know that this is truly that high overhead, at least on linux, spawning processes is not too fast but is also not all that slow. Before we invest too much into figuring out how to do this I would like to see a hacked together prototype and results on alloc, core, and std's tests as to how much of a performance win this is. If it's a few percent it seems not worth it.

[Lokathor]

Just anecdotally using two crates I had at hand:

• 809 doctests take 30.3 seconds to run (26.6 / second)
• 67 integration tests take 0.6 seconds to run (111.66 / second)

The timings were just taken by hand with a stopwatch, so they're only approximate, but both test suites were doing similar work per test, just math functions that setup some inputs, run the op, and check the outputs.

It probably varies by test suite, but I expect the difference to come out to a lot more than "a few percent".

[Mark-Simulacrum]

That seems like not quite an apples-to-apples comparison; for one thing the doctests today are all compiled out of process. Plus, is that including the rustc time for compiling those integration tests?

[Lokathor]

that's a cargo test with the tests having already been built. Building the integration tests adds about 2s to that case, putting them about even in terms of tests per second.

I don't recall the extra time taken to build the tests in the doctest case.

Are you saying that the doctests get built, run, and then immediately discarded as part of the cargo test command?

[Mark-Simulacrum]

That's the case today -- currently doctesting involves:

• rustdoc collection of doctests (amortized across all of them), generating the altered source code (e.g., adding fn main)
• spawn a rustc process to compile the doctest
• execute the doctest itself

I have a long-term outstanding TODO for myself to verify that the rustc spawned there receives appropriate access to the jobserver, so I am uncertain whether we're managing parallelism correctly today, which might be another source of wins.

[Nemo157]

On a very small crate (bs58) I copied all my doc-tests into a docs integration test file then tested how long they take to run including build time each way:

> touch src/lib.rs && time cargo test --all-features --test docs
test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
cargo test --all-features --test docs  0.46s user 0.19s system 103% cpu 0.635 total

> touch src/lib.rs && time cargo test --all-features --doc
test result: ok. 29 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
cargo test --all-features --doc  13.20s user 2.85s system 277% cpu 5.777 total

so writing doctests seems to give an ~9x walltime or ~28x cputime slowdown compared to identical integration tests.

(EDIT: splitting into one test per integration test file gives ~the same time as --doc, so it seems like there isn't much to be gained from optimizing how rustdoc runs the tests if they're still individual binaries).

[Lokathor]

If doctests were to be merged, i'd expect doctests without a main to just be merged into a single file and run as a batch.

[ssokolow]

Regardless of the solution, this performance issue is the number-one reason that I opt my examples out of being doctests and leave a note asking people to report a bug if they drift out of sync with the integration test they mirror.

It's simply too much of a drag on my code-compile-test iteration cycle.

On my CPU, I'm taking 3 seconds to rebuild a set of 13 tests as some mix of unit and integration tests and 37 seconds to rebuild those same tests as doctests.

[camelid]

Perhaps an easier solution (which would only solve part of the problem) would be to invoke rustc's API rather than invoking it as the rustc binary. This would also probably help with #81070 (see #81070 (comment)).

[ppershing]

In my experience doctests are performance drag especially in huge workspace which require nontrivial linking time. This lead us to abandon doctests in many places in our project

As for running concurrently - I am not sure how many problems would be caused by running tests concurrently/sequentially in a single process but if we think the main problem is linking performance, a workaround would be to just compile a single binary which would be invoked multiple times with different commandline argument indicating which test to run.

Also, given the previous comment on line mapping and multiple main collisions - a solution might be to create one sub-module per doctest and then collect this to a single fn main() runner? This way, people could keep using fn main() in their doctests, the top-level main would just delegate to these invididual modules. A downside would be that people could start exploiting e.g. super:: or crate:: inside doctests in accordance with Hyrum's law.