Doctests don't work in bin targets, non-public items

[kornelski]

It's surprising that doctests sometimes don't run at all, and there's no warning about this.

cargo new foo --bin

/// ```rust
/// assert!(false);
/// ```
fn main() {
    println!("Hello, world!");
}

cargo test --all

I'd expect the above to fail, but it looks like the doc-comment code is never ran. Even if I explicitly enable doctests for the binary:

[[bin]]
name = "foo"
path = "src/main.rs"
doctest = true

they still aren't run.

(moved from rust-lang/cargo#5477)

[GuillaumeGomez]

It sounds weird to have documentation examples in a bin target, don't you think? There is no documentation to generate so why would there be documentation's code to run?

[kornelski]

This bug is based on feedback from a user.

Cargo makes new users create bin targets by default, so the first time a new Rust user tries doc tests, it's probably in a bin.

There's a mismatch between what rustdoc does (tests publicly-visible documentation) and expectations (that tests in doccomments, in general, are run).

[GuillaumeGomez]

Maybe we give bad information ahead? I wonder if users know the difference between comments and doc comments... For now I don't see this as a rustdoc bug but as a misunderstanding that should be resolved in tutorials/books.

cc @rust-lang/docs

[vi]

Maybe doctests should just run unconditionally?

The code for a non-public bin-crate item may end up in a public, lib-crate item later, so what's wrong having documentation and tests a bit earlier?

[QuietMisdreavus]

The problem with running doctests on bin targets lies with how doctests work. The way rustdoc creates doctests is by building the code sample as a bin target of its own, and linking it against your crate. If "your crate" is really an executable, there's nothing to link against.

If you run cargo test --doc --verbose, you can see that Cargo needs to build your crate ahead of time, and pass the rlib to rustdoc:

[misdreavus@tonberry asdf]$ cargo test --doc --verbose
   Compiling asdf v0.1.0 (file:///home/misdreavus/git/asdf)
     Running `rustc --crate-name asdf src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=31e1a62d36de9a99 -C extra-filename=-31e1a62d36de9a99 --out-dir /home/misdreavus/git/asdf/target/debug/deps -C incremental=/home/misdreavus/git/asdf/target/debug/incremental -L dependency=/home/misdreavus/git/asdf/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.79 secs
   Doc-tests asdf
     Running `rustdoc --test /home/misdreavus/git/asdf/src/lib.rs --crate-name asdf -L dependency=/home/misdreavus/git/asdf/target/debug/deps -L dependency=/home/misdreavus/git/asdf/target/debug/deps --extern asdf=/home/misdreavus/git/asdf/target/debug/deps/libasdf-31e1a62d36de9a99.rlib`

running 1 test
test src/lib.rs - SomeStruct (line 3) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Semantically, i find nothing wrong with allowing doctests in bin crates. However, to pull this off, we would need to tell Cargo to compile bin crates as libs, then hand that rlib to rustdoc for testing. There's nothing stopping us from doing that right now:

[misdreavus@tonberry asdf]$ rustc --crate-type lib --crate-name asdf src/main.rs
[misdreavus@tonberry asdf]$ rustdoc --test --crate-name asdf src/main.rs --extern asdf=libasdf.rlib

running 1 test
test src/main.rs - SomeStruct (line 3) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cc @rust-lang/cargo

[kornelski]

Wow, that's a nice surprise that it works. I was afraid that duplicate main would be a problem, but I guess rlib namespaces it!

[QuietMisdreavus]

Yeah, the main in your bin goes inside the lib crate, so it's just a regular function at that point. I did have to tag it with #[allow(dead_code)] to silence a warning, but otherwise it's all good.

[PhilipDaniels]

I just hit this today. I came here after finding rust-lang/cargo#2009 in a Google search, which mentions that it is 'addressed in the documentation'. I suspect it may have been at one point but it doesn't appear to be now. I checked

https://doc.rust-lang.org/book/ch11-01-writing-tests.html

and from there the link

https://doc.rust-lang.org/book/ch14-02-publishing-to-crates-io.html#documentation-comments-as-tests

(Of course it's possible its mentioned somewhere and I missed it. A very careful reading of the surrounding text implies it, because it's all talking about libs, but nowhere I see is it mentioned explicitly.)

[joshtriplett]

I ran into this today, and found it confusing. I would love to be able to write doctests in the modules of my binary crate.

What would it take to make that work out-of-the-box?

[ehuss]

I can think of a few hurdles that someone would need to address:

• How to resolve symbols in a binary? Doctests behave as-if they are a separate crate linking in the library. But a binary is normally not used a library, so it has no name to refer to. For example:

/// ```
/// do_something();
/// ```
fn do_something() {}

fn main() {}

Here, do_something(); would fail because it is not in scope. You could try to scope it with the name of the binary (mything::do_something();), but then you have two new problems: what if you have a lib target of the same name? And, deal with the fact that you normally don't export any public items in a binary.

• How to deal with lack of public exports? Building a binary as a library will result in a lot of unused warnings. And since rustdoc imports the crate, usually nothing will be publicly exported, so you won't be able to access anything.

One (rough) idea to address this would be to fundamentally change how rustdoc --test works. Instead of creating a synthetic test that links to a library, embed the tests directly in the module. So it would rewrite the above example to:

fn do_something() {}

#[test]
fn do_something_doctest() {
    do_something();
}

So all local (private) symbols would be in scope.

That's a pretty radical difference, so I'm not sure if it would fly (would probably be confusing if there are two styles of doctests).

Also, I think it's worth examining if this is a good idea from the start. Documenting binaries still seems like a strange concept to me. Why can't you just move all the code to a library?

[joshtriplett]

On Sat, Mar 28, 2020 at 02:23:19PM -0700, Eric Huss wrote: I can think of a few hurdles that someone would need to address: * How to resolve symbols in a binary? Doctests behave as-if they are a separate crate linking in the library. But a binary is normally not used a library, so it has no name to refer to. For example: ```rust /// ``` /// do_something(); /// ``` fn do_something() {} fn main() {} ``` Here, `do_something();` would fail because it is not in scope. You could try to scope it with the name of the binary (`mything::do_something();`), but then you have two new problems: what if you have a lib target of the same name? And, deal with the fact that you normally don't export any public items in a binary. * How to deal with lack of public exports? Building a binary as a library will result in a lot of unused warnings. And since rustdoc imports the crate, usually nothing will be publicly exported, so you won't be able to access anything. One (rough) idea to address this would be to fundamentally change how `rustdoc --test` works. Instead of creating a synthetic test that links to a library, embed the tests directly in the module. So it would rewrite the above example to: ```rust fn do_something() {} #[test] fn do_something_doctest() { do_something(); } ``` So all local (private) symbols would be in scope.

I would love to see that as the *normal* approach to doctests, personally. It would also unify the different kinds of tests. I do agree that that would be a more invasive change, though.

Also, I think it's worth examining if this is a good idea from the start. Documenting binaries still seems like a strange concept to me. Why can't you just move all the code to a library?

Why should the concept of documentation and testing be limited to libraries, and not supported in binary crates? I use documentation in binary crates to document the code for later understanding. I'd like to use tests to verify individual modules of functionality; that doesn't necessarily mean I want to split that functionality into a separate library crate.

[dkasak]

Just hit this today. I also frequently document internal modules of binary crates and intuitively expected that doctests would still work. Effectively, I use internal modules as non-public libraries. The reason I do not make them into separate library crates is that they are often not useful in a general context, but I do sometimes end up turning them into libraries. This kind of flexibility seems like a win.

[ayroblu]

Hey, so I just learnt about this while learning about the language. It's disappointing, and I'm not sure this has been prioritised so it doesn't sound like it's going to be addressed in the near future.

Anyways, I guess the only way then is to do the plebeian way and write a test for it

[GuillaumeGomez]

You can split your crate into two parts: the binary part (containing only the main function and the args parsing eventually) which calls all other functions and the "lib" part. Then you can document all functions inside the lib part. I did that for a few crates of mine and I think it's more than enough and is a better match for how doc should be handled. What do you think?

[ayroblu]

@GuillaumeGomez Do you have a good example or minimum POC with like cargo build? Sorry still learning here

[GuillaumeGomez]

Sure, here you go: https://github.com/gtk-rs/gir/blob/master/Cargo.toml

[slightlytwisted]

It sounds weird to have documentation examples in a bin target, don't you think? There is no documentation to generate so why would there be documentation's code to run?

I think mostly for consistency. If someone is used to writing docs with rustdoc and doctests on their library functions, it makes sense that they would want apply the same documentation style to their bin functions. On a small project, having to separate out helper functions to a separate library is an extra hoop to jump through, especially if it is just for the sake of getting tests to run.

I think underneath this is the assumption that people don't need to generate documentation for binaries. But in a team environment, its a good idea to document as much as you can, including examples. Having to switch documentation and test styles between libraries and binaries seems like an arbitrary constraint.

[dkasak]

But in a team environment, its a good idea to document as much as you can, including examples.

Yes, I even document extensively for myself! And you can never be sure whether someone will join you on a project in the future.

[steveklabnik]

I think underneath this is the assumption that people don't need to generate documentation for binaries.

It's not quite this, exactly. Rustdoc is a tool for documenting a Rust API. A binary does not have a Rust API. So rustdoc is not really meant for documenting binaries.

Where the friction really comes is when you want to produce internal documentation. This is a concern for libraries too, and that's why --document-private-items exists. But the way that rustdoc goes about generating this information ends up being tough for binaries, because it's oriented around lbiraries, by nature.

[kornelski]

Doccomments unfortunately couple two things, but I think in this case the concern of running (doc)tests could be separate from concern of documenting things. Some tests like compile_fail just have to live in a doccomment, regardless whether one wants to generate documentation or not.

To look at it from a different perspective: tests are run by cargo test --all, not cargo doc. The fact that rustdoc is involved in testing is an implementation detail. If you ignore the mechanics of how the tests are extracted, in the grand scheme it's just an issue of cargo test --all not running all tests.

[dogweather]

@joshtriplett Take a look at how Elixir does doc tests: in the form of an interactive REPL session:

defmodule MyModule do
  @doc """
  Given a number, returns true if the number is even and else otherwise.

  ## Example
    iex> MyModule.even?(2)
    true
    iex> MyModule.even?(3)
    false
  """
  def even?(number) do
    rem(number, 2) == 0
  end
end

This would be a big change, and implies a REPL that everyone is familiar with. But it's a very low-friction way to write tests. Maybe as a custom testing framework when that hits stable...

[samwightt]

Just hit this roadblock today with a smaller bin project that I was working on and it was very annoying. Doc tests should be standard across all project types, not just library.

[ghost]

Could rustdoc tests be turned into regular tests? That way a use super::*; could be added and then it would actually be possible to test crate private things and in binaries as well.

[dcormier]

Could rustdoc tests be turned into regular tests?

They cannot. You can write doctests that are expected to not compile (with the compile_fail attribute). You cannot do the same with regular tests.

[vi]

Just add compile_fail regular tests? (one such test per file, triggered by some special comment)

[jyn514]

https://github.com/dtolnay/trybuild is a library for testing compile errors. I think it's unlikely it will become part of libtest itself.

[DragonDev1906]

Not knowing anything about how they work, would it be possible to have rustdoc generate unit tests from doctests, rather than separate bin crates? I realize this would probably be a breaking change to existing (and functional) doctests, not to mention may violate the original intent of documenting public APIs, so perhaps in the same way we can add attributes such as ignore or compile_fail, could a unit_test attribute be added to doctests we want to use this behavior?

I'd like to point out this comment by @Kromey:

• We can not convert all doctests to unit tests, because that would mean compile_fail would not work.
• But it should be possible to add the attributes unit_test and unit_test_should_fail (or something like that), which converts only this single example to a unit test.

While it would be impossible to use the compile_fail attribute for private functions (use ignore as a fallback or keep compile_fail even if it always fails compilation due to using private methods), this would at least allow us to use doctests for private functions or in private modules that correctly compile (i.e. everything usually done without attributes or should_panic. At the same time this would not be a breaking change, since these examples currently do not compile anyways.

For those tests we could automatically use super::* as with unit tests and it would make sense to do so because they should behave like a unit test.

Alternatively, this could be done automatically (without a unit_test attribute) for all doctests in private modules, where it is already impossible to run the examples if they use private functions. This might be a breaking change, since existing doctests for private modules/functions that do not use these functions would be run in a different context.

If you ask me, having the ability to execute doctests in private modules iff they are supposed to compile is a big benefit for internal documentation of private modules in a library, even if it has the limitation that the example must compile. Especially when this can be done without a breaking change or in a way where adding the attribute can be automated later. It is really annoying that we have to add ignore to all examples in private modules just because we can not import the function the example is about for the doctest.

[TheButlah]

This bit me today. I am writing a backend service, which has a JSON api. I wanted to document an example of valid JSON, for someone reading the code who is unfamiliar with rust.

Conceptually this should only need to be a bin crate - its an application. But I still have a need to document its API. It has an API because its exposing HTTP endpoints, and that is a form of API.

It would be good if I didn't have to resort to making a trivial main.rs that calls a lib.rs with a main() function. That is not a good developer experience.

[chrisp60]

This caught me today. It seems like very unintuitive behavior. Since I am still very new to the language, the docs are mostly for my own benefit and sanity. I know there is an unstable feature to scrape examples, but I really wish I could just write my examples outside of markdown blocks while still having them be within the same file of what they are documenting.

[Ofenhed]

I will humbly admit I'm not well versed in rustc or cargo internals, so I may be way off. I would still like to make a few comments based on what I would want and which behaviors I would expect as a developer when running cargo test:

1. Public facing documentation targets in libraries should really work as they do today. Giving them access to super:: would be detrimental to the quality of documentation.
2. Private facing documentation targets (such as private functions in libraries or any function in a binary) could be converted to anonymous submodules where they are documented. This would give us access to super::.
   1. For documentation tests marked as compile_fail, the use statement should be conditional. Maybe behind a feature flag, maybe conditionally generated.
   2. Fully private documentation targets could be converted to tests in the same module, as the declared use statements are valid for all functions which can call this function. This is obviously not true for tests marked with compile_fail.
   3. Tests for documentation targets which are not fully private should never just be converted to tests in the same module. I think that declaring usage makes the documentation clearer.
   4. The anonymous submodules could be confusing in compiler error text (file paths), so we would need to process that output. Can we do this?
3. I guess that for this to work we would also need to inject code into main.rs/lib.rs to publish the anonymous test modules to the function which runs the tests?
   1. An alternative could be to parse the main function for mod statements (or something smarter) to create a new library source file we can use for the tests.
   2. It would be great if we could disable unused warnings completely. It would be perfect if we could do it for all modules except our anonymous test modules.

[khionu]

@daniel-pfeiffer People aren't always going to have the variety of perspectives at the early stages of a given exploration. We should be empathetic to that people come from all walks of life and may need some help to understand other perspectives. Your response here was very much not how we should help each other understand things, and that's putting aside that you're replying to something from almost 5 years ago. In addition to the empathy, please watch for context.

Lastly, a lot of your response was loaded with emotional context that didn't do anything for the conversation. All feelings are valid, but how you act on them matters. This wasn't the place to vent about your feelings with how contributions are done.

[JeffThomas]

Yeah this caught me today too. I'm writing something that's not intended to be used as-is (such as a library) but is meant to be forked and expanded on for the users special case. Working examples in the documentation would be very valuable in demonstrating this use, I think. More-so than making potential users dig into tests. I understand that this is a tricky issue but since people are still commenting on it some 4 years after it was first reported, maybe it can be moved up on in priority?

Having functioning examples in the documentation is such a powerful and useful feature it's a shame that it's limited in such a way.

[GuillaumeGomez]

As suggested in the first comments, it could already work by providing a small fix in cargo, which would be the simplest way to achieve this feature. You can try your luck there directly.

[modulok]

I was bit by this today too. I'm in favor of doctests just running, binary or not. It can be very useful when teaching Rust to be able to put a doctest in a binary. Given the number of people just here who have hit this - and bothered to comment - and the number of years this has bitten folks, it'd probably more of an expected behavior if they just always run, even in binaries.

[vi]

If it is tricky to run those doctests, maybe it can be easier to at least detect them and warn.

/// ```
///   2+2
/// ```
fn main() {
    println!("Hello, world!");
}

$ cargo test
warning: doctests do not work in [[bin]] targets or in private items
 --> src/main.rs:1:4

[modulok]

Agreed. That would be helpful until an always-run-tests feature can (if possible and desired) be properly designed and implemented. I doubt implementing such a warning could break anyone's existing code.

[velllu]

I am writing my backend in rust, and I would love to write doctests for the sql-related stuff, but my project is a binary 🤔

I really cannot see a reason to not want this, it does not change any existing programs.

[WyvernIXTL]

The best part is, that in rust by example there is no mention of doctest not working with bin targets:
"Rust takes care about correctness, so these code blocks are compiled and used as documentation tests."
"Code blocks in documentation are automatically tested when running the regular cargo test command"

It should just work.
(A lengthy bump.)

[Ziothh]

This also bit me a while ago and every couple of months I'm searching the issues for progress because I do think this could greatly help people understand other peoples code and intentions.

These are my main pain points:

• Having both a main.rs and a lib.rs is a minor pain point but you shouldn't have to. It could make external people think you're sharing lib code or publishing it to crates.io.
• I'd guess that most rust projects are binaries so the majority of projects don't directly benefit from doctests.
• Even if you have a lib, private function doctests don't work (I know this isn't trivial to implement atm).
  • Because I'm not a library author my private and public functions benefit the same amount from doctests.
  • Rust shouldn't force you tot make functions public just to be testable in the way that you prefer.
• Having great examples for internal code helps team members to better understand each others intentions.
• I often find it easier to understand open source libs instead of bins because they're incentivized to write examples.

If we don't want to break older binaries we could do smth like this:

# Cargo.toml
[[bin]]
name = "mybin"
doctest = true # Defaulting to false

[nm-remarkable]

2024 reporting in! Was very confused why my tests were not running after following rust by example. The part where it says For example, you could write this into your lib.rs to test your README as part of your doctests: makes it sound optional, and there is no documentation (in the same page) that documentation tests do not work with bin.

On a cheerful note, it's a good time to drop this article.

(Once more a lenghty bump on this topic)

[williamareynolds]

I spent an hour trying to figure out why my doctests wouldn't run. So happy I finally found this issue.

[GuillaumeGomez]

Little update for everyone interested: discussions on how to fix this issue (re)started recently. As you might have guessed, it's a tricky issue to solve as it requires to completely change how doctests work. The "fun" part is allowing them to work on internal APIs (the current issue) and also allow them to work as if your crate is a dependency to showcase how to use an API (the original intent and current way they work).

[GuillaumeGomez]

Just to keep everyone up-to-date: since #126245 was merged, the first step is now done. Here is the plan I currently have in mind for allowing this feature to work: just like currently, rustdoc will process all doctests. However, all private items (not reexported or not accessible outside of the current crate, so all items in a binary crate) will now be transformed using the same logic as unit tests (#[test]).

How does it work? If any doctest matches the parameter listed above, rustdoc will call rustc with a flag telling rustc to convert these doctests into something similar to unit tests and to run them. It means they will be compiled alongside their crate, giving access to the module where they are declared. An alternative to this solution would be for rustdoc to duplicate the source code and modify as it sees fit to "expand" the doctests and then compile and run them. It has the advantage of not requiring any change in rustc (possible "optimization": only duplicate files with modifications, the rest can be include!()).

Now what happens if you want a doctest on a public/reexported item to still have access to local code? For that I suggest adding a new code block attribute named unit_test (name is very much up to debate!). These doctests will be treated the same as the ones on private items.

So from this point, the issue I can think of are:

• For doctests on impl items. It will mean that the test will need to be generated outside of the impl block. I'll experiment a bit around that to see how to make it possible.
• For compile_fail doctests. This one is clearly more tricky: we can do a first check on the syntax like we already currently do. If the syntax parsing failed, then the doctest is failing as expected, so all good. But in other cases, we'll need to check with the compiler team if there is a way to say "if this piece of code fails to compile, it's all good, please ignore it and make the test success".

I think that's it. Work on it began. A bit more patience folks. Binacry crate/private items doctests are coming!