Integrate doctest with testsets

[davidanthoff]

This issue is a follow up on #774 and #198, and what a nice public API for that functionality might look like.

My own preference would be something like this in my runtests.jl:

using Documenter, Test, MyPackage

@testset "MyPackage" begin

include("test_featureA.jl")

Documenter.run_all_doctests("MyPackage")

end

I.e. ideally I don't really want to deal with any subtleties at all, but would just like to call one function, and have it sort out everything. In terms of testset integration, my preference would be if run_all_doctests started a new (child) testset and then put each doctest into that testset.

The name for the function is not great, this is more an idea how the API should feel like.

[mortenpi]

There are actually two separate ideas here: (1) provide a simplest possible API for runtests.jl, and (2) integrate it with testsets. I'll dedicate this issue for the testset integration and and leave the API discussion for #1051.

Regarding the testset integration: yep, I think it's a good idea. For the time being though, it's up for grabs -- it needs a slightly more sophisticate implementation for doctest.

[fingolfin]

What exactly is left to be done here, given that doctest(MyPackage) runs all the doctests? We've been using that to run doctests inside of runtests.jl for several years now.

So perhaps it can be closed? Otherwise someone could perhaps clarify what is missing...

---

The main slightly annoying thing I know of is that a few global settings like DocMeta.setdocmeta and doctestfilters need to be duplicated between docs/make.jl and test/runtests.jl.

For us, we have essentially resolved this by establishing a pattern like this:

DocMeta.setdocmeta!(MyPackage, :DocTestSetup, MyPackage.doctestsetup(); recursive=true)
doctest(MyPackage)

where MyPackage.doctestsetup returns the code block.

This then has the added advantage that we can also reuse it in our .md files like so:

```@meta
CurrentModule = MyPackage
DocTestSetup = MyPackage.doctestsetup()
```

(It is still annoying that I have to remember to duplicate this in each .md file or else risk forgetting it in some).

I am not sure whether I should try to derive feature requests for this. But perhaps it would be worth documenting this workflow somewhere, perhaps some other people would benefit from it?

[mortenpi]

Yea, I think this should have been closed ages ago, so I'll do it now. Let's have separate issues for any follow-ups.

I am not sure whether I should try to derive feature requests for this. But perhaps it would be worth documenting this workflow somewhere, perhaps some other people would benefit from it?

I think it's worth documenting common and useful workarounds. And we should also try to figure out if there's some nice approach how we could avoid the duplication -- I agree it's a wart. I don't have any immediately obvious ideas though.