Add a secondary mode with --document-private-items

[hdevalence]

Context

Often, the internal parts of a crate have their own documentation, even if they're not exposed as part of the public API. It can be pretty useful to be able to have a rendered version of that documentation for reference and development purposes:

• the doc comments are rendered as text in a way that may be nicer to read;
• the docs are linkable, making it easy to share references to specific items' docs with collaborators;
• the rustdoc interface can show highlights of the internal structure (e.g., all trait implementations in one place);
• the internal documentation may be of independent interest.

For instance, a bunch of notes on the internal details of a crypto implementation I wrote are kept in documentation for internal modules, hosted separately from the "external" docs. Although that's probably an extremal example in terms of "effort put into documentation of private items", I think this would still be useful ecosystem-wide.

Proposal

docs.rs could render documentation for all crates twice, once as normal and once with --document-private-items, then host a second copy of the documentation for all crates under a different URL. Each set of docs would link internally to itself, so following links in the "public" section would remain in the "public" section, and following links in the "private" section would remain in the "private" section.

I'm not attached to this in particular, but my suggestion would be to place all private docs on a separate subdomain, say private.docs.rs, with an identical URL structure to the "public" docs.rs. The advantage of this choice is that:

• it's really easy to switch between internal and external docs by adding or removing the subdomain;
• it's clear from looking at the URL bar which version you're seeing, even if the full path is too long to display.

[jpgoldberg]

I would very much like to see this or perhaps some other mechanism that would allow me to produce

1. Properly rendered documentation for non-public components
2. The source for the documentation should be close to the source code
3. A way to make very very clear to the reader of the documentation that the private parts are private.

The distinction between public API and private components is vital when documenting for external consumers, but that is not the only reason to document code. Well organized documentation should be read and created for those who are developing the unit.

Currently, for example, I'm working on a module that provides only a single public public method, but there are enough different internal parts that it would be extremely useful (even for me, who is the sole author of this at the moment) to have a navigable and reasonably rendered docs. Once I ask colleagues to look at this, it will be even more important that there be something more readable than the raw comments in the source.

[jyn514]

You can currently set this for the entire crate by setting the --document-private-items flag in your cargo.toml (docs). It sounds like you're asking for even more than that though: both public and private documentation. That sounds hard to do in general, I want to avoid building crates twice. Maybe rustdoc could get a feature that marks items as private when passing --document-private-items?

[hdevalence]

Right, setting the --document-private-items flag in a single crate doesn't provide the feature I think would be great: linkable, crossreferenced documentation for the internals for every crate in the ecosystem. Having rustdoc mark items as private when rendering them with --document-private-items would be great, but I think it's independent of this feature request, which is really about rendering crates twice.

Can I ask what the motivation for avoiding building crates twice is? Is it just server load, or is there some other reason?

I'm not sure how the docs.rs infrastructure is organized, but if the private docs were rendered independently on a subdomain, would it be possible to scale out rather than up by running a second copy of the existing infrastructure?

[jyn514]

Can I ask what the motivation for avoiding building crates twice is? Is it just server load, or is there some other reason?

The reason is storage costs, not server load. All the documentation we build we have to store in perpetuity: we currently have 3 TB of docs and that number will only go up over time.

I think this is a reasonable request, but I don't want to implement it until we reduce some of our needless storage use (#379 and #343). See also #608 and #174 (comment) which have been postponed for similar reasons.

[hdevalence]

Ah, that's totally reasonable. Thanks for the information and for your work on docs.rs :)

[pvshvp-oss]

@jyn514 Can private items be labeled as private in any way using rustdoc? Is that too dependent on other requests?

[jyn514]

@shivanandvp not currently, no - if you pass --document-private-items, rustdoc won't distinguish between private and public items. I think it's useful to do so, though - can you open an issue in rust-lang/rust? https://github.com/rust-lang/rust/issues/new/choose

[vi]

Note: private items are now marked with a lock icon in --document-private-items docs.

[jyn514]

I think this is now feasible to implement. We implemented compression a while ago, which greatly decreased our storage costs, and rustdoc now distinguishes private items in the documentation. My main concern is that this will show private items even for crates that have not opted in, which both makes the UI more overwhelming (more docs overall) and exposes the implementation details, even if they're marked with a lock.

That said, I would be very willing to expose this as a config in package.metadata.docs.rs as long as it's opt-in and not opt-out.

[jyn514]

Oh, I missed that the original suggestion was for a separate private.docs.rs domain. Yes, I agree that makes sense to do for all crates without a config.

[Nemo157]

We do have a build duration issue right now, the number of published crates has increased a lot in the last few years and the build system hasn't scaled with it. We're pretty close to doing that scaling though, and then we should have the capacity to double the number of builds.

[syphar]

IMO even after having additional build capacity / scaling, #343 would still be a good idea before adding this additional build (or storing the json output additionally, for that matter :) )

[jyn514]

@syphar let's not block anything on #343, I don't have the energy to make an rfc for that. We can limit the private docs to a single target if that makes you happier.

[jyn514]

@Nemo157 I suspect 99% of our build issues come from constantly rebuilding dependencies, and if we reused the build cache for the public and private doc builds, it would add hardly any time. But we can wait for #1757 if that would make you comfortable.

[Mingun]

I think, it would more preferrable to have a switch, like a theme switch, to show the all documentation or only the public one. This of course would require such support from rustdoc.

[ojeda]

@Mingun Agreed, this would be nice to have in rustdoc and not just docs.rs (potentially also for hidden ones, not just private).

In general, switches like that (or collapsed-by-default elements) in rustdoc may be useful in other cases too, e.g. I suggested it as a solution for the fallible allocation APIs (or other uncommon/advanced APIs that most users may not need/want).