rustdoc: include external files in documentation (RFC 1990) #44781

QuietMisdreavus · 2017-09-23T01:59:57Z

Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR)

cc #44732

Also fixes #42760, because the prep work for the error reporting made it easy to fix that at the same time.

rust-highfive · 2017-09-23T02:00:09Z

r? @GuillaumeGomez

(rust_highfive has picked a reviewer for you, use r? to override)

QuietMisdreavus · 2017-09-23T03:13:21Z

Travis failure is because my test for 42760 failed -_- Back to the drawing board...

QuietMisdreavus · 2017-09-23T04:04:55Z

update: apparently is_sugared_doc is false for doc comments when you get them in with_desugared_doc. The issue was not fixed, because there's no way for it to see the difference between a doc comment and a raw doc attribute where it's extracting them. >:(

QuietMisdreavus · 2017-09-23T17:08:34Z

So i wound up modifying syntax::ast::Attribute::with_desugared_doc to set is_sugared_doc on the attribute when it calls the given closure. I'm not sure if this is entirely valid w.r.t. the original intent of that method, but rustdoc was the only in-tree user of that method that i found, so it may work out anyway. It fixed the issue in the test rendering i had, so i'll let travis see if i messed anything else up.

QuietMisdreavus · 2017-09-23T18:29:02Z

Travis was green, so i'm declaring this Ready To Be Looked At. I updated one test to make sure combining included docs and inline docs worked and squashed the commits. The libsyntax change is still in its own commit, though, since it was only included to make the fix for #42760 work properly, and i'm still not sure whether it's the totally right thing to do.

GuillaumeGomez · 2017-09-23T22:36:10Z

src/librustdoc/clean/mod.rs

+                            match Attributes::load_include(include_path, &filename) {
+                                Ok(frag) => {
+                                    let line = doc_line;
+                                    doc_line += frag.len();


Oh so you decided to put in the "real" line number instead of the one from the original file?

Yeah. My thought was that if you're trying to trace a line number from the total markdown blob, then you could use that to get the doc fragment, since each attribute is joined with a line break anyway.

GuillaumeGomez · 2017-09-23T22:37:37Z

It'd be nice to have some outputs (or even tests?) for the case of failing doc codes.

QuietMisdreavus · 2017-09-23T22:39:28Z

It'd be nice to have some outputs (or even tests?) for the case of failing doc codes.

That's what i was forgetting! Give me a moment, i can make a quick run.

QuietMisdreavus · 2017-09-23T23:52:26Z

With the following files:

lib.rs:

#![feature(external_doc)]

/// hey yo check out these docs, i can splice them in from another file
///
/// ```rust
/// panic!("oh no");
/// ```
///
#[doc(include = "some-file.md")]
pub struct SomeStruct;

/// Item docs.
///
#[doc="Hello there!"]
///
/// # Example
///
/// ```rust
/// // some code here
/// panic!("oh no");
/// ```
pub struct OtherStruct;

#[doc(include = "bad-docs.md")]
#[doc(include = "no-docs.md")]
pub struct AllTheStruct;

some-file.md:

hey check out this included file

this is so sick, i can just paste this anywhere

bad-docs.md:

# some bad docs

```rust
panic!("oh no");
```

(no-docs.md deliberately missing)

Running cargo test gives the following output:

This is with the latest force push. (I neglected to ask rustdoctest to combine doc fragments, so that's fixed now.)

EDIT: The current line numbering refers to the line within the final markdown, not the line in the source file. So i guess to properly translate between this line number and the source line, you need to store both the "markdown line" and the "source line". This could get hairy to properly account for multiple #[doc] attributes written on the same line (rocket has a macro that does this), but i guess if you check the spans while loading, it may work out.

GuillaumeGomez · 2017-09-24T12:07:22Z

And a column as well for the "real" line number (so a Span?). If you want, I can work on that part (the test one).

QuietMisdreavus · 2017-09-28T13:52:02Z

@GuillaumeGomez I added a Span to each DocFragment and made sure that they get properly combined in collapse-docs, does this seem all right to you?

GuillaumeGomez · 2017-09-28T15:41:35Z

Please post a test failure output. :)

QuietMisdreavus · 2017-09-28T16:03:28Z

I haven't touched the failure reporting code, so it'll look the same as in #44781 (comment). Do you want me to make that part of this PR, or would you like to merge this as-is and take it up yourself afterward?

GuillaumeGomez · 2017-09-29T09:41:46Z

I don't mind merging as is. Just tell me what you prefer to do.

QuietMisdreavus · 2017-09-29T14:13:04Z

If i remember right, we discussed fixing up the error reporting in a separate PR, so that's what i was assuming we would do here (and my preference, just so we can get this in >_>). If you want to wrap it all into one PR, just tell me what code is likely to need the span information and i can take a shot at it.

ollie27 · 2017-09-29T18:00:19Z

This doesn't work cross-crate. If you re-export an item that uses #[doc(include)] then it will look for the file relative to the current crate not the crate that actually contains the #[doc(include)].

QuietMisdreavus · 2017-10-04T20:43:33Z

@ollie27 Oh yikes, that's a good point. On the other hand, i don't really see a good way to reliably do it cross-crate without asking rustc to load the file in. In the RFC discussion we talked about making it an alias for #[doc=include_str!("file.md")] which would do exactly that, but would require work within the compiler to make sure certain macros could be used within attributes and expand correctly. (That's also outside my expertise. >_>)

SimonSapin · 2017-10-05T08:58:00Z

Having a special case for doc include in rustc might be easier than supporting arbitrary macros in arbitrary attributes. (Or maybe not, I don’t know.)

bors · 2017-10-05T16:57:09Z

☔ The latest upstream changes (presumably #45046) made this pull request unmergeable. Please resolve the merge conflicts.

aidanhs · 2017-10-12T15:32:12Z

@QuietMisdreavus I've set this back as with author for the merge conflicts. After that, I'm also not clear after that if you're going to do more work or it's going back to review by @GuillaumeGomez?

QuietMisdreavus · 2017-10-12T16:07:21Z

@aidanhs I'm actually not sure either. I'm concerned it may need to be scrapped entirely in favor of the #[doc=include_str!()] method alluded to above. I'm not entirely sure how to go about that, though - it would require work on the compiler frontend to extend #34981 to include (at least) some builtin macros as an allowed expression in attributes, then also adding #[doc(include="file.md")] as sugar for that. Should we pull in compiler-team to take a look?

GuillaumeGomez · 2017-10-13T13:40:34Z

Won't kill to add them in the loop. :)

cc @rust-lang/compiler

alexcrichton · 2017-10-19T14:45:27Z

triage ping for @QuietMisdreavus, do you know what the next steps here are?

GuillaumeGomez · 2017-11-21T20:01:48Z

src/librustdoc/html/render.rs

-    if let Some(s) = item.doc_value() {
-        render_markdown(w, s, item.source.clone(), cx.render_type, prefix, &cx.shared)?;
+    if let Some(s) = cx.shared.maybe_collapsed_doc_value(item) {
+        trace!("Doc block: =====\n{}\n=====", s);


Shouldn't debug! be used instead? (Don't really know the good macro we're supposed to use in such cases...)

Probably? I'm not that solid on what counts as what either, but i'll go ahead and switch it over.

GuillaumeGomez · 2017-11-21T20:02:26Z

src/librustdoc/passes/collapse_docs.rs

+                match curr_frag {
+                    DocFragment::SugaredDoc(_, _, ref mut doc_string)
+                        | DocFragment::RawDoc(_, _, ref mut doc_string) => {
+                            //add a newline for extra padding between segments


Whitespace after // please.

src/libsyntax/ext/expand.rs

GuillaumeGomez · 2017-11-21T20:06:48Z

src/libsyntax/ext/expand.rs

+                        let err_count = self.cx.parse_sess.span_diagnostic.err_count();
+                        self.check_attribute(&at);
+                        if self.cx.parse_sess.span_diagnostic.err_count() > err_count {
+                            //avoid loading the file if they haven't enabled the feature


GuillaumeGomez · 2017-11-21T20:08:12Z

All good for me from code reading. Just a few nits and it's good to go imo.

Partial implementation of rust-lang/rfcs#1990 (needs error reporting work) cc rust-lang#44732

QuietMisdreavus · 2017-11-21T21:47:46Z

I've force-pushed to address the review comments.

GuillaumeGomez · 2017-11-21T22:16:49Z

Then it's all good for me. r=me once CI is green.

QuietMisdreavus · 2017-11-21T23:10:56Z

@bors r=GuillaumeGomez

bors · 2017-11-21T23:10:57Z

📌 Commit 52ee203 has been approved by GuillaumeGomez

bors · 2017-11-22T01:54:43Z

⌛ Testing commit 52ee203 with merge 0893ecc8955191f702513ff3a19d420f9650fae5...

bors · 2017-11-22T03:00:14Z

💔 Test failed - status-travis

QuietMisdreavus · 2017-11-22T04:14:25Z

Failure... doesn't look legit? I'm not caught up on the known spurious failures.

Installing deploy dependencies
ERROR:  Could not find a valid gem 'aws-sdk-resources' (= 2.10.88) in any repository
ERROR:  Possible alternatives: aws-sdk-resources
/home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- aws-sdk (LoadError)
	from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:85:in `requires'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:6:in `<class:S3>'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:5:in `<class:Provider>'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:4:in `<module:DPL>'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider/s3.rb:3:in `<top (required)>'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:59:in `const_get'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:59:in `block in new'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/cli.rb:41:in `fold'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/provider.rb:56:in `new'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/cli.rb:31:in `run'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/lib/dpl/cli.rb:7:in `run'
	from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.8.43/bin/dpl:5:in `<top (required)>'
	from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `load'
	from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `<main>'

kennytm · 2017-11-22T08:47:57Z

@bors retry #44159

Looks like that previous attempt to fix it doesn't work.

bors · 2017-11-22T09:58:15Z

⌛ Testing commit 52ee203 with merge 3755fe9...

rustdoc: include external files in documentation (RFC 1990) Part of rust-lang/rfcs#1990 (needs work on the error reporting, which i'm deferring to after this initial PR) cc #44732 Also fixes #42760, because the prep work for the error reporting made it easy to fix that at the same time.

bors · 2017-11-22T12:34:50Z

☀️ Test successful - status-appveyor, status-travis
Approved by: GuillaumeGomez
Pushing 3755fe9 to master...

rust-highfive assigned GuillaumeGomez Sep 23, 2017

estebank added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 23, 2017

QuietMisdreavus force-pushed the doc-include branch from 6f188ec to b25e8bd Compare September 23, 2017 18:23

QuietMisdreavus changed the title ~~[WIP] rustdoc: include external files in documentation (RFC 1990)~~ rustdoc: include external files in documentation (RFC 1990) Sep 23, 2017

GuillaumeGomez reviewed Sep 23, 2017

View reviewed changes

QuietMisdreavus force-pushed the doc-include branch from b25e8bd to 7e482de Compare September 23, 2017 23:51

aidanhs added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 12, 2017

GuillaumeGomez reviewed Nov 21, 2017

View reviewed changes

src/libsyntax/ext/expand.rs Show resolved Hide resolved

GuillaumeGomez reviewed Nov 21, 2017

View reviewed changes

kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 21, 2017

QuietMisdreavus added 2 commits November 21, 2017 15:46

allow loading external files in documentation

f9f3611

Partial implementation of rust-lang/rfcs#1990 (needs error reporting work) cc rust-lang#44732

make with_unsugared_doc preserve is_sugared_doc

52ee203

QuietMisdreavus force-pushed the doc-include branch from abc9aca to 52ee203 Compare November 21, 2017 21:47

kennytm added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 22, 2017

bors merged commit 52ee203 into rust-lang:master Nov 22, 2017

Eh2406 mentioned this pull request Nov 30, 2017

What needs to be done? aldanor/hdf5-rust#17

Closed

QuietMisdreavus deleted the doc-include branch December 6, 2017 20:16

ozkriff mentioned this pull request May 19, 2018

Are badges in ggez's top level doc comment really required? ggez/ggez#379

Closed

Nemo157 mentioned this pull request Apr 7, 2020

doc attribute removes whitespace #70732

Closed

jyn514 added the T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. label Jan 9, 2021

rustdoc: include external files in documentation (RFC 1990) #44781

rustdoc: include external files in documentation (RFC 1990) #44781

Conversation

QuietMisdreavus commented Sep 23, 2017

rust-highfive commented Sep 23, 2017

QuietMisdreavus commented Sep 23, 2017

QuietMisdreavus commented Sep 23, 2017

QuietMisdreavus commented Sep 23, 2017

QuietMisdreavus commented Sep 23, 2017

GuillaumeGomez Sep 23, 2017

Choose a reason for hiding this comment

QuietMisdreavus Sep 23, 2017

Choose a reason for hiding this comment

GuillaumeGomez commented Sep 23, 2017

QuietMisdreavus commented Sep 23, 2017

QuietMisdreavus commented Sep 23, 2017 • edited Loading

GuillaumeGomez commented Sep 24, 2017

QuietMisdreavus commented Sep 28, 2017

GuillaumeGomez commented Sep 28, 2017

QuietMisdreavus commented Sep 28, 2017

GuillaumeGomez commented Sep 29, 2017

QuietMisdreavus commented Sep 29, 2017

ollie27 commented Sep 29, 2017

QuietMisdreavus commented Oct 4, 2017

SimonSapin commented Oct 5, 2017

bors commented Oct 5, 2017

aidanhs commented Oct 12, 2017

QuietMisdreavus commented Oct 12, 2017

GuillaumeGomez commented Oct 13, 2017

alexcrichton commented Oct 19, 2017

GuillaumeGomez Nov 21, 2017

Choose a reason for hiding this comment

QuietMisdreavus Nov 21, 2017

Choose a reason for hiding this comment

GuillaumeGomez Nov 21, 2017

Choose a reason for hiding this comment

GuillaumeGomez Nov 21, 2017

Choose a reason for hiding this comment

GuillaumeGomez commented Nov 21, 2017

QuietMisdreavus commented Nov 21, 2017

GuillaumeGomez commented Nov 21, 2017

QuietMisdreavus commented Nov 21, 2017

bors commented Nov 21, 2017

bors commented Nov 22, 2017

bors commented Nov 22, 2017

QuietMisdreavus commented Nov 22, 2017 • edited Loading

kennytm commented Nov 22, 2017

bors commented Nov 22, 2017

bors commented Nov 22, 2017

QuietMisdreavus commented Sep 23, 2017 •

edited

Loading

QuietMisdreavus commented Nov 22, 2017 •

edited

Loading