Update 0000-rustdoc-scrape-examples.md

text/0000-rustdoc-scrape-examples.md

@@ -81,15 +81,23 @@ I have implemented a prototype of the `scrape-examples` feature as modifications
 * rustdoc: https://github.com/willcrichton/rust/compare/master...willcrichton:example-analyzer?expand=1
 * cargo: https://github.com/willcrichton/cargo/compare/master...willcrichton:example-analyzer?expand=1
 
-At a high level, the feature uses the following flow.
+The feature uses the following high-level flow, with some added technical details as necessary.
+
 1. The user gives `--scrape-examples` as an argument to `cargo doc`.
 2. Cargo runs the equivalent of `cargo build --examples` ([source](https://github.com/willcrichton/cargo/blob/fd25a0301314a9eba6beb5239891fc5902a9a9a9/src/cargo/ops/cargo_compile.rs#L618-L631)).
-3. Cargo generates build flags for each example using the same functionality as `cargo test --doc` ([source](https://github.com/willcrichton/cargo/blob/fd25a0301314a9eba6beb5239891fc5902a9a9a9/src/cargo/ops/cargo_compile.rs#L633-L646)).
-4. Cargo identifies a remote repository URL for linking to the examples ([source](https://github.com/willcrichton/cargo/blob/fd25a0301314a9eba6beb5239891fc5902a9a9a9/src/cargo/ops/cargo_compile.rs#L594-L608)).
-5. Cargo invokes rustdoc with added flags: `--repository-url https://github.com/... --scrape-examples "rustc examples/foo.rs --extern ..."`
-6. Rustdoc iterates through each example and uses a visitor to identify spans of calls to functions in the crate being documented ([source](https://github.com/willcrichton/rust/blob/2653c671a4ae89070fdf00f9e149486146e7fc18/src/librustdoc/scrape_examples.rs)).
-7. Rustdoc adds the scraped examples to the documentation for each function ([source](https://github.com/willcrichton/rust/blob/2653c671a4ae89070fdf00f9e149486146e7fc18/src/librustdoc/html/render/mod.rs#L2394-L2471)).
-8. Rustdoc's Javascript adds interactivity to the examples when loaded ([source](https://github.com/willcrichton/rust/blob/2653c671a4ae89070fdf00f9e149486146e7fc18/src/librustdoc/html/static/main.js#L1415-L1599)).
+    *  Specifically, for each unit being documented, it copies the Config and CliFeatures from the input CompileOpts. Then it sets the CompileFilter to only match examples.    
+4. Cargo generates build flags for each example. ([source](https://github.com/willcrichton/cargo/blob/fd25a0301314a9eba6beb5239891fc5902a9a9a9/src/cargo/ops/cargo_compile.rs#L633-L646)).
+    * This is implemented by repurposing the `Doctest` target, which also is used to generate build flags to pass to rustdoc.
+6. Cargo identifies a remote repository URL for linking to the examples ([source](https://github.com/willcrichton/cargo/blob/fd25a0301314a9eba6beb5239891fc5902a9a9a9/src/cargo/ops/cargo_compile.rs#L594-L608)).
+    * Currently this is done by retrieving `package.repository` from the manifest and casing on the domain name. If examples were packaged with rustdoc like other source files, then this could instead link to the generated `src` directory.
+7. Cargo invokes rustdoc with added flags: `--repository-url https://github.com/... --scrape-examples "rustc examples/foo.rs --extern ..."`
+9. Rustdoc iterates through each example and uses a visitor to identify spans of calls to functions in the crate being documented ([source](https://github.com/willcrichton/rust/blob/2653c671a4ae89070fdf00f9e149486146e7fc18/src/librustdoc/scrape_examples.rs)).
+    * This means that rustc is invoked multiple times within a single process before the core of rustdoc is actually executed. Care will be needed to avoid issues with global state like the string interner.
+11. Rustdoc adds the scraped examples to the documentation for each function ([source](https://github.com/willcrichton/rust/blob/2653c671a4ae89070fdf00f9e149486146e7fc18/src/librustdoc/html/render/mod.rs#L2394-L2471)).
+12. Rustdoc's Javascript adds interactivity to the examples when loaded ([source](https://github.com/willcrichton/rust/blob/2653c671a4ae89070fdf00f9e149486146e7fc18/src/librustdoc/html/static/main.js#L1415-L1599)).
+    * Most of the logic here is to extend the code viewer with additional features like toggling between snippet / full file, navigating between call sites, and highlighting code in-situ.
+
+The primary use case for this will be on docs.rs. My expectation is that docs.rs would use the `--scrape-examples` flag, and all docs hosted there would have the scraped examples.
 
 # Drawbacks
 [drawbacks]: #drawbacks
@@ -121,6 +129,7 @@ I have never seen a documentation generator with this exact feature before. Ther
 2. **Tooling integration:** Are there better ways to accomplish the tooling sub-tasks? Specifically:
     * Is there a robust way of generating links to examples based on the Cargo.toml `package.repository` field, especially that generalizes across choice of VCS? Is there a way to reliably get the current commit so as to generate stable links?
     * Is invoking rustc on each example within rustdoc the best way to analyze the examples for call sites? In my [original prototype](https://github.com/willcrichton/example-analyzer), I wrote a standalone tool that output JSON which was then read in by Rustdoc. One benefit of this approach is that Rustdoc could then integrate with any tool that analyzes call sites. But the downside is requiring yet another tool to be in-tree.
+    * What is the best way to handle Cargo workspaces? For example, some workspaces like [wasmtime](https://github.com/bytecodealliance/wasmtime) have a single examples directory at the root with many crates in a `crates/` subfolder. However, under my current strategy for finding examples, they would only be scraped during documentation of the root crate, not the other crates in the workspace.
 
 # Future possibilities
 [future-possibilities]: #future-possibilities