Add JsonDocCk Tool for rustdoc-json #81063

CraftSpider · 2021-01-16T01:38:53Z

Implements a new test system for rustdoc JSON output, jsondocck. Modeled after htmldocck, this tool reads directives in the test file and checks them against the output. These directives use JSONPath, a pair to XPath for json. This obsoletes the old strict subset tool, allowing both finer-grained control of what is tested and better errors on failure.

Not sure on the changes to Cargo.lock, I can back that out if needed.

r? @jyn514

rust-log-analyzer · 2021-01-16T01:52:16Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
test runtest::tests::normalize_platform_differences ... ok

failures:

---- header::tests::aux_build stdout ----
thread 'header::tests::aux_build' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19

---- header::tests::cross_compile stdout ----
---- header::tests::cross_compile stdout ----
thread 'header::tests::cross_compile' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::no_system_llvm stdout ----
---- header::tests::no_system_llvm stdout ----
thread 'header::tests::no_system_llvm' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::only_target stdout ----
---- header::tests::only_target stdout ----
thread 'header::tests::only_target' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::revisions stdout ----
---- header::tests::revisions stdout ----
thread 'header::tests::revisions' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::llvm_version stdout ----
---- header::tests::llvm_version stdout ----
thread 'header::tests::llvm_version' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::debugger stdout ----
---- header::tests::debugger stdout ----
thread 'header::tests::debugger' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::sanitizers stdout ----
---- header::tests::sanitizers stdout ----
thread 'header::tests::sanitizers' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::ignore_target stdout ----
error: test failed, to rerun pass '--bin compiletest'
error: test failed, to rerun pass '--bin compiletest'
thread 'header::tests::ignore_target' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::should_fail stdout ----
---- header::tests::should_fail stdout ----
thread 'header::tests::should_fail' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::stage stdout ----
---- header::tests::stage stdout ----
thread 'header::tests::stage' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19

failures:
    header::tests::aux_build
    header::tests::cross_compile

rust-log-analyzer · 2021-01-16T02:22:42Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
test runtest::tests::normalize_platform_differences ... ok

failures:

---- header::tests::ignore_target stdout ----
thread 'header::tests::ignore_target' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19

---- header::tests::debugger stdout ----
---- header::tests::debugger stdout ----
thread 'header::tests::debugger' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::only_target stdout ----
---- header::tests::only_target stdout ----
thread 'header::tests::only_target' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::revisions stdout ----
---- header::tests::revisions stdout ----
thread 'header::tests::revisions' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::no_system_llvm stdout ----
---- header::tests::no_system_llvm stdout ----
thread 'header::tests::no_system_llvm' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::cross_compile stdout ----
---- header::tests::cross_compile stdout ----
thread 'header::tests::cross_compile' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::llvm_version stdout ----
---- header::tests::llvm_version stdout ----
thread 'header::tests::llvm_version' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::aux_build stdout ----
---- header::tests::aux_build stdout ----
thread 'header::tests::aux_build' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::stage stdout ----
---- header::tests::stage stdout ----
thread 'header::tests::stage' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::should_fail stdout ----
---- header::tests::should_fail stdout ----
thread 'header::tests::should_fail' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19
---- header::tests::sanitizers stdout ----
---- header::tests::sanitizers stdout ----
thread 'header::tests::sanitizers' panicked at 'OptionMissing("jsondocck-path")', src/tools/compiletest/src/main.rs:154:19

failures:
    header::tests::aux_build
    header::tests::cross_compile

jyn514

This looks great :) Can you add a test that intentionally fails and post the error output?

src/tools/compiletest/src/runtest.rs

src/tools/jsondocck/src/cache.rs

src/tools/jsondocck/src/config.rs

src/tools/jsondocck/src/main.rs

CraftSpider · 2021-01-16T05:12:51Z

This looks great :) Can you add a test that intentionally fails and post the error output?

Do you mean add one to git, or just write one up and post the results?

jyn514 · 2021-01-16T05:13:30Z

Just write one up and post the results.

CraftSpider · 2021-01-16T05:52:41Z

Failure Output

running 8 tests
iiiiiiFF
failures:

---- [rustdoc-json] rustdoc-json\bad.rs stdout ----

error: jsondocck failed!
status: exit code: 1
command: "D:\\Dev\\Git\\rust\\build\\x86_64-pc-windows-msvc\\stage1-tools-bin\\j
sondocck.exe" "--doc-dir" "D:\\Dev\\Git\\rust\\build\\x86_64-pc-windows-msvc\\te
st\\rustdoc-json\\bad" "--template" "D:\\Dev\\Git\\rust\\src/test\\rustdoc-json\
\bad.rs"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
Invalid command: Tried to use the previous path in the first command on line 2
Invalid command: `!@has`, (help: try with `@!has`) on line 3
Invalid command: Incorrect number of arguments to `@has` on line 4
Invalid command: Unrecognized command name `@unknown` on line 5
Invalid command: Third argument to @count must be a valid usize on line 6
Error: "Jsondocck failed for D:\\Dev\\Git\\rust\\src/test\\rustdoc-json\\bad.rs"

------------------------------------------


---- [rustdoc-json] rustdoc-json\bad2.rs stdout ----

error: jsondocck failed!
status: exit code: 1
command: "D:\\Dev\\Git\\rust\\build\\x86_64-pc-windows-msvc\\stage1-tools-bin\\j
sondocck.exe" "--doc-dir" "D:\\Dev\\Git\\rust\\build\\x86_64-pc-windows-msvc\\te
st\\rustdoc-json\\bad2" "--template" "D:\\Dev\\Git\\rust\\src/test\\rustdoc-json
\\bad2.rs"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
Failed check: `@has bad2.json $.not.a.real.path` didn't match when it should on
line 2
Failed check: `@count bad2.json $.index.* 1` didn't match when it should on line
 3
Error: "Jsondocck failed for D:\\Dev\\Git\\rust\\src/test\\rustdoc-json\\bad2.rs
"

------------------------------------------



failures:
    [rustdoc-json] rustdoc-json\bad.rs
    [rustdoc-json] rustdoc-json\bad2.rs

test result: FAILED. 0 passed; 2 failed; 6 ignored; 0 measured; 0 filtered out;
finished in 6.99s

Note that the system won't display syntax errors at the same time as check errors. I could change this, it was just a bit easier to implement this way.

Edit: Note, I didn't yet implement it telling you where exactly the path failed, but I added a FIXME to do it later. The current output is about as informative as htmldocck puts out, so I figured it would be alright to start with this, and add the even better output later.

src/etc/check_missing_items.py

src/bootstrap/builder.rs

src/bootstrap/tool.rs

src/test/rustdoc-json/nested.rs

jyn514 · 2021-01-16T17:27:08Z

src/tools/compiletest/src/main.rs

@@ -196,6 +197,7 @@ pub fn parse_config(args: Vec<String>) -> Config {
    let has_tidy = Command::new("tidy")
        .arg("--version")
        .stdout(Stdio::null())
+        .stderr(Stdio::null())


Does tidy write to stderr on your machine?

Yes, and in fact it should be on everyone's machine. The first 3 lines of tidy/src/main.rs:

let root_path: PathBuf = env::args_os().nth(1).expect("need path to root of repo").into(); let cargo: PathBuf = env::args_os().nth(2).expect("need path to cargo").into(); let output_directory: PathBuf = env::args_os().nth(3).expect("need path to output directory").into();

Thinking about it, this check is actually broken and always false, because --version is entirely ignored and tidy just unconditionally panics. This is another change I can back out here and make into its own PR though, as it is unrelated to the function of the json tests.

@CraftSpider there are two different tidy commands. One is src/tools/tidy, one is https://www.w3.org/People/Raggett/tidy/. This is checking HTML tidy.

Can you revert this change?

Interesting. It was definitely invoking the one I pointed to, I was getting the full panic. I'll revert the change and look into why.

src/tools/jsondocck/src/config.rs

src/tools/jsondocck/src/cache.rs

src/tools/jsondocck/src/main.rs

bors · 2021-01-16T23:52:45Z

☔ The latest upstream changes (presumably #81089) made this pull request unmergeable. Please resolve the merge conflicts.

rust-log-analyzer · 2021-01-19T21:56:12Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
skip untracked path cpu-usage.csv during rustfmt invocations
skip untracked path src/doc/book/ during rustfmt invocations
skip untracked path src/doc/rust-by-example/ during rustfmt invocations
skip untracked path src/llvm-project/ during rustfmt invocations
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/src/tools/jsondocck/src/main.rs"` failed.
Diff in /checkout/src/tools/jsondocck/src/main.rs at line 149:
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
         }
 
 
-        let args = cap.name("args")
-            .map_or(vec![], |m| shlex::split(m.as_str()).unwrap());
+        let args = cap.name("args").map_or(vec![], |m| shlex::split(m.as_str()).unwrap());
 
         if !cmd.validate(&args, commands.len(), lineno) {
             errors = true;
Build completed unsuccessfully in 0:00:22

src/bootstrap/test.rs

jyn514 · 2021-01-20T03:20:15Z

@bors r+

Thanks so much!

bors · 2021-01-20T03:20:17Z

📌 Commit ba6803e has been approved by jyn514

bors · 2021-01-20T04:10:46Z

⌛ Testing commit ba6803e with merge e05409a...

bors · 2021-01-20T07:15:33Z

☀️ Test successful - checks-actions
Approved by: jyn514
Pushing e05409a to master...

aDotInTheVoid · 2021-01-24T18:57:19Z

@rustbot modify labels +A-rustdoc-json

rust-highfive assigned jyn514 Jan 16, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 16, 2021

jyn514 added the T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. label Jan 16, 2021

CraftSpider force-pushed the jsondocck branch from 87d5933 to 9fe5308 Compare January 16, 2021 03:19

jyn514 reviewed Jan 16, 2021

View reviewed changes

CraftSpider mentioned this pull request Jan 16, 2021

String type mismatch intellij-rust/intellij-rust#6687

Open

CraftSpider added 3 commits January 19, 2021 14:24

Add jsondocck tool, and use it for rustdoc JSON

7715656

Address review comments

66a5714

Shift another panic into an exit

a3df483

CraftSpider force-pushed the jsondocck branch from d11c5af to 29b7b02 Compare January 19, 2021 21:44

Address review v2

728ffc8

CraftSpider force-pushed the jsondocck branch from 29b7b02 to 728ffc8 Compare January 19, 2021 22:02

jyn514 reviewed Jan 19, 2021

View reviewed changes

src/bootstrap/test.rs Outdated Show resolved Hide resolved

No longer require unstable for jsondocck, only build it for json tests

ba6803e

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 20, 2021

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 20, 2021

bors merged commit e05409a into rust-lang:master Jan 20, 2021

rustbot added this to the 1.51.0 milestone Jan 20, 2021

rustbot added the A-rustdoc-json Area: Rustdoc JSON backend label Jan 24, 2021

CraftSpider deleted the jsondocck branch February 23, 2022 15:53

aDotInTheVoid added the A-testsuite Area: The testsuite used to check the correctness of rustc label Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add JsonDocCk Tool for rustdoc-json #81063

Add JsonDocCk Tool for rustdoc-json #81063

CraftSpider commented Jan 16, 2021

rust-log-analyzer commented Jan 16, 2021

rust-log-analyzer commented Jan 16, 2021

jyn514 left a comment

CraftSpider commented Jan 16, 2021

jyn514 commented Jan 16, 2021

CraftSpider commented Jan 16, 2021 •

edited

Loading

jyn514 Jan 16, 2021

CraftSpider Jan 16, 2021 •

edited

Loading

jyn514 Jan 16, 2021

CraftSpider Jan 16, 2021

bors commented Jan 16, 2021

rust-log-analyzer commented Jan 19, 2021

jyn514 commented Jan 20, 2021

bors commented Jan 20, 2021

bors commented Jan 20, 2021

bors commented Jan 20, 2021

aDotInTheVoid commented Jan 24, 2021

Add JsonDocCk Tool for rustdoc-json #81063

Add JsonDocCk Tool for rustdoc-json #81063

Conversation

CraftSpider commented Jan 16, 2021

rust-log-analyzer commented Jan 16, 2021

rust-log-analyzer commented Jan 16, 2021

jyn514 left a comment

Choose a reason for hiding this comment

CraftSpider commented Jan 16, 2021

jyn514 commented Jan 16, 2021

CraftSpider commented Jan 16, 2021 • edited Loading

jyn514 Jan 16, 2021

Choose a reason for hiding this comment

CraftSpider Jan 16, 2021 • edited Loading

Choose a reason for hiding this comment

jyn514 Jan 16, 2021

Choose a reason for hiding this comment

CraftSpider Jan 16, 2021

Choose a reason for hiding this comment

bors commented Jan 16, 2021

rust-log-analyzer commented Jan 19, 2021

jyn514 commented Jan 20, 2021

bors commented Jan 20, 2021

bors commented Jan 20, 2021

bors commented Jan 20, 2021

aDotInTheVoid commented Jan 24, 2021

CraftSpider commented Jan 16, 2021 •

edited

Loading

CraftSpider Jan 16, 2021 •

edited

Loading