compiletest: show the difference between the normalized output and the actual output for lines which didn't match #133733

jyn514 · 2024-12-02T05:40:15Z

example output:

failures:

---- [ui] tests/ui/layout/enum.rs stdout ----
diff of stderr:

-	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN }
+	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN }
2	  --> $DIR/enum.rs:9:1
3	   |
4	LL | enum UninhabitedVariantAlign {

Note: some mismatched output was normalized before being compared
-	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) }
-	  --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1
+	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN }

rustbot · 2024-12-02T05:40:26Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

jyn514 · 2024-12-02T05:41:24Z

let's just r? jieyouxu since you have context from discord already

jyn514 · 2024-12-02T05:45:08Z

it takes up less than a tenth of the total stderr output

i am a little unsure about this heuristic, i worry that people will still want to see some of the diff to get an idea of what the errors are?

jyn514 · 2024-12-02T05:47:20Z

oh, also i am going to change this to unconditionally print the full stdout if you pass --verbose

jieyouxu

Thanks for the PR. Some general feedback:

We shouldn't hide the rustc command by default because very often contributors will need to look at the command to determine the exact invocation used which can also be used to repro outside of the test framework. I asked about this previously over at https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Vibe.20check.20on.20suppressing.20failed.20test.20command.20invocation, see PR compiletest failing tests aren't that noisy by default #131182 and summary. Maybe a config toggle (extra work impl wise). Note that --verbose will also dump a bunch of other things IIRC.
~~I'm not sure if we want to show only one of the diffs, otherwise a contributor will have to rerun this multiple times instead of getting the full list.~~ EDIT: probably misinterpreted
I think hiding the full stderr is okay outside of CI, but I'll make a poll about this in compiler zulip channel. https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Vibe.20check.20on.20hiding.20full.20test.20stderr.20outside.20of.20CI

jyn514 · 2024-12-03T02:16:58Z

I'm not sure if we want to show only one of the diffs, otherwise a contributor will have to rerun this multiple times instead of getting the full list.

i don't understand what you mean by this. what are the other diffs you think are being hidden?

jyn514 · 2024-12-03T02:24:27Z

i have changed this to do nothing except show the diff between the actual stderr and the normalized output.

jieyouxu · 2024-12-03T03:13:51Z

i don't understand what you mean by this. what are the other diffs you think are being hidden?

I think I misinterpreted when you said

Outside of CI, only show one of the diff and the full stderr

jieyouxu

Thanks, you can r=me after PR CI is green.

jieyouxu · 2024-12-03T03:23:14Z

@bors delegate+

bors · 2024-12-03T03:23:18Z

✌️ @jyn514, you can now approve this pull request!

If @jieyouxu told you to "r=me" after making some further change, please make that change, then do @bors r=@jieyouxu

bors · 2024-12-04T10:55:13Z

☔ The latest upstream changes (presumably #133841) made this pull request unmergeable. Please resolve the merge conflicts.

jyn514 · 2024-12-05T02:32:00Z

@bors r=jieyouxu

jieyouxu · 2024-12-07T02:12:52Z

@bors r- (needs a rebase)

…e actual output for lines which didn't match example output: ``` failures: ---- [ui] tests/ui/layout/enum.rs stdout ---- diff of stderr: - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN } + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } 2 --> $DIR/enum.rs:9:1 3 | 4 LL | enum UninhabitedVariantAlign { Note: some mismatched output was normalized before being compared - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) } - --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1 + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } ```

jieyouxu · 2024-12-07T02:45:43Z

Rebased due to one single eprintln vs println difference, no functional changes. Retested locally e.g.

Note: some mismatched output was normalized before being compared
-       error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
-         --> /home/joe/repos/rust/tests/ui/alias-uninit-value.rs:8:12
+       error[E0277]: the meow values of type `[u8]` cannot be known at compilation time
+          = help: unsized fn params are gated as an unstable feature

jieyouxu · 2024-12-07T02:46:28Z

@bors r+ rollup (trivial rebase, no functional changes)

bors · 2024-12-07T02:46:30Z

📌 Commit 8aacd1c has been approved by jieyouxu

It is now in the queue for this repository.

compiletest: show the difference between the normalized output and the actual output for lines which didn't match example output: ``` failures: ---- [ui] tests/ui/layout/enum.rs stdout ---- diff of stderr: - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN } + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } 2 --> $DIR/enum.rs:9:1 3 | 4 LL | enum UninhabitedVariantAlign { Note: some mismatched output was normalized before being compared - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) } - --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1 + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } ```

…kingjubilee Rollup of 11 pull requests Successful merges: - rust-lang#131669 (lint: change help for pointers to dyn types in FFI) - rust-lang#133265 (Add a range argument to vec.extract_if) - rust-lang#133733 ( compiletest: show the difference between the normalized output and the actual output for lines which didn't match) - rust-lang#133955 (Pass the arch rather than full target name to windows_registry::find_tool) - rust-lang#133967 ([AIX] Pass -bnoipath when adding rust upstream dynamic crates) - rust-lang#133976 (Removed Unnecessary Spaces From RELEASES.md) - rust-lang#133980 ([AIX] Remove option "-n" from AIX "ln" command) - rust-lang#133987 (Define acronym for thread local storage) - rust-lang#133992 (Actually walk into lifetimes and attrs in `EarlyContextAndPass`) - rust-lang#133993 (Fix: typo in E0751 error explanation) - rust-lang#133996 (Move most tests for `-l` and `#[link(..)]` into `tests/ui/link-native-libs`) r? `@ghost` `@rustbot` modify labels: rollup

compiletest: show the difference between the normalized output and the actual output for lines which didn't match example output: ``` failures: ---- [ui] tests/ui/layout/enum.rs stdout ---- diff of stderr: - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN } + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } 2 --> $DIR/enum.rs:9:1 3 | 4 LL | enum UninhabitedVariantAlign { Note: some mismatched output was normalized before being compared - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) } - --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1 + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } ```

…kingjubilee Rollup of 10 pull requests Successful merges: - rust-lang#133522 (Don't suggest restricting bound with unstable traits on stable and mention it's unstable on nightly) - rust-lang#133733 ( compiletest: show the difference between the normalized output and the actual output for lines which didn't match) - rust-lang#133861 (Add allocate_bytes and refactor allocate_str in InterpCx for raw byte…) - rust-lang#133967 ([AIX] Pass -bnoipath when adding rust upstream dynamic crates) - rust-lang#133976 (Removed Unnecessary Spaces From RELEASES.md) - rust-lang#133980 ([AIX] Remove option "-n" from AIX "ln" command) - rust-lang#133987 (Define acronym for thread local storage) - rust-lang#133992 (Actually walk into lifetimes and attrs in `EarlyContextAndPass`) - rust-lang#133993 (Fix: typo in E0751 error explanation) - rust-lang#133996 (Move most tests for `-l` and `#[link(..)]` into `tests/ui/link-native-libs`) r? `@ghost` `@rustbot` modify labels: rollup

compiletest: show the difference between the normalized output and the actual output for lines which didn't match example output: ``` failures: ---- [ui] tests/ui/layout/enum.rs stdout ---- diff of stderr: - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN } + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } 2 --> $DIR/enum.rs:9:1 3 | 4 LL | enum UninhabitedVariantAlign { Note: some mismatched output was normalized before being compared - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) } - --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1 + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } ```

[TEST] Rollup of 5 pull requests Successful merges: - rust-lang#133733 ( compiletest: show the difference between the normalized output and the actual output for lines which didn't match) - rust-lang#133861 (Add allocate_bytes and refactor allocate_str in InterpCx for raw byte…) - rust-lang#133967 ([AIX] Pass -bnoipath when adding rust upstream dynamic crates) - rust-lang#133976 (Removed Unnecessary Spaces From RELEASES.md) - rust-lang#133980 ([AIX] Remove option "-n" from AIX "ln" command) r? `@ghost` `@rustbot` modify labels: rollup try-job: i686-mingw

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#133424 (Parse guard patterns) - rust-lang#133733 ( compiletest: show the difference between the normalized output and the actual output for lines which didn't match) - rust-lang#133993 (Fix: typo in E0751 error explanation) - rust-lang#134013 (Adds new intrinsic declaration) - rust-lang#134020 (Remove unnecessary `int_type_width_signed` function) - rust-lang#134024 (Advent of `tests/ui` (misc cleanups and improvements) [2/N]) - rust-lang#134038 (deps: Update psm) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#133733 - jyn514:compiletest-diffs, r=jieyouxu compiletest: show the difference between the normalized output and the actual output for lines which didn't match example output: ``` failures: ---- [ui] tests/ui/layout/enum.rs stdout ---- diff of stderr: - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN } + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } 2 --> $DIR/enum.rs:9:1 3 | 4 LL | enum UninhabitedVariantAlign { Note: some mismatched output was normalized before being compared - error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) } - --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1 + error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN } ```

rustbot assigned onur-ozkan Dec 2, 2024

rustbot assigned jieyouxu and unassigned onur-ozkan Dec 2, 2024

jyn514 force-pushed the compiletest-diffs branch 2 times, most recently from 4682a01 to 50fd83c Compare December 2, 2024 05:58

jyn514 mentioned this pull request Dec 2, 2024

compiletest should note when test output is normalized #125537

Open

jieyouxu reviewed Dec 2, 2024

View reviewed changes

jyn514 force-pushed the compiletest-diffs branch from 50fd83c to fa85f12 Compare December 3, 2024 02:24

jyn514 changed the title ~~Make compiletest diffs more useful~~ compiletest: show the difference between the normalized output and th… Dec 3, 2024

jyn514 changed the title ~~compiletest: show the difference between the normalized output and th…~~ compiletest: show the difference between the normalized output and the actual output for lines which didn't match Dec 3, 2024

This comment has been minimized.

Sign in to view

jieyouxu approved these changes Dec 3, 2024

View reviewed changes

jieyouxu added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 4, 2024

jyn514 force-pushed the compiletest-diffs branch from fa85f12 to 4ba7019 Compare December 5, 2024 01:11

fmease mentioned this pull request Dec 5, 2024

Rollup of 10 pull requests #133894

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 7, 2024

jieyouxu force-pushed the compiletest-diffs branch from 4ba7019 to 8aacd1c Compare December 7, 2024 02:44

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 7, 2024

workingjubilee mentioned this pull request Dec 7, 2024

Rollup of 11 pull requests #134002

Closed

workingjubilee mentioned this pull request Dec 8, 2024

Rollup of 10 pull requests #134019

Closed

workingjubilee mentioned this pull request Dec 8, 2024

[TEST] Rollup of 5 pull requests #134025

Draft

matthiaskrgr mentioned this pull request Dec 8, 2024

Rollup of 7 pull requests #134039

Merged

bors merged commit c05e7bd into rust-lang:master Dec 9, 2024
6 checks passed

rustbot added this to the 1.85.0 milestone Dec 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compiletest: show the difference between the normalized output and the actual output for lines which didn't match #133733

compiletest: show the difference between the normalized output and the actual output for lines which didn't match #133733

jyn514 commented Dec 2, 2024 •

edited

Loading

rustbot commented Dec 2, 2024

jyn514 commented Dec 2, 2024

jyn514 commented Dec 2, 2024

jyn514 commented Dec 2, 2024

jieyouxu left a comment •

edited

Loading

jyn514 commented Dec 3, 2024

jyn514 commented Dec 3, 2024 •

edited

Loading

This comment has been minimized.

jieyouxu commented Dec 3, 2024 •

edited

Loading

jieyouxu left a comment

jieyouxu commented Dec 3, 2024

bors commented Dec 3, 2024

bors commented Dec 4, 2024

jyn514 commented Dec 5, 2024

jieyouxu commented Dec 7, 2024

jieyouxu commented Dec 7, 2024 •

edited

Loading

jieyouxu commented Dec 7, 2024

bors commented Dec 7, 2024

compiletest: show the difference between the normalized output and the actual output for lines which didn't match #133733

compiletest: show the difference between the normalized output and the actual output for lines which didn't match #133733

Conversation

jyn514 commented Dec 2, 2024 • edited Loading

rustbot commented Dec 2, 2024

jyn514 commented Dec 2, 2024

jyn514 commented Dec 2, 2024

jyn514 commented Dec 2, 2024

jieyouxu left a comment • edited Loading

Choose a reason for hiding this comment

jyn514 commented Dec 3, 2024

jyn514 commented Dec 3, 2024 • edited Loading

This comment has been minimized.

jieyouxu commented Dec 3, 2024 • edited Loading

jieyouxu left a comment

Choose a reason for hiding this comment

jieyouxu commented Dec 3, 2024

bors commented Dec 3, 2024

bors commented Dec 4, 2024

jyn514 commented Dec 5, 2024

jieyouxu commented Dec 7, 2024

jieyouxu commented Dec 7, 2024 • edited Loading

jieyouxu commented Dec 7, 2024

bors commented Dec 7, 2024

jyn514 commented Dec 2, 2024 •

edited

Loading

jieyouxu left a comment •

edited

Loading

jyn514 commented Dec 3, 2024 •

edited

Loading

jieyouxu commented Dec 3, 2024 •

edited

Loading

jieyouxu commented Dec 7, 2024 •

edited

Loading