Detect typos for compiletest test directives #121561

jieyouxu · 2024-02-24T17:32:59Z

Checks directives against a known list of compiletest directives collected during migration from legacy-style compiletest directives. A suggestion for the best matching known directive will be made if an invalid directive is found.

This PR does not attempt to implement checks for Makefile directives because they still have the problem of regular comments and directives sharing the same comment prefix #.

Closes #83551.

rustbot · 2024-02-24T17:33:07Z

r? @onur-ozkan

rustbot has assigned @onur-ozkan.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

onur-ozkan · 2024-02-29T13:40:22Z

src/tools/compiletest/src/util.rs

+// Edit distance taken from rustc's `rustc_span::edit_distance`.
+
+/// Finds the [edit distance] between two strings.


Suggested change

// Edit distance taken from rustc's `rustc_span::edit_distance`.

/// Finds the [edit distance] between two strings.

/// Edit distance taken from rustc's `rustc_span::edit_distance`.

///

/// Finds the [edit distance] between two strings.

Changed it to a top-level comment for the edit_distance module.

onur-ozkan · 2024-02-29T13:41:29Z

tests/ui/type-alias-impl-trait/normalize-hidden-types.rs

@@ -23,7 +23,6 @@ mod typeof_1 {
    use super::*;
    type Opaque = impl Sized;
    fn define() -> Opaque {
-        //[current]~^ ERROR concrete type differs


Q: Why these are removed?

Tidy enforces that if a revision like current is associated with a known-bug directive, then it cannot have any revisioned //~ERRORs. At least that's what I understood from tidy's error message. This tidy check was not previously triggered because of the extra colon after //@ [current]...

onur-ozkan · 2024-02-29T17:08:22Z

src/tools/compiletest/src/header.rs

+// tidy-alphabetical-start
+const KNOWN_DIRECTIVE_NAMES: &[&str] = &[


Shouldn't we put tidy-alphabetical-start inside the list?

Oops, yeah that would make more sense...

onur-ozkan · 2024-02-29T17:35:01Z

src/tools/compiletest/src/util.rs

What are your thoughts on creating a separate module for this implementation? Putting these functions and explanations in the generic module util seems to complicate things a bit, in my opinion.

Yeah, I can move it out to some edit_distance module. This code is like copy-pasta'd (including this) 3 places over rustc/cargo/compiletest, ideally we would share a single crate for this, but alas...

Moved to new edit_distance module.

onur-ozkan · 2024-02-29T17:37:12Z

src/tools/compiletest/src/header.rs

+        } else if let Some((header_revision, original_directive_line)) = line_directive(comment, ln)
+        {
+            // Let Makefiles and non-rs files just get handled in the regular way.
+            if testfile.extension().map(|e| e != "rs").unwrap_or(true) {
+                it(HeaderLine {
+                    line_number,
+                    original_line,
+                    header_revision,
+                    directive: original_directive_line,
+                });
+                continue;
+            }
+
+            let directive_ln = original_directive_line.trim();
+
+            if let Some((directive_name, _)) = directive_ln.split_once([':', ' ']) {
+                if KNOWN_DIRECTIVE_NAMES.contains(&directive_name) {
+                    it(HeaderLine {
+                        line_number,
+                        original_line,
+                        header_revision,
+                        directive: original_directive_line,
+                    });
+                    continue;
+                } else {
+                    *poisoned = true;
+                    eprintln!(
+                        "error: detected unknown compiletest test directive `{}` in {}:{}",
+                        directive_ln,
+                        testfile.display(),
+                        line_number,
+                    );
+
+                    if let Some(suggestion) =
+                        find_best_match_for_name(KNOWN_DIRECTIVE_NAMES, directive_name, None)
+                    {
+                        eprintln!("help: did you mean `{}` instead?", suggestion);
+                    }
+                    return;
+                }
+            } else if KNOWN_DIRECTIVE_NAMES.contains(&directive_ln) {
+                it(HeaderLine {
+                    line_number,
+                    original_line,
+                    header_revision,
+                    directive: original_directive_line,
+                });
+                continue;
+            } else {
+                *poisoned = true;
+                eprintln!(
+                    "error: detected unknown compiletest test directive `{}` in {}:{}",
+                    directive_ln,
+                    testfile.display(),
+                    line_number,
+                );
+
+                if let Some(suggestion) =
+                    find_best_match_for_name(KNOWN_DIRECTIVE_NAMES, directive_ln, None)
+                {
+                    eprintln!("help: did you mean `{}` instead?", suggestion);
+                }
+                return;
+            }


Can we write some unit tests for this logic (or maybe for the whole feature)?

~~or maybe for the whole compiletest~~ But yeah, I'll think about how to test this / make this more testable.

Simplified this logic and added some (integration) tests for iter_header and directives checking.

jieyouxu · 2024-03-07T19:13:28Z

Summary:

Moved edit_distance to its own module
Reworked directive detection logic to make it simpler
Added integration tests for iter_header directive checking
Added missing known directive needs-sanitizer-dataflow
Fixed tests:
- tests/ui/borrowck/two-phase-reservation-sharing-interference.rs: missing @
- tests/ui/type-alias-impl-trait/normalize-hidden-types.rs: extra : after revision name causes tidy check to not fire

jieyouxu · 2024-03-07T21:09:41Z

@rustbot ready

…o-check, r=onur-ozkan Detect typos for compiletest test directives Checks directives against a known list of compiletest directives collected during migration from legacy-style compiletest directives. A suggestion for the best matching known directive will be made if an invalid directive is found. This PR does not attempt to implement checks for Makefile directives because they still have the problem of regular comments and directives sharing the same comment prefix `#`. Closes rust-lang#83551.

…iaskrgr Rollup of 10 pull requests Successful merges: - rust-lang#121561 (Detect typos for compiletest test directives) - rust-lang#121567 (Avoid some interning in bootstrap) - rust-lang#121685 (Fixing shellcheck comments on lvi test script) - rust-lang#121833 (Suggest correct path in include_bytes!) - rust-lang#121860 (Add a tidy check that checks whether the fluent slugs only appear once) - rust-lang#121907 (skip sanity check for non-host targets in `check` builds) - rust-lang#122029 (When displaying multispans, ignore empty lines adjacent to `...`) - rust-lang#122221 (match lowering: define a convenient struct) - rust-lang#122244 (fix: LocalWaker memory leak and some stability attributes) - rust-lang#122251 (Add test to check unused_lifetimes don't duplicate "parameter is never used" error) r? `@ghost` `@rustbot` modify labels: rollup

…o-check, r=onur-ozkan Detect typos for compiletest test directives Checks directives against a known list of compiletest directives collected during migration from legacy-style compiletest directives. A suggestion for the best matching known directive will be made if an invalid directive is found. This PR does not attempt to implement checks for Makefile directives because they still have the problem of regular comments and directives sharing the same comment prefix `#`. Closes rust-lang#83551.

…kingjubilee Rollup of 10 pull requests Successful merges: - rust-lang#112136 (Add std::ffi::c_str module) - rust-lang#113525 (Dynamically size sigaltstk in std) - rust-lang#121561 (Detect typos for compiletest test directives) - rust-lang#121685 (Fixing shellcheck comments on lvi test script) - rust-lang#121833 (Suggest correct path in include_bytes!) - rust-lang#121860 (Add a tidy check that checks whether the fluent slugs only appear once) - rust-lang#122125 (Revert back to Git-for-Windows for MinGW CI builds) - rust-lang#122221 (match lowering: define a convenient struct) - rust-lang#122251 (Add test to check unused_lifetimes don't duplicate "parameter is never used" error) - rust-lang#122264 (add myself to rotation) r? `@ghost` `@rustbot` modify labels: rollup

matthiaskrgr · 2024-03-10T06:53:52Z

@bors r- rollup=iffy

#122265

2024-03-10T06:25:45.0257378Z test [mir-opt] tests/mir-opt/inline/polymorphic_recursion.rs ... ok
2024-03-10T06:25:45.0257864Z 
2024-03-10T06:25:45.0259633Z test result: ok. 274 passed; 0 failed; 11 ignored; 0 measured; 0 filtered out; finished in 30.17s
2024-03-10T06:25:45.0260248Z 
2024-03-10T06:25:45.0262781Z ##[endgroup]
2024-03-10T06:25:45.0265007Z [TIMING] core::build_steps::test::Compiletest { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, target: x86_64-unknown-linux-gnu, mode: "mir-opt", suite: "mir-opt", path: "tests/mir-opt", compare_mode: None } -- 30.313
2024-03-10T06:25:45.0506328Z [TIMING] core::build_steps::test::MirOpt { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, target: x86_64-unknown-linux-gnu } -- 0.024
2024-03-10T06:25:45.0549273Z ##[group]Testing stage1 compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu)
2024-03-10T06:25:45.1679235Z error: detected unknown compiletest test directive `filecheck-flags: --check-prefix=CUSTOM` in /checkout/tests/codegen/meta-filecheck/filecheck-flags.rs:4
2024-03-10T06:25:45.1680772Z errors encountered during EarlyProps parsing: /checkout/tests/codegen/meta-filecheck/filecheck-flags.rs

- Fix invalid directive in `normalize-hidden-types` - Update legacy directive in `two-phase-reservation-sharing-interference`

jieyouxu · 2024-03-10T13:14:24Z

Added missing known filecheck-flags directive.
@rustbot ready

onur-ozkan · 2024-03-10T15:12:34Z

@bors r+

bors · 2024-03-10T15:12:37Z

📌 Commit 64dda8c has been approved by onur-ozkan

It is now in the queue for this repository.

bors · 2024-03-10T16:26:21Z

⌛ Testing commit 64dda8c with merge af69f4c...

bors · 2024-03-10T18:26:38Z

☀️ Test successful - checks-actions
Approved by: onur-ozkan
Pushing af69f4c to master...

rust-timer · 2024-03-10T21:21:26Z

Finished benchmarking commit (af69f4c): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.4%	[2.4%, 2.4%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.3%	[0.9%, 1.6%]	2
Regressions ❌ (secondary)	3.1%	[2.1%, 4.4%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.3%	[0.9%, 1.6%]	2

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.8%	[2.8%, 2.8%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 646.803s -> 647.057s (0.04%)
Artifact size: 309.98 MiB -> 309.92 MiB (-0.02%)

…notriddle rustdoc: fix up old test `tests/rustdoc/line-breaks.rs` had several issues: 1. It used `//`@count`` instead of `// `@count`` (notice the space!) which gets treated as a `ui_test` directive instead of a `htmldocck` one. `compiletest` didn't flag it as an error because it's allowlisted ([rust-lang#121561](rust-lang#121561)) presumably precisely because of this test. And before the compiletest→ui_test migration, these directives must've been ignored, too, because … 2. … the checks themselves no longer work either: The count of ` `s is actually 0 in all 3 cases because – well – we no longer generate any ` `s inside `<pre>`s. Since I don't know how to ``@count`` `\n`s instead of ` `s, I've turned them into ``@matches`.` Btw, I don't know if this test is still desirable or if we have other tests that cover this (I haven't checked). r? rustdoc

Rollup merge of rust-lang#122355 - fmease:rustdoc-fix-up-old-test, r=notriddle rustdoc: fix up old test `tests/rustdoc/line-breaks.rs` had several issues: 1. It used `//`@count`` instead of `// `@count`` (notice the space!) which gets treated as a `ui_test` directive instead of a `htmldocck` one. `compiletest` didn't flag it as an error because it's allowlisted ([rust-lang#121561](rust-lang#121561)) presumably precisely because of this test. And before the compiletest→ui_test migration, these directives must've been ignored, too, because … 2. … the checks themselves no longer work either: The count of ` `s is actually 0 in all 3 cases because – well – we no longer generate any ` `s inside `<pre>`s. Since I don't know how to ``@count`` `\n`s instead of ` `s, I've turned them into ``@matches`.` Btw, I don't know if this test is still desirable or if we have other tests that cover this (I haven't checked). r? rustdoc

jieyouxu marked this pull request as ready for review February 24, 2024 17:33

rustbot assigned onur-ozkan Feb 24, 2024

This comment has been minimized.

Sign in to view

jieyouxu force-pushed the compiletest-directive-typo-check branch from 7ecce91 to f285c6e Compare February 24, 2024 18:21

This comment has been minimized.

Sign in to view

jieyouxu force-pushed the compiletest-directive-typo-check branch from f285c6e to c230627 Compare February 24, 2024 18:56

This comment has been minimized.

Sign in to view

jieyouxu force-pushed the compiletest-directive-typo-check branch from e7e58a0 to 6579112 Compare February 24, 2024 20:32

This comment has been minimized.

Sign in to view

jieyouxu marked this pull request as draft February 24, 2024 21:21

jieyouxu force-pushed the compiletest-directive-typo-check branch from 6579112 to 1e86c63 Compare February 25, 2024 11:39

jieyouxu marked this pull request as ready for review February 25, 2024 12:05

onur-ozkan reviewed Feb 29, 2024

View reviewed changes

onur-ozkan added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 29, 2024

jieyouxu force-pushed the compiletest-directive-typo-check branch from ffde4b6 to 32e713f Compare March 7, 2024 17:19

This comment has been minimized.

Sign in to view

jieyouxu force-pushed the compiletest-directive-typo-check branch from a7ee012 to a816cc5 Compare March 7, 2024 18:38

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 7, 2024

matthiaskrgr mentioned this pull request Mar 9, 2024

Rollup of 10 pull requests #122260

Closed

workingjubilee mentioned this pull request Mar 10, 2024

Rollup of 10 pull requests #122265

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 10, 2024

jieyouxu added 2 commits March 10, 2024 13:13

Error on invalid compiletest directives in Rust test files

5877e7e

Fix invalid compiletest directives in tests

64dda8c

- Fix invalid directive in `normalize-hidden-types` - Update legacy directive in `two-phase-reservation-sharing-interference`

jieyouxu force-pushed the compiletest-directive-typo-check branch from 046c28f to 64dda8c Compare March 10, 2024 13:14

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 10, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 10, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Mar 10, 2024

bors merged commit af69f4c into rust-lang:master Mar 10, 2024
12 checks passed

rustbot added this to the 1.78.0 milestone Mar 10, 2024

jieyouxu deleted the compiletest-directive-typo-check branch March 10, 2024 18:39

fmease mentioned this pull request Mar 12, 2024

rustdoc: fix up old test #122355

Merged

jieyouxu mentioned this pull request May 31, 2024

compiletest: technically potential false positives for filecheck annotations #125825

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect typos for compiletest test directives #121561

Detect typos for compiletest test directives #121561

jieyouxu commented Feb 24, 2024 •

edited

Loading

rustbot commented Feb 24, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

onur-ozkan Feb 29, 2024

jieyouxu Mar 7, 2024

onur-ozkan Feb 29, 2024

jieyouxu Feb 29, 2024 •

edited

Loading

onur-ozkan Feb 29, 2024

jieyouxu Feb 29, 2024

jieyouxu Mar 7, 2024

onur-ozkan Feb 29, 2024

jieyouxu Feb 29, 2024

jieyouxu Mar 7, 2024

onur-ozkan Feb 29, 2024

jieyouxu Feb 29, 2024

jieyouxu Mar 7, 2024

This comment has been minimized.

This comment has been minimized.

jieyouxu commented Mar 7, 2024

jieyouxu commented Mar 7, 2024

matthiaskrgr commented Mar 10, 2024

jieyouxu commented Mar 10, 2024

onur-ozkan commented Mar 10, 2024

bors commented Mar 10, 2024

bors commented Mar 10, 2024

bors commented Mar 10, 2024

rust-timer commented Mar 10, 2024

		// Edit distance taken from rustc's `rustc_span::edit_distance`.

		/// Finds the [edit distance] between two strings.

		// tidy-alphabetical-start
		const KNOWN_DIRECTIVE_NAMES: &[&str] = &[

Detect typos for compiletest test directives #121561

Detect typos for compiletest test directives #121561

Conversation

jieyouxu commented Feb 24, 2024 • edited Loading

rustbot commented Feb 24, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieyouxu Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

jieyouxu commented Mar 7, 2024

jieyouxu commented Mar 7, 2024

matthiaskrgr commented Mar 10, 2024

jieyouxu commented Mar 10, 2024

onur-ozkan commented Mar 10, 2024

bors commented Mar 10, 2024

bors commented Mar 10, 2024

bors commented Mar 10, 2024

rust-timer commented Mar 10, 2024

Overall result: ❌ regressions - no action needed

jieyouxu commented Feb 24, 2024 •

edited

Loading

jieyouxu Feb 29, 2024 •

edited

Loading