Include test suite metadata in the build metrics #111936

pietroalbini · 2023-05-25T07:59:43Z

This PR enhances the build metadata to include structured information about the test suites being executed, allowing external tools consuming the metadata to understand what was being tested.

The included metadata is:

Target triple
Host triple
Stage number
For compiletest tests:
- Suite name
- Mode
- Comparing mode
For crate tests:
- List of crate names

This is implemented by replacing the test JSON node with a new test_suite node, which contains the metadata and the list of tests. This change also improves the handling of multiple test suites executed in the same step (for example in compiletest tests with a compare mode), as the multiple test suite executions will now be tracked in separate test_suite nodes.

This included a breaking change in the build metrics metadata format. To better handle this, in the second commit this PR introduces the metadata_version top-level field. The old version is considered to be 0, while the new one 1. Bootstrap will also gracefully handle existing metadata of a different version.

r? @jyn514

jyn514

This seems broadly good 👍 except that I'm confused why METADATA_VERSION is necessary. Thank you for using the new run_cargo_test abstraction instead of duplicating the code ❤️

src/bootstrap/metrics.rs

pietroalbini · 2023-05-26T07:48:07Z

Addressed or responded to all feedback.

@rustbot ready

The new name is more accurate.

pietroalbini · 2023-05-26T13:26:05Z

Also renamed metadata_version to format_version since the latter name is more accurate.

jyn514 · 2023-05-26T13:31:49Z

r=me with CI passing :)

pietroalbini · 2023-05-26T14:48:16Z

@bors r=jyn514

bors · 2023-05-26T14:48:18Z

📌 Commit c5139b9 has been approved by jyn514

It is now in the queue for this repository.

…llaumeGomez Rollup of 6 pull requests Successful merges: - rust-lang#111936 (Include test suite metadata in the build metrics) - rust-lang#111952 (Remove DesugaringKind::Replace.) - rust-lang#111966 (Add #[inline] to array TryFrom impls) - rust-lang#111983 (Perform MIR type ops locally in new solver) - rust-lang#111997 (Fix re-export of doc hidden macro not showing up) - rust-lang#112014 (rustdoc: get unnormalized link destination for suggestions) r? `@ghost` `@rustbot` modify labels: rollup

232: Split and refactor the test outcomes page r=Veykril a=pietroalbini This PR changes how the test outcomes page's content is structured, splitting it into one page per target. This has two purposes: * Reduce the clutter on the page, which would otherwise mix information of all qualified targets. * Provide an accurate list of ignored tests, as each target now has the list of tests it ignored rather than just displaying a list of tests ignored by *all* targets. To simplify the generation of the separate pages, I changed how the `ferrocene_test_outcomes` extension works. Rather than having `.. ignored-tests::` and `.. suite-summary::` directives that are rendered in Python code, the extension now gathers and organizes the data, and defers the rendering of the information to a Jinja2 rST template. This results in the whole test results page being rendered by the template, which eases the maintenance of those pages. For example, a note that should be displayed only on cross-compiled targets can now be gated behind `{% if host != target %}`. Finally, while refactoring I removed the whole parsing code of the debug representation of steps, replacing it with the [structured test metadata](rust-lang/rust#111936) I added a while back. This will increase the robustness of the tool. There are still some open issues on this I'll address in a future PR: * The list of crates for the bootstrap test suite is empty. * The note for doc-tests not being executed is not present on aarch64. * I'd like to add references to the test suite definitions in the evaluation plan. * I'd like to see if I can make the information displayed in the page more concise. 238: Code Coverage Support for Core Library [DUPLICATE] r=pietroalbini a=Ax9DTW Bors is stuck on #180, so this duplicate. 243: Automated pull from `rust-lang/libc` r=Veykril a=github-actions[bot] This PR pulls the following changes from the [`rust-lang/libc`](https://github.com/rust-lang/libc) repository: * `3570`: [[Backport #3548] Add ioctl FS_IOC_{G,S}{ETVERSION,ETFLAGS} for LoongArch64](https://www.github.com/rust-lang/libc/issues/3570) * `3553`: [Add MFD_NOEXEC_SEAL and MFD_EXEC](https://www.github.com/rust-lang/libc/issues/3553) * `3554`: [Backport of #3546 and update crate version to 0.2.153](https://www.github.com/rust-lang/libc/issues/3554) Co-authored-by: Pietro Albini <[email protected]> Co-authored-by: Atri Sarkar <[email protected]> Co-authored-by: Gnana Ganesh <[email protected]> Co-authored-by: rusty-snake <[email protected]> Co-authored-by: Guillaume Gomez <[email protected]> Co-authored-by: WANG Rui <[email protected]> Co-authored-by: Yuki Okushi <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

232: Split and refactor the test outcomes page r=Veykril a=pietroalbini This PR changes how the test outcomes page's content is structured, splitting it into one page per target. This has two purposes: * Reduce the clutter on the page, which would otherwise mix information of all qualified targets. * Provide an accurate list of ignored tests, as each target now has the list of tests it ignored rather than just displaying a list of tests ignored by *all* targets. To simplify the generation of the separate pages, I changed how the `ferrocene_test_outcomes` extension works. Rather than having `.. ignored-tests::` and `.. suite-summary::` directives that are rendered in Python code, the extension now gathers and organizes the data, and defers the rendering of the information to a Jinja2 rST template. This results in the whole test results page being rendered by the template, which eases the maintenance of those pages. For example, a note that should be displayed only on cross-compiled targets can now be gated behind `{% if host != target %}`. Finally, while refactoring I removed the whole parsing code of the debug representation of steps, replacing it with the [structured test metadata](rust-lang/rust#111936) I added a while back. This will increase the robustness of the tool. There are still some open issues on this I'll address in a future PR: * The list of crates for the bootstrap test suite is empty. * The note for doc-tests not being executed is not present on aarch64. * I'd like to add references to the test suite definitions in the evaluation plan. * I'd like to see if I can make the information displayed in the page more concise. Co-authored-by: Pietro Albini <[email protected]>

232: Split and refactor the test outcomes page r=Veykril a=pietroalbini This PR changes how the test outcomes page's content is structured, splitting it into one page per target. This has two purposes: * Reduce the clutter on the page, which would otherwise mix information of all qualified targets. * Provide an accurate list of ignored tests, as each target now has the list of tests it ignored rather than just displaying a list of tests ignored by *all* targets. To simplify the generation of the separate pages, I changed how the `ferrocene_test_outcomes` extension works. Rather than having `.. ignored-tests::` and `.. suite-summary::` directives that are rendered in Python code, the extension now gathers and organizes the data, and defers the rendering of the information to a Jinja2 rST template. This results in the whole test results page being rendered by the template, which eases the maintenance of those pages. For example, a note that should be displayed only on cross-compiled targets can now be gated behind `{% if host != target %}`. Finally, while refactoring I removed the whole parsing code of the debug representation of steps, replacing it with the [structured test metadata](rust-lang/rust#111936) I added a while back. This will increase the robustness of the tool. There are still some open issues on this I'll address in a future PR: * The list of crates for the bootstrap test suite is empty. * The note for doc-tests not being executed is not present on aarch64. * I'd like to add references to the test suite definitions in the evaluation plan. * I'd like to see if I can make the information displayed in the page more concise. 245: Inject channel dynamically into sphinx substitutions r=pietroalbini a=Veykril Co-authored-by: Pietro Albini <[email protected]> Co-authored-by: Lukas Wirth <[email protected]>

rustbot assigned jyn514 May 25, 2023

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels May 25, 2023

This comment has been minimized.

Sign in to view

pietroalbini added 2 commits May 25, 2023 10:14

include test suite metadata in build metrics

3c9b076

introduce build metrics version numbers to handle breaking changes

0553f71

pietroalbini force-pushed the pa-test-suite-metadata branch from 2718e65 to 0553f71 Compare May 25, 2023 08:14

jyn514 reviewed May 26, 2023

View reviewed changes

src/bootstrap/metrics.rs Outdated Show resolved Hide resolved

src/bootstrap/metrics.rs Outdated Show resolved Hide resolved

src/bootstrap/metrics.rs Outdated Show resolved Hide resolved

src/bootstrap/metrics.rs Outdated Show resolved Hide resolved

jyn514 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 26, 2023

address review feedback

cb68c05

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 26, 2023

jyn514 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 26, 2023

pietroalbini added 2 commits May 26, 2023 15:21

rename metadata_version to format_version

7040d41

The new name is more accurate.

add reasoning for introducing a metrics format version

c5139b9

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 26, 2023

GuillaumeGomez mentioned this pull request May 27, 2023

Rollup of 6 pull requests #112016

Merged

bors merged commit 859068c into rust-lang:master May 27, 2023

rustbot added this to the 1.71.0 milestone May 27, 2023

pietroalbini deleted the pa-test-suite-metadata branch June 5, 2023 07:34

jyn514 mentioned this pull request Jun 5, 2023

Track which tests are run in CI #95995

Open

pietroalbini mentioned this pull request Jan 29, 2024

Split and refactor the test outcomes page ferrocene/ferrocene#232

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include test suite metadata in the build metrics #111936

Include test suite metadata in the build metrics #111936

pietroalbini commented May 25, 2023

This comment has been minimized.

jyn514 left a comment

pietroalbini commented May 26, 2023

pietroalbini commented May 26, 2023

jyn514 commented May 26, 2023

pietroalbini commented May 26, 2023

bors commented May 26, 2023

Include test suite metadata in the build metrics #111936

Include test suite metadata in the build metrics #111936

Conversation

pietroalbini commented May 25, 2023

This comment has been minimized.

jyn514 left a comment

Choose a reason for hiding this comment

pietroalbini commented May 26, 2023

pietroalbini commented May 26, 2023

jyn514 commented May 26, 2023

pietroalbini commented May 26, 2023

bors commented May 26, 2023