coverage: Explicitly test the coverage maps produced by codegen/LLVM #114843

Zalathar · 2023-08-15T08:22:49Z

Our existing coverage tests verify the output of end-to-end coverage reports, but we don't have any way to test the specific mapping information (code regions and their associated counters) that are emitted by rustc_codegen_llvm and LLVM. That makes it harder to to be confident in changes that would modify those mappings (whether deliberately or accidentally).

This PR addresses that by adding a new coverage-map test suite that does the following:

Compiles test files to LLVM IR assembly (.ll)
Feeds those IR files to a custom tool (src/tools/coverage-dump) that extracts and decodes coverage mappings, and prints them in a more human-readable format
Checks the output of that tool against known-good snapshots

I recommend excluding the last commit while reviewing the main changes, because that last commit is just ~40 test files copied over from tests/run-coverage, plus their blessed coverage-map snapshots and a readme file. Those snapshots aren't really intended to be checked by hand; they're mostly there to increase the chances that an unintended change to coverage maps will be observable (even if it requires relatively specific circumstances to manifest).

rustbot · 2023-08-15T08:22:55Z

r? @compiler-errors

(rustbot has picked a reviewer for you, use r? to override)

rustbot · 2023-08-15T08:22:58Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Zalathar · 2023-08-15T08:24:20Z

I plan to use these tests to help ensure that the changes in #114399 don't regress/bloat the emitted coverage mappings.

Zalathar · 2023-08-15T10:22:10Z

src/tools/coverage-dump/Cargo.toml

+[dependencies]
+anyhow = "1.0.71"
+md5 = { package = "md-5" , version = "0.10.5" }
+miniz_oxide = "0.7.1"
+regex = "1.8.4"
+rustc-demangle = "0.1.23"


All of these version numbers are just copied from whatever happened to be in Cargo.lock at the time, to avoid bumping the existing versions.

src/tools/coverage-dump/src/parser.rs

tests/coverage-map/if.cov-map

Zalathar · 2023-08-16T02:03:39Z

Cargo.lock

+[[package]]
+name = "leb128"
+version = "0.2.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "884e2677b40cc8c339eaefcb701c32ef1fd2493d71118dc0ca4b6a736c93bd67"
+


As far as I can tell, leb128 is maintained and published by the same org as gimli, under the same Apache2/MIT license.

(gimli previously used leb128 as a dependency, but nowadays it uses its own slightly-modified copy in a submodule.)

Swatinem

🎉 its a lot more readable like this :-)

Zalathar · 2023-08-17T02:53:31Z

I decided to remove #![deny(warnings)] from the new tests, because I realised that it would be really annoying for someone else to have their CI run fail just because of a new warning.

Zalathar · 2023-09-05T09:47:54Z

Looks like CI flaked out again on that run.

Zalathar · 2023-09-05T09:49:38Z

Also note that this and #114656 are racing in silent conflict; whichever one lands first will require the other to update #[no_coverage] in this PR’s tests.

oli-obk · 2023-09-05T12:01:41Z

@bors retry

bors · 2023-09-05T15:31:03Z

⌛ Testing commit 3141177 with merge ab45885...

bors · 2023-09-05T17:20:25Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing ab45885 to master...

rust-timer · 2023-09-05T22:07:03Z

Finished benchmarking commit (ab45885): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.0%	[-2.5%, -1.0%]	6
Improvements ✅ (secondary)	-1.7%	[-1.7%, -1.7%]	1
All ❌✅ (primary)	-2.0%	[-2.5%, -1.0%]	6

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 628.171s -> 626.58s (-0.25%)
Artifact size: 316.34 MiB -> 316.27 MiB (-0.02%)

coverage: Don't bother renumbering expressions on the Rust side The LLVM API that we use to encode coverage mappings already has its own code for removing unused coverage expressions and renumbering the rest. This lets us get rid of our own complex renumbering code, making it easier to change our coverage code in other ways. --- Now that we have tests for coverage mappings (rust-lang#114843), I've been able to verify that this PR doesn't make the coverage mappings worse, thanks to an explicit simplification step.

These multi-file tests were not copied over in rust-lang#114843 because they weren't working, but it turns out that they just need the correct crate-type.

compiletest: Unify `cmd2procres` with `run_command_to_procres` Historical context: I originally added `run_command_to_procres` in rust-lang#112300 and rust-lang#114843, because I didn't like the overly-specific failure message in `cmd2procres`, but at the time I didn't feel confident enough to change the existing code, so I just added my own similar code. Now I'm going back to remove this redundancy by eliminating `cmd2procress`, and adjusting all callers to use a slightly-tweaked `run_command_to_procres` instead.

Rollup merge of rust-lang#125753 - Zalathar:procres, r=jieyouxu compiletest: Unify `cmd2procres` with `run_command_to_procres` Historical context: I originally added `run_command_to_procres` in rust-lang#112300 and rust-lang#114843, because I didn't like the overly-specific failure message in `cmd2procres`, but at the time I didn't feel confident enough to change the existing code, so I just added my own similar code. Now I'm going back to remove this redundancy by eliminating `cmd2procress`, and adjusting all callers to use a slightly-tweaked `run_command_to_procres` instead.

rustbot assigned compiler-errors Aug 15, 2023

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Aug 15, 2023

rustbot added the A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) label Aug 15, 2023

This comment has been minimized.

Sign in to view

Zalathar force-pushed the test-coverage-map branch from 2a4d4e3 to 37c010e Compare August 15, 2023 09:44

This comment has been minimized.

Sign in to view

This comment was marked as resolved.

Sign in to view

Zalathar force-pushed the test-coverage-map branch from 37c010e to 6085e7f Compare August 15, 2023 10:15

Zalathar commented Aug 15, 2023

View reviewed changes

Swatinem reviewed Aug 15, 2023

View reviewed changes

src/tools/coverage-dump/src/parser.rs Outdated Show resolved Hide resolved

tests/coverage-map/if.cov-map Outdated Show resolved Hide resolved

Zalathar force-pushed the test-coverage-map branch 2 times, most recently from 6c19230 to 5a5511c Compare August 16, 2023 01:59

Zalathar commented Aug 16, 2023

View reviewed changes

Zalathar force-pushed the test-coverage-map branch from 5a5511c to 317c18d Compare August 16, 2023 07:59

Swatinem approved these changes Aug 16, 2023

View reviewed changes

Zalathar force-pushed the test-coverage-map branch 3 times, most recently from 2d68e55 to 33f3722 Compare August 17, 2023 02:10

Zalathar mentioned this pull request Aug 17, 2023

Dummy PR to test coverage test changes in CI #114917

Draft

Zalathar force-pushed the test-coverage-map branch from 33f3722 to 55f2e57 Compare August 17, 2023 02:51

Zalathar force-pushed the test-coverage-map branch 3 times, most recently from db419f6 to 307d599 Compare August 20, 2023 02:02

Zalathar mentioned this pull request Aug 20, 2023

coverage: Don't bother renumbering expressions on the Rust side #114399

Merged

Zalathar mentioned this pull request Sep 5, 2023

Rework no_coverage to coverage(off) #114656

Merged

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 5, 2023

bors merged commit ab45885 into rust-lang:master Sep 5, 2023
11 checks passed

rustbot added this to the 1.74.0 milestone Sep 5, 2023

Zalathar deleted the test-coverage-map branch September 6, 2023 00:37

Zalathar mentioned this pull request May 30, 2024

compiletest: Unify cmd2procres with run_command_to_procres #125753

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

coverage: Explicitly test the coverage maps produced by codegen/LLVM #114843

coverage: Explicitly test the coverage maps produced by codegen/LLVM #114843

Zalathar commented Aug 15, 2023 •

edited

Loading

rustbot commented Aug 15, 2023

rustbot commented Aug 15, 2023

Zalathar commented Aug 15, 2023

This comment has been minimized.

This comment has been minimized.

This comment was marked as resolved.

Zalathar Aug 15, 2023

Zalathar Aug 16, 2023

Swatinem left a comment

Zalathar commented Aug 17, 2023

Zalathar commented Sep 5, 2023

Zalathar commented Sep 5, 2023

oli-obk commented Sep 5, 2023

bors commented Sep 5, 2023

bors commented Sep 5, 2023

rust-timer commented Sep 5, 2023

coverage: Explicitly test the coverage maps produced by codegen/LLVM #114843

coverage: Explicitly test the coverage maps produced by codegen/LLVM #114843

Conversation

Zalathar commented Aug 15, 2023 • edited Loading

rustbot commented Aug 15, 2023

rustbot commented Aug 15, 2023

Zalathar commented Aug 15, 2023

This comment has been minimized.

This comment has been minimized.

This comment was marked as resolved.

Zalathar Aug 15, 2023

Choose a reason for hiding this comment

Zalathar Aug 16, 2023

Choose a reason for hiding this comment

Swatinem left a comment

Choose a reason for hiding this comment

Zalathar commented Aug 17, 2023

Zalathar commented Sep 5, 2023

Zalathar commented Sep 5, 2023

oli-obk commented Sep 5, 2023

bors commented Sep 5, 2023

bors commented Sep 5, 2023

rust-timer commented Sep 5, 2023

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Zalathar commented Aug 15, 2023 •

edited

Loading