Code coverage broken for org codebase

[tsr-boxbot]

Description of the bug:

Hello, I'm working on a company codebase and really want to get code coverage working. So far I've failed do so and would like to ask for help. What I am seeing is that I'm unable to generate code coverage reports for a code base I am working in. I see that .dat files are generated but they're usually empty for some reason.

Which category does this issue belong to?

C++/Objective-C Rules, CLI

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I've created an extremely small version of the project with a simplistic cpp file and test to demonstrate the issue, you can find that code in: https://github.com/tsr-boxbot/bazel_code_coverage_problems

A key thing to note here is we are using clang-13 via a toolchain spec.

Here's what I see:

$ bazel-5.4.0 coverage ... --combined_report=lcov -c dbg --config=clang_config                                                                                    
INFO: Using default value for --instrumentation_filter: "^//".                                                                                                                                         
INFO: Override the above default with --instrumentation_filter                                                                                                                                         
INFO: Analyzed 6 targets (50 packages loaded, 966 targets configured).                                                                                                                                 
INFO: Found 5 targets and 1 test target...                                                                                                                                                             
INFO: LCOV coverage report is located at /home/tyler/.cache/bazel/_bazel_tyler/af64a4260e07ac56294cb50354aadc2f/execroot/test/bazel-out/_coverage/_coverage_report.dat                                 
 and execpath is bazel-out/_coverage/_coverage_report.dat                                                                                                                                              
INFO: From Coverage report generation:                                                                                                                                                                 
Aug 23, 2023 2:17:13 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles                                                                                                                 
INFO: Found 1 tracefiles.                                                                                                                                                                              
Aug 23, 2023 2:17:13 PM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially                                                                                                        
INFO: Parsing file bazel-out/k8-dbg/testlogs/add_test/coverage.dat                                                                                                                                     
Aug 23, 2023 2:17:13 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles                                                                                                              
INFO: No gcov info file found.                                                                                                                                                                         
Aug 23, 2023 2:17:13 PM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles                                                                                                          
INFO: No gcov json file found.                                                                                                                                                                         
Aug 23, 2023 2:17:13 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull                                                                                                         
INFO: No .profdata file found.                                                                                                                                                                         
Aug 23, 2023 2:17:13 PM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.
INFO: Elapsed time: 8.765s, Critical Path: 4.81s
INFO: 45 processes: 16 internal, 27 linux-sandbox, 2 worker.
INFO: Build completed successfully, 45 total actions
//:add_test                                                              PASSED in 0.2s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 45 total actions

Where although everything completes okay but I still get: WARNING: There was no coverage found.

I've also tried crazy things like:

$ bazel-5.4.0 coverage --config=clang_config --test_env='BAZEL_CC_COVERAGE_TOOL=LCOV' --test_env='LCOV_MERGER=/usr/bin/true' --experimental_use_llvm_covmap --experimental_generate_llvm_lcov --instrument_test_targets --coverage_report_generator="@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main" --combined_report=lcov ...
INFO: Using default value for --instrumentation_filter: "^//".
INFO: Override the above default with --instrumentation_filter
INFO: Analyzed 6 targets (51 packages loaded, 967 targets configured).
INFO: Found 5 targets and 1 test target...
INFO: LCOV coverage report is located at /home/tyler/.cache/bazel/_bazel_tyler/af64a4260e07ac56294cb50354aadc2f/execroot/test/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
INFO: From Coverage report generation:
Aug 23, 2023 2:24:33 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: Found 1 tracefiles.
Aug 23, 2023 2:24:33 PM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file bazel-out/k8-fastbuild/testlogs/add_test/coverage.dat
Aug 23, 2023 2:24:33 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Aug 23, 2023 2:24:33 PM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Aug 23, 2023 2:24:33 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Aug 23, 2023 2:24:33 PM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.
INFO: Elapsed time: 8.868s, Critical Path: 4.99s
INFO: 46 processes: 17 internal, 27 linux-sandbox, 2 worker.
INFO: Build completed successfully, 46 total actions
//:add_test                                                              PASSED in 0.0s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 46 total actions

Which operating system are you running Bazel on?

Ubuntu 22.04

What is the output of bazel info release?

release 5.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

$ git remote get-url origin; git rev-parse main; git rev-parse HEAD
[email protected]:tsr-boxbot/bazel_code_coverage_problems.git
6e92d782706555b329d9664255c4c8c6ed8d9026
6e92d782706555b329d9664255c4c8c6ed8d9026

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Unsure

Have you found anything relevant by searching the web?

I've found tons of stuff that seems related but so far they haven't solved my problem, I will link some of the resources here
#16208
#7719
#11059
#8284
bazelbuild/rules_apple#338
https://github.com/bazelbuild/rules_scala/blob/master/docs/coverage.md
#11962
https://stackoverflow.com/questions/60751912/output-coverage-coverage-report-dat-was-not-created-when-using-bazel-coverag

Any other information, logs, or outputs that you want to share?

I've looked all over for a functional guide for code coverage but I cannot seem to tell what the correct way to use coverage is.

[c-mita]

Coverage support with LLVM tools is somewhat buggy. In particular, collect_cc_coverage.sh expects llvm-cov to be under the name "gcov". I would argue this is a bug, but I don't know what's depending on this still so I haven't changed it yet.

You may enjoy some success by doing the following:

• Add tool_path(name = "gcov", path = "/usr/bin/llvm-cov-13") to your set of tool paths in your tool chain (as well as the original llvm-cov definition).
• Use a version of Bazel that includes 7f628d7 (I think 6.3.0 is the first)

[tsr-boxbot]

@c-mita thank you, your suggestions at least bring my small sample into working order.

I again will try updating my org's codebase to bazel 6 but protobuf is giving me a headache 😢