No coverage data if testing code from external repositories #16208

celkas · 2022-09-02T20:28:02Z

Description of the bug:

Using software and tests in the same repository is no problem, coverage data are produced.
Splitting the two makes problems. There is "no coverage data" produced anymore.
It seems that coverage data is generated, but not consumed, maybe because of some path issue/mismatch.
If I also instrument the tests, the test's coverage will be copied into the combined coverage file, but still no software coverage.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Minimal example in this test repo. This will use this software repo and produce no coverage.

Simply check it out and run

bazel coverage -c dbg --combined_report=lcov --instrumentation_filter="@sw_repo//:funclib" --action_env=COVERAGE_GCOV_OPTIONS=-b //:mytest

This will say WARNING: There was no coverage found.

Which operating system are you running Bazel on?

Debian (WSL2)

What is the output of `bazel info release`?

release 5.3.0

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

No response

Have you found anything relevant by searching the web?

I did not find anything searching.

In this Slack discussion I was encouraged to create an issue.

Any other information, logs, or outputs that you want to share?

I can see in the testlog that coverage info is build. Then it is processed but no data is found.

Executing tests from //:mytest
-----------------------------------------------------------------------------
File 'test/test.c'
Lines executed:83.33% of 6
Branches executed:100.00% of 2
Taken at least once:50.00% of 2
Calls executed:50.00% of 2

Creating 'test.pic.gcov.json.gz'
Lines executed:83.33% of 6
File 'external/sw_repo/sw/func.c'
Lines executed:53.85% of 13
Branches executed:100.00% of 3
Taken at least once:33.33% of 3
No calls

Creating 'func.pic.gcov.json.gz'
Lines executed:53.85% of 13
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: No lcov file found.
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: Found 2 gcov json files.
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file /home/lukas/.cache/bazel/_bazel_lukas/7da12e39a3158a3f59d65f10c3647c59/sandbox/linux-sandbox/13/execroot/test_repo/_coverage/mytest/test/test.pic.gcov.json.gz
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file /home/lukas/.cache/bazel/_bazel_lukas/7da12e39a3158a3f59d65f10c3647c59/sandbox/linux-sandbox/13/execroot/test_repo/_coverage/mytest/test/func.pic.gcov.json.gz
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Sep 02, 2022 7:49:56 PM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.

The text was updated successfully, but these errors were encountered:

Removes hardcoded filters that result in no coverage being reported for `cc_*` targets in external repositories even when matched by the `--instrumentation_filter`. Work towards #16228 Work towards #16208 Closes #16261. PiperOrigin-RevId: 483911568 Change-Id: Ibca6fb39a8e946e06beeb03414ad8ccbc42f53d6

Removes hardcoded filters that result in no coverage being reported for `cc_*` targets in external repositories even when matched by the `--instrumentation_filter`. Work towards bazelbuild#16228 Work towards bazelbuild#16208 Closes bazelbuild#16261. PiperOrigin-RevId: 483911568 Change-Id: Ibca6fb39a8e946e06beeb03414ad8ccbc42f53d6

Removes hardcoded filters that result in no coverage being reported for `cc_*` targets in external repositories even when matched by the `--instrumentation_filter`. Work towards #16228 Work towards #16208 Closes #16261. PiperOrigin-RevId: 483911568 Change-Id: Ibca6fb39a8e946e06beeb03414ad8ccbc42f53d6

* Removes a hardcoded filter that results in the LCOV merger filtering out coverage of external targets even if these are matched by `--instrumentation_filter`. * Lets `LazyWritePathsFileAction` write out exec rather than root-relative paths, as advertised by its javadocs, so that the paths match those that the LCOV mergers filters by. Work towards #16228 Work towards #16208 Closes #16268. PiperOrigin-RevId: 485580267 Change-Id: I830d3cf903462ca1799ef5f659a620dbdb92f349

* Removes a hardcoded filter that results in the LCOV merger filtering out coverage of external targets even if these are matched by `--instrumentation_filter`. * Lets `LazyWritePathsFileAction` write out exec rather than root-relative paths, as advertised by its javadocs, so that the paths match those that the LCOV mergers filters by. Work towards bazelbuild#16228 Work towards bazelbuild#16208 Closes bazelbuild#16268. PiperOrigin-RevId: 485580267 Change-Id: I830d3cf903462ca1799ef5f659a620dbdb92f349

* Removes a hardcoded filter that results in the LCOV merger filtering out coverage of external targets even if these are matched by `--instrumentation_filter`. * Lets `LazyWritePathsFileAction` write out exec rather than root-relative paths, as advertised by its javadocs, so that the paths match those that the LCOV mergers filters by. Work towards #16228 Work towards #16208 Closes #16268. PiperOrigin-RevId: 485580267 Change-Id: I830d3cf903462ca1799ef5f659a620dbdb92f349

celkas · 2022-11-10T12:22:33Z

@fmeum This will be available in 6.0.0 if I understand it correctly, right? Should I close it only then, after having verified it?

fmeum · 2022-11-10T12:27:56Z

@celkas Yes, both commits are going to be in 6.0.0. You can close the issue if everything works for you with these changes.

celkas · 2022-11-11T10:30:40Z

@fmeum Tested it with 6.0.0rc1. Still failing. States WARNING: There was no coverage found. for both versions (5.3.2 and 6.0.0rc1) alike.

Did also some more tests with bazel clean (using both versions) in between the runs, but to no avail.

celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$ bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Multiplexer process for Javac has closed its output stream
INFO: Multiplexer process for Javac has closed its output stream
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$ bazel coverage -c dbg --combined_report=lcov --instrumentation_filter="@sw_repo//:funclib" --action_env=COVERAGE_GCOV_OPTIONS=-b //:mytest
INFO: Analyzed target //:mytest (49 packages loaded, 898 targets configured).
INFO: Found 1 test target...
INFO: LCOV coverage report is located at /home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/execroot/test_repo/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
INFO: From Coverage report generation:
Nov 11, 2022 11:17:26 AM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: Found 1 tracefiles.
Nov 11, 2022 11:17:26 AM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file bazel-out/k8-dbg/testlogs/mytest/coverage.dat
Nov 11, 2022 11:17:26 AM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Nov 11, 2022 11:17:26 AM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Nov 11, 2022 11:17:26 AM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Nov 11, 2022 11:17:26 AM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.
Target //:mytest up-to-date:
  bazel-bin/mytest
INFO: Elapsed time: 15.584s, Critical Path: 12.25s
INFO: 24 processes: 12 internal, 10 linux-sandbox, 2 worker.
INFO: Build completed successfully, 24 total actions
//:mytest                                                                PASSED in 1.8s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 24 total actions
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$ USE_BAZEL_VERSION=6.0.0rc1 bazel coverage -c dbg --combined_report=lcov --instrumentation_filter="@sw_repo//:funclib" --action_env=COVERAGE_GCOV_OPTIONS=-b //:mytest
WARNING: Waiting for server process to terminate (waited 5 seconds, waiting at most 60)
WARNING: Waiting for server process to terminate (waited 10 seconds, waiting at most 60)
Starting local Bazel server and connecting to it...
DEBUG: Rule 'sw_repo' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "bd866e57098202c29f04e804b98a559e9ea6df53", shallow_since = "1662150717 +0200" and dropping ["branch"]
DEBUG: Repository sw_repo instantiated at:
  /home/celkas/work/git/bazel-issue/bazel-issue-tests/WORKSPACE.bazel:8:15: in <toplevel>
Repository rule git_repository defined at:
  /home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/external/bazel_tools/tools/build_defs/repo/git.bzl:176:33: in <toplevel>
INFO: Analyzed target //:mytest (48 packages loaded, 706 targets configured).
INFO: Found 1 test target...
INFO: LCOV coverage report is located at /home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/execroot/test_repo/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
INFO: From Coverage report generation:
Nov 11, 2022 11:19:01 AM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: Found 1 tracefiles.
Nov 11, 2022 11:19:01 AM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file bazel-out/k8-dbg/testlogs/mytest/coverage.dat
Nov 11, 2022 11:19:01 AM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Nov 11, 2022 11:19:01 AM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Nov 11, 2022 11:19:01 AM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Nov 11, 2022 11:19:01 AM com.google.devtools.coverageoutputgenerator.Main runWithArgs
WARNING: There was no coverage found.
Target //:mytest up-to-date:
  bazel-bin/mytest
INFO: Elapsed time: 56.646s, Critical Path: 17.26s
INFO: 24 processes: 12 internal, 10 linux-sandbox, 2 worker.
INFO: Build completed successfully, 24 total actions
//:mytest                                                                PASSED in 1.9s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$

fmeum · 2022-11-11T10:32:24Z

@celkas Sorry, the changes were cherry-picked after the RC cut. Could you try again with last_green? All of this should be in 6.0.0rc2 when it is cut.

celkas · 2022-11-11T13:40:40Z

@fmeum Great, it works! I still need to somehow tackle the path problems with genhtml, but that is probably another issue.

celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$ USE_BAZEL_VERSION=last_green bazel coverage -c dbg --combined_report=lcov --instrumentation_filter="@sw_repo//:funclib" --action_env=COVERAGE_GCOV_OPTIONS=-b //:mytest
2022/11/11 14:33:55 Using unreleased version at commit 6e945e8ab422ca0a195d09899cce61ba6c0cfa59
WARNING: Waiting for server process to terminate (waited 5 seconds, waiting at most 60)
WARNING: Waiting for server process to terminate (waited 10 seconds, waiting at most 60)
WARNING: Waiting for server process to terminate (waited 30 seconds, waiting at most 60)
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:mytest (48 packages loaded, 706 targets configured).
INFO: Found 1 test target...
INFO: LCOV coverage report is located at /home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/execroot/test_repo/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
Target //:mytest up-to-date:
  bazel-bin/mytest
INFO: Elapsed time: 59.587s, Critical Path: 0.14s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
//:mytest                                                       (cached) PASSED in 1.0s
  /home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/execroot/test_repo/bazel-out/k8-dbg/testlogs/mytest/coverage.dat

Executed 0 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$
celkas@somemachine:~/work/git/bazel-issue/bazel-issue-tests$ cd -
/home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59
celkas@somemachine:~/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59$ genhtml --branch-coverage --output asdf execroot/test_repo/bazel-out/k8-dbg/testlogs/mytest/coverage.dat
Reading data file execroot/test_repo/bazel-out/k8-dbg/testlogs/mytest/coverage.dat
Resolved relative source file path "external/sw_repo/sw/func.c" with CWD to "/home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/external/sw_repo/sw/func.c".
Found 1 entries.
Found common filename prefix "/home/celkas/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59/external/sw_repo"
Writing .css and .png files.
Generating output.
Processing file sw/func.c
Writing directory view page.
Overall coverage rate:
  lines......: 53.8% (7 of 13 lines)
  functions..: 100.0% (1 of 1 function)
  branches...: 33.3% (1 of 3 branches)
celkas@somemachine:~/.cache/bazel/_bazel_celkas/7da12e39a3158a3f59d65f10c3647c59$

celkas · 2022-11-11T13:41:05Z

Solved in 6.0.0

susinmotion · 2022-11-14T21:22:06Z

We're rolling #16637 back since it's breaking some internal tests :( Reopening to keep track of it again.

celkas · 2022-11-16T20:45:46Z

As of now as a workaround for this problem, I use an intermediate version, e.g. 7.0.0-pre.20221102.3.

Usage for this test repo:

USE_BAZEL_VERSION=7.0.0-pre.20221102.3 bazel build -c dbg //:mytest
USE_BAZEL_VERSION=7.0.0-pre.20221102.3 bazel coverage -c dbg --combined_report=lcov --instrumentation_filter="@sw_repo//:funclib" --action_env=COVERAGE_GCOV_OPTIONS=-b //:mytest
[ -L external ] && rm external
ln -s $(bazel info execution_root)/external external
genhtml --branch-coverage --output html "$(bazel info output_path)/_coverage/_coverage_report.dat"

celkas · 2022-11-16T20:49:26Z

@fmeum As an aside: genhtml is not able to find the path if I do not create a symlink into the bazel execution runtime folder. Is that expected? It is reasonable as genthml does not know anything about the bazel environment. I could of course also cd into that directory temporarily, but I do not like both solutions.

fmeum · 2022-11-17T01:47:22Z

@celkas That's always going to be a problem as external repositories aren't materialized in your workspace, they only exist under the Bazel output base. Does genhtml let you specify additional source roots? If so, you could use that instead of the symlink. If there isn't, some sed or the symlink is going to be your best bet.

fmeum · 2023-01-30T15:30:08Z

@sgowroji This can be closed now that 00e9af1 landed.

sgowroji · 2023-01-30T15:37:27Z

Thanks @fmeum. We are closing this issue w.r.t 00e9af1 commit.

brentleyjones · 2023-01-30T16:06:04Z

@fmeum does that need to/can that be cherry-picked into 6.1.0?

fmeum · 2023-01-30T16:31:05Z

@bazel-io flag

ShreeM01 · 2023-01-30T16:42:42Z

@bazel-io fork 6.1.0

sgowroji added type: bug team-Rules-Server Issues for serverside rules included with Bazel untriaged labels Sep 5, 2022

celkas mentioned this issue Sep 8, 2022

Empty code coverage report is generated when test files are referenced as external repository #16228

Closed

fmeum mentioned this issue Sep 13, 2022

Allow C/C++ coverage collection for external targets #16261

Closed

fmeum mentioned this issue Oct 12, 2022

Allow Java coverage collection for external targets #16268

Closed

comius added P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed untriaged labels Oct 17, 2022

sgowroji mentioned this issue Oct 20, 2022

bazel coverage generating empty coverage.dat on single workspace #16515

Closed

fmeum mentioned this issue Oct 26, 2022

Allow C/C++ coverage collection for external targets #16566

Merged

fmeum mentioned this issue Nov 2, 2022

Allow Java coverage collection for external targets #16637

Merged

celkas closed this as completed Nov 11, 2022

susinmotion reopened this Nov 14, 2022

sgowroji closed this as completed Jan 30, 2023

bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Jan 30, 2023

bazel-io mentioned this issue Jan 30, 2023

[6.1.0] No coverage data if testing code from external repositories #17359

Closed

bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Jan 30, 2023

Pavank1992 mentioned this issue Jun 7, 2023

No coverage found inside package "external" #18597

Closed

tsr-boxbot mentioned this issue Aug 23, 2023

Code coverage broken for org codebase #19310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No coverage data if testing code from external repositories #16208

No coverage data if testing code from external repositories #16208

celkas commented Sep 2, 2022

celkas commented Nov 10, 2022

fmeum commented Nov 10, 2022

celkas commented Nov 11, 2022

fmeum commented Nov 11, 2022

celkas commented Nov 11, 2022

celkas commented Nov 11, 2022

susinmotion commented Nov 14, 2022

celkas commented Nov 16, 2022

celkas commented Nov 16, 2022

fmeum commented Nov 17, 2022

fmeum commented Jan 30, 2023

sgowroji commented Jan 30, 2023

brentleyjones commented Jan 30, 2023

fmeum commented Jan 30, 2023

ShreeM01 commented Jan 30, 2023

No coverage data if testing code from external repositories #16208

No coverage data if testing code from external repositories #16208

Comments

celkas commented Sep 2, 2022

Description of the bug:

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

celkas commented Nov 10, 2022

fmeum commented Nov 10, 2022

celkas commented Nov 11, 2022

fmeum commented Nov 11, 2022

celkas commented Nov 11, 2022

celkas commented Nov 11, 2022

susinmotion commented Nov 14, 2022

celkas commented Nov 16, 2022

celkas commented Nov 16, 2022

fmeum commented Nov 17, 2022

fmeum commented Jan 30, 2023

sgowroji commented Jan 30, 2023

brentleyjones commented Jan 30, 2023

fmeum commented Jan 30, 2023

ShreeM01 commented Jan 30, 2023

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?