Support coverage

[ulfjack]

We've gotten a number of requests to support coverage. Here's an email I wrote in response to one of those requests, which I'm reposting here as a feature request:

Ideally, we'd want blaze coverage to exist and work in Bazel, just as it does in Blaze. The code is mostly there, but not quite. The coverage command itself doesn't actually do a lot of work; it sets --collect_code_coverage and a default --instrumentation_filter, and it optionally runs a tool to generate an HTML report (which is almost never used internally, where we push to a service instead). Otherwise it's identical to the test command, and the two flags can be manually set.

Internally, we have a wrapper script (collect_coverage.sh) that wraps an individual test invocation and uses a tool (lcov_merger.par) to merge the coverage data from that invocation, which both aren't in Bazel, and Bazel also doesn't setup the environment correctly to tell the wrapper script about output locations (StandaloneTestStrategy.java).

If you actually tried, you'd get an error like this one:
ERROR: /usr/local/google/home/ulfjack/Projects/os-bazel/src/test/cpp/BUILD:5:1: output 'src/test/cpp/blaze_util_test/coverage.dat' was not created.

It doesn't seem very hard to make the necessary Bazel changes, but we can't export the scripts as-is.

From my reading of the lcov documentation, lcov --capture looks a lot like what lcov_merger.par does, though I can't be completely certain (the docs are a bit scarce). It may make sense for us to integrate with lcov rather than export lcov_merger.par (or rewrite from scratch).

[ulfjack]

From the lcov docs, I'd have gone for lcov -a, not lcov --capture.

[marczych]

Any update on this?

I'm very interested in gathering code coverage using Bazel in the near future. It looks like the 0.6 release is slated for 2016-11 assuming that the roadmap is up to date. How could I gather code coverage metrics for my code in Bazel if I needed it in the next month or two?

[ulfjack]

No updates. We'll likely not be able to work on this before October.

Java or C++ code? Or some other language?

[marczych]

C++. I see some support for gcov in the CROSSTOOL as well as the --collect_code_coverage flag but it's failing with the following error:

ERROR: /home/<redacted>/test/BUILD:3:1: output 'test/test/coverage.dat' was not created.

[ulfjack]

Is that a build or a test run? Can you give the bazel command lines you're running?

[marczych]

I made a really simple test workspace to test on:

[marc@marc-desktop] - [~/code/bazel-external-deps] - [Wed Aug 31, 02:15:15]
[I]> bazel clean
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
[marc@marc-desktop] - [~/code/bazel-external-deps] - [Wed Aug 31, 02:15:19]
[I]> bazel build --collect_code_coverage --test_output=all ...
INFO: Found 1 target...
Target //test:test up-to-date:
  bazel-bin/test/test
INFO: Elapsed time: 1.883s, Critical Path: 0.51s
[marc@marc-desktop] - [~/code/bazel-external-deps] - [Wed Aug 31, 02:15:24]
[I]> bazel test --collect_code_coverage --test_output=all ...
INFO: Found 1 test target...
INFO: From Testing //test:test:
==================== Test output for //test:test:
Running test!
================================================================================
ERROR: /home/marc/code/bazel-external-deps/test/BUILD:3:1: output 'test/test/coverage.dat' was not created.
ERROR: /home/marc/code/bazel-external-deps/test/BUILD:3:1: not all outputs were created.
Target //test:test up-to-date:
  bazel-bin/test/test
INFO: Elapsed time: 0.320s, Critical Path: 0.09s
//test:test                                                              PASSED in 0.1s

Executed 1 out of 1 test: 1 test passes.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
One or more non-test targets failed to build.

So, curiously, the build and test passed but the test command exited with 1 because One or more non-test targets failed to build..

//test:test is just a cc_test with a single source file.

[marczych]

I also found test.pic.gcno and baseline_coverage.dat in bazel-out after bazel build --collect_code_coverage ...:

bazel-out
├── local-fastbuild
│   ├── bin
│   │   └── test
│   │       ├── _objs
│   │       │   └── test
│   │       │       └── test
│   │       │           ├── test.pic.d
│   │       │           ├── test.pic.gcno
│   │       │           └── test.pic.o
│   │       ├── test
│   │       ├── test-2.params
│   │       ├── test.runfiles
│   │       │   ├── bazelexternaldeps
│   │       │   │   └── test
│   │       │   │       └── test -> /home/marc/.cache/bazel/_bazel_marc/ad6921e960253c5ea1a81aa9e8086de8/execroot/bazel-external-deps/bazel-out/local-fastbuild/bin/test/test
│   │       │   └── MANIFEST
│   │       └── test.runfiles_manifest
│   ├── genfiles
│   └── testlogs
│       └── test
│           └── test
│               ├── baseline_coverage.dat
│               ├── test.cache_status
│               ├── test.log
│               └── test.xml
├── stable-status.txt
└── volatile-status.txt

13 directories, 14 files

[ddwolf]

is this being processed?

[damienmg]

Sorry forgot to assign it but @hermione521 is working on it this quarter.

[damienmg]

Also /cc @lberki

[marczych]

Just in case anybody was wondering or wanted to get some code coverage support before it's fully incorporated into Bazel, I came up with a usable workaround:

• Build without sandboxing.
• Add these compiler flags on the build and test command:

        "--linkopt=--coverage",
        "--linkopt=-lgcov",
        "--linkopt=-lgcc",
        "--linkopt=-lc",
        # Don't allow cached test results. We need the app to run each time to generate gcda files
        "--nocache_test_results",

• That will put the various coverage files in bazel-out which get updated when the binaries are executed.
• You can then run your lcov command in bazel-out to generate a report.

I left out a lot of details but that's the general approach that is mostly working for me.

[ulfjack]

With the above patches, I was able to get C++ coverage results on Linux, with a local install of lcov (it currently uses a hard-coded path to lcov). We have a follow-up change to add support for Java / Jacoco.

[hermione521]

For reference, the 4 commits for java coverage are here:
7ecf871
af1e3c2
af3c412
af878d0

[hanwen]

looks like the code is there (thanks!), but I couldn't find any documentation.

[hanwen]

also, I can't find anything to generate HTML reports. Or is there a standard tool to interpret the .dat files that I overlooked?

[ulfjack]

I've been testing the C++ coverage with the lcov HTML generator. See here: http://ltp.sourceforge.net/coverage/lcov.php Instructions are in the commit description, commit 8829aba.

…

On Mon, Jan 30, 2017 at 7:05 PM, Han-Wen Nienhuys ***@***.***> wrote: also, I can't find anything to generate HTML reports. Or is there a standard tool to interpret the .dat files that I overlooked? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHA9YdP4FndI1S_nEUN0F6NW43QK8Zdzks5rXiZngaJpZM4H_9LK> .

[tmarkovich]

I'm trying to get coverage reports for a C++ project on OS X 10.12.6.

╰─ bazel version
Build label: 0.12.0-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Oct 2 08:08:38 +50246 (1523467123718)
Build timestamp: 1523467123718
Build timestamp as int: 1523467123718
╰─ gcov --version
Apple LLVM version 9.0.0 (clang-900.0.39.2)
  Optimized build.
  Default target: x86_64-apple-darwin16.7.0
  Host CPU: skylake
╰─ clang --version
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

When I try and us either bazel coverage test... or bazel test test:test-graph-debug --collect_code_coverage (even when I explicitly add "-fprofile-arcs" to linkopts) I get the following output:

╰─ bazel coverage test:test-graph-debug
INFO: Using default value for --instrumentation_filter: "//test".
INFO: Override the above default with --instrumentation_filter
INFO: Analysed target //test:test-graph-debug (0 packages loaded).
INFO: Found 1 test target...
ERROR: /private/var/tmp/_bazel_thomasmarkovich/1f9f2436753d5519b9fc860f09b2662d/external/boost/BUILD.bazel:1098:1: Linking of rule '@boost//:serialization' failed (Exit 1)
ld: library not found for -lgcov
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //test:test-graph-debug failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 26.643s, Critical Path: 11.58s
FAILED: Build did NOT complete successfully

Executed 0 out of 1 test: 1 fails to build.

It appears to occur because

bazel/tools/cpp/crosstool_lib.bzl

Line 256 in 5cee7f4

def _coverage_feature(use_llvm_format):

doesn't handle the darwin case. The darwin branch should be (although there is no such thing at the moment):

    compile_flags = flags("-fprofile-arcs", "-ftest-coverage")
    link_flags = flags("-fprofile-arcs")

This appears related to this issue: #2275

As a work around, I explicitly changed my build rule to be:

    copts = [
        "-fprofile-arcs",
        "-ftest-coverage",
    ],
    linkopts = [
        "-fprofile-arcs",
    ],

with the hope that I could then collect the gcno and gcda artifacts to generate these artifacts. I can't seem to find where these artifacts are put in the various bazel directories when I run without specifying GCOV* env variables. By specifying the environment variables using the action_env command line arg like so bazel test test:test-graph-debug --action_env=GCOV_PREFIX="/Users/thomasmarkovich/Documents/knowledge-graph/coverage" --action_env=GCOV_PREFIX_STRIP=17, my program runs cleaning and provides the following minor complaint at the end:

[==========] 45 tests from 4 test cases ran. (2 ms total)
[  PASSED  ] 45 tests.
profiling: /Users/thomasmarkovich/Documents/knowledge-graph/coverage//gtest_main.gcda: cannot open: Operation not permitted
profiling: /Users/thomasmarkovich/Documents/knowledge-graph/coverage//test-graph.gcda: cannot open: Operation not permitted

Note the double slash. This appears to largely be a permissions issue, but I can't for the life of me figure out how to fix this.

By contrast, when I locate the test-graph-debug executable directly, and run the following, I get the following:

╰─ export GCOV_PREFIX="/Users/thomasmarkovich/Documents/knowledge-graph/coverage/"
╰─ export GCOV_PREFIX_STRIP=17
╰─ ./test-graph-debug
...
[==========] 45 tests from 4 test cases ran. (2 ms total)
[  PASSED  ] 45 tests.
╰─ ls /Users/thomasmarkovich/Documents/knowledge-graph/coverage/
gtest_main.gcda test-graph.gcda

This is to say that gcov appears to be working fine. I just can't figure out how to get bazel to directly write these files for me, where to find them, or where to get the gcno file that's constructed at compile time; either with the bazel coverage tools or with my own home-rolled solution. Any suggestions?

[siddharthab]

In collect_coverage.sh, path to lcov is hardcoded. Can we make it configurable some how? Maybe just use PATH, or an environment variable. On darwin, it is not easy to install lcov to /usr/bin. Using brew or other package managers will install it in /usr/local/bin or some other location.

bazel/tools/test/collect_coverage.sh

Line 149 in 1441a3a

/usr/bin/lcov -c --no-external --ignore-errors graph \

[ulfjack]

@siddharthab that's fair. It's not obvious to me how best to do that. We probably need to create a 'coverage toolchain' to put everything in the right place, but we might get away with piggybacking on the cc_configure script for now.

[siddharthab]

Actually I realised that on macOS, we will mostly be taking the llvm coverage path. So this might not be that important. I was hitting this only because the stat command in the if clause has an extra flag for bsd stat.

[ulfjack]

I made some recent changes that should improve the llvm coverage path. I think they're not released yet. See #5036 (also contains rudimentary instructions for how to use it). Let me know if that works for you.

[siddharthab]

Yes, I was able to get coverage reports on macOS with llvm following your instructions with just one patch -- #5162

I also needed to remove sandbox directory segments from the paths in the profdata file but other than that, everything worked as you said.

[siddharthab]

A question: do you plan on keeping coverage.dat as lcov intermediate format. Are all languages expected to do the same? I am asking because some of us are trying to pipe coverage information from coverage.dat to end user presentation systems. But I am not sure which format to expect in coverage.dat.

[ulfjack]

I'm not sure. @iirina is currently working on this, and we'll send out a proposal soon.

[eugene-bulkin]

I'd second @siddharthab's concern from earlier, some environments still don't have /usr/bin/lcov available at all times, and it may be installed elsewhere.

[siddharthab]

@eugene-bulkin Note that lcov is only needed when using gcc as the compiler. So if your different environment happens to be macOS and you are using the default toolchain, or if you are using an LLVM toolchain, then you don't really need it.

[eugene-bulkin]

@siddharthab My environment, for example, does use gcc as the compiler, so it still might help some people to take it from an environment variable or toolchain.

[ulfjack]

There isn't a way to inject the path to lcov at the moment. It is straightforward to add an env variable for this, but I'd like to hear from Irina first.

[iirina]

I'm currently doing some performance measurements to see if it's worth keeping lcov or not. This will also have an impact on the coverage format that Bazel will output. I'll come back with data.

[robfig]

Any indications on whether lcov is here to stay? I'm trying to integrate code coverage into our CI pipeline and wondering if we should wait.

[ulfjack]

@iirina has a doc (which she unfortunately hasn't shared yet), where she proposes that we should go with cobertura XML as the format. We've been looking at gcovr to generate it from C/C++ coverage, and there are a lot of other tools available as well.

[iirina]

Here is the document. Please comment, any feedback is welcomed.

We'll gradually change the standard coverage format to Cobertura XML. Both lcov and Cobertura XML will be available for a transitioning period.

[th0br0]

Could the code coverage approach also consider an interface for setting up static analysis tools (e.g. clang-tidy)?

[robfig]

Very much looking forward to this. Will this new set of work include collecting baseline coverage?

https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/runtime/commands/CoverageCommand.java#L61

[iirina]

@th0br0 Static analysis was not considered as a part of the code coverage collection. There are no plans to support it right now.

@robfig Yes, it will be part of it. I opened #5716 to track progress if you want to.

[iirina]

I will close this issue since its scope is very wide. To track code coverage progress you can check:

• Bazel Code Coverage Roadmap (should be updated every quarter)
• Make C++ code coverage collection faster #5882: Make C++ code coverage collection faster
• Use gcov for collecting code coverage for gcc  #5880: Use gcov for collecting code coverage for gcc
• Use profdata for collecting code coverage for clang #5881: Use profdata for collecting code coverage for clang
• Have a general mechanism for collecting code coverage #5885: Have a general mechanism for collecting code coverage

Further issues have the coverage label.