Prototype support for llvm coverage

[ulfjack]

https://clang.llvm.org/docs/SourceBasedCodeCoverage.html

[iirina]

This looks reasonable. Does it work for multiple tests? Did you try to generate the HTML report?

[ulfjack]

Right now, it's hard-coding llvm-profdata, so we shouldn't merge it like this. I think we should merge #5040 first and then 'abuse' GCOV to pass the right binary name to the collect-coverage.sh script. You can't use both gcov and llvm coverage in the same invocation anyway.

With that change (and a small change here to use GCOV), usage is like this:
BAZEL_USE_LLVM_NATIVE_COVERAGE=1 GCOV=llvm-profdata-3.9 CC=clang-3.9 bazel coverage --instrumentation_filter=src/main/cpp src/test/cpp:all

The patch works with sandboxing, i.e., it successfully copies the coverage data out of the sandbox to the local workspace.

I successfully generated an HTML report at least for an individual test, like this:
(LLVM_COV_CMD="llvm-cov-3.9 show -format=html -output-dir=$(pwd)/report/ -instr-profile bazel-out/k8-fastbuild/testlogs/src/test/cpp/option_processor_test/coverage.dat bazel-out/k8-fastbuild/bin/src/test/cpp/option_processor_test $(cat bazel-out/k8-fastbuild/bin/src/test/cpp/option_processor_test.runfiles_manifest | cut -d' ' -f 2 | egrep ".so$" | xargs -n 1 -I xxx echo -n "-object xxx ")" && cd bazel-os-bazel && $LLVM_COV_CMD)

The HTML report partially ignores the --instrumentation_filter option - it shows all the files in the HTML view, even those that weren't covered. There's probably a way to make it work by changing the llvm-cov command - we have a list of all the files that should be covered.

[hlopko]

My machine has llvm-profdata-3.8 and llvm-profdata-4.0 :) Quick googling around suggests that its only llvm-profdata on osx. To make it run correctly on osx with xcode installed I read we should invoke it via xcrun (xcrun llvm-profdata).

One solution that we can use is to generate this file similarly to how crosstool is generated and expect users to provide relevant env variables. Thoughts?

[ulfjack]

I've changed the patch to use the GCOV variable that's now set by cc_configure (since 9566f67 is in). Ideally, we'd make cc_configure automatically detect that it's configuring clang and which version, and set GCOV accordingly, including on OSX. For now, it's at least possible to use the LLVM coverage implementation.

[ulfjack]

I've also added usage instructions to the change description.

[hlopko]

LGTM.

[hlopko]

Testing cc_configure would be awesome. But the best approach I thought of is to have multiple CI slaves, each set up with different combination and versions of gcc/clang/gcov/libstdc++/libc++/full Xcode/command line tools only, and running all of this on ubuntu debian and mac :)

[ulfjack]

We could mock out the entire OS?

…

On Wed, Apr 25, 2018 at 5:16 PM Marcel Hlopko ***@***.***> wrote: Testing cc_configure would be awesome. But the best approach I thought of is to have multiple CI slaves, each set up with different combination and versions of gcc/clang/gcov/libstdc++/libc++/full Xcode/command line tools only, and running all of this on ubuntu debian and mac :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5036 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHA9YT-FsiGaSEMTMEMGVQ3FkPKSZ_7Nks5tsJNLgaJpZM4TYa31> .