//src/test/shell/bazel:bazel_cc_code_coverage_test is failing in Bazel postsubmit

[meteorcloudy]

Build: https://buildkite.com/bazel/bazel-bazel/builds/20593#018316e3-41c1-4899-b92d-4f2f8dd3f468
Test log: https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/01831621-521b-4076-8bbf-169cf72b957b/src/test/shell/bazel/bazel_cc_code_coverage_test/test_attempts/attempt_1.log

/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/sandbox/linux-sandbox/6514/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/test/shell/bazel/bazel_cc_code_coverage_test.runfiles/io_bazel/src/test/shell/bazel/bazel_cc_code_coverage_test: line 277: gcov_location: unbound variable
ls: cannot access '/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/sandbox/linux-sandbox/6514/execroot/io_bazel/_tmp/de074246e1edcaf3b1945672957f04d1/workspace/coverage_dir/*a.gcda.gcov.json.gz': No such file or directory

[meteorcloudy]

Turns out it's caused by the docker image update for gcr.io/bazel-public/ubuntu2004-java11, temporarily rollbacked the latest tag. Still investigating what has changed.

[meteorcloudy]

To reproduce

docker run -it gcr.io/bazel-public/ubuntu2004-java11@sha256:5e7de0eb4365bf1524d8bab06d0b36b16fbb5e6aa587d1b7e7fcdae0598b5010 
mkdir workdir
cd workdir
git clone https://github.com/bazelbuild/bazel.git
cd bazel
bazel test --sandbox_debug //src/test/shell/bazel:bazel_cc_code_coverage_test --test_output=all

[meteorcloudy]

@c-mita @oquenchil Can you help debug why a.gcda.gcov.json.gz was not generated?

[meteorcloudy]

Since the test is broken with latest Ubuntu 20.04 LTS release (perhaps due to different g++ or gcov version), we should fix the test. Therefore we'll disable the test for now to unblock docker container upgrade.

[c-mita]

Failures in 20.04 can be explained by this change in GCC: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=1f2bb38a85710f650d1ea87f0765cb50e19c3212

With this change, the filenames for output gcda files started being mangled (sometimes, but for the purposes of this test that seems to actually be "always"). This change was introduced in GCC 9.1.0
The documentation is here: https://gcc.gnu.org/onlinedocs/gcc-9.4.0/gcc/Instrumentation-Options.html#index-fprofile-dir

Except this test only appears to have started failing recently with the latest LTS release, during which GCC was updated from 9.3.0 to 9.4.0. I cannot explain why it would have been passing before.

Later versions of GCC also introduce some changes.

Given a simple project structure:

project
|-- build
|-- data
`-- src
    `-- main.cc

And the following compile and run commands.

$ cd project
$ g++ -fprofile-arcs -ftest-coverage -fprofile-dir=data src/main.cc -o build/main
$ ./build/main

What we end up with will vary significantly depending on the version of GCC.

GCC 8.4.0

With this version, there's a bug where the gcno file does not end up in the expected location (next to the object file), but rather gets dumped out in the current working directory.

$ tree project
project
|-- build
|   `-- main
|-- data
|   `-- main.gcda
|-- main.gcno
`-- src
    `-- main.cc

GCC 9.4.0

With this version, we start seeing mangled paths. The full absolute path for the gcno file appears to be taken and converted into a "friendly" form (i.e. separators are replaced with #)

$ tree project
project
|-- build
|   `-- main
|-- data
|   `-- #home#c-mita#project#main.gcda
|-- main.gcno
`-- src
    `-- main.cc

GCC 10.4.0

This is the same as 9.4.0

GCC 11.4.0

The gcno file is now put next to the compiled object. This changes the name of the gcda file slightly too.

$ tree project
project
|-- build
|   |-- main
|   `-- main.gcno
|-- data
|   `-- #home#c-mita#project#build#main.gcda
`-- src
    `-- main.cc

GCC 12.1.0

This is the same as 11.4.0

The documentation for -fprofile-dir suggests that, if the output object name is absolute, the path is not mangled.
In our simple case, as far as I can tell, this is a lie.

$ g++ -fprofile-arcs -ftest-coverage -fprofile-dir=$PWD/data $PWD/src/main.cc -o $PWD/build/main
./build/main

Converting everything to a full path yields identical outputs for versions 8, 9 and 10.
With 11 and 12 however, something different happens.

$ tree project
project
|-- build
|   |-- main
|   `-- main.gcno
|-- data
|   `-- home
|       `-- c-mita
|           `-- project
|               `-- build
|                   `-- main.gcda
`-- src
    `-- main.cc

That's three distinct cases to handle for just this test.

I do not yet know how this interacts with Bazel's regular C++ coverage logic; collect_cc_coverage.sh is certainly not expecting mangled gcda file names or them to be in nested directories.

[c-mita]

Removing the -fprofile-dir flag stops all of the name mangling on the gcda file, which is fortuitous, because Bazel does not pass this flag.

There is still some differences in behaviour as to exactly where the main.gcda file ends up, but it always seems to be created wherever the gcno file was created.

[c-mita]

A quick test with GCC 9.3.0 I think explains why this test passed before:
With the same test project as before

$ cd project
$ g++ -fprofile-arcs -ftest-coverage -fprofile-dir=data src/main.cc -o build/main
$ ./build/main
$ tree .
.
|-- build
|   `-- main
|-- data
|   `-- #home#c-mita#project#main.gcda
|-- #home#c-mita#project#main.gcno
`-- src
    `-- main.cc

With 9.3.0, both the gcno and gcda file names are mangled, but the gcno file can be found by the test script because it's in the expected location and is still matched by the wildcard in the ls command used to find it, and the gcda file is found by the collect_cc_coverage script because its name matches that of the gcno file.

With the move to 9.4.0 the latter is no longer true, so collect_cc_coverage never finds the gcda file and never runs gcov.

[fmeum]

@bazel-io flag

[fmeum]

@meteorcloudy This issue also affects the pipeline for the release-5.3.1 branch, see #16253.

[meteorcloudy]

@bazel-io fork 5.3.1