Documentation improvement

[dalibor-matura]

The Readme contains just a minimal usage information and almost no examples.

I would like to get my standard Rust lib.rs project connected with CodeCov (https://codecov.io/), so I would like to gather the Test Coverage data with grcov and send it to CodeCov.

The only information provided on this topic in Readme is:

Coveralls/Codecov output

grcov ~/Documents/FD/mozilla-central/build -t coveralls -s ~/Documents/FD/mozilla-central --token YOUR_COVERALLS_TOKEN > coveralls.json

and I don't know what the directories should I used in my case instead of the

• ~/Documents/FD/mozilla-central/build
• ~/Documents/FD/mozilla-central should

It would be nice to provide more detailed description. I'm willing to help and I've already cloned this repo and will look/search through the codebase to get better understanding how it works.

Thanks for a cool tool ;)

[dalibor-matura]

Here's a related Rust Users forum topic I've just created: https://users.rust-lang.org/t/test-coverage-with-grcov/24851

[marco-c]

Thanks @dalibor-matura!

You can use the Travis or Appveyor config as examples (since we are collecting coverage for grcov itself).

[marco-c]

@calixteman improved the docs a little by adding an example of how to use grcov on Travis with #250.

[dalibor-matura]

Thanks for the info @marco-c and thanks to @calixteman for the documentation improvement: https://github.com/mozilla/grcov#grcov-with-travis

I already tried it a few days ago, but got into problems with the coverage itself: https://users.rust-lang.org/t/test-coverage-with-grcov/24851/8?u=dalibor, so I'm still using the cargo-tarpaulin.

[calixteman]

If you want to remove yellow lines, you can remove --branch option from grcov.
Here are few explanations on how ccov is working with -Zprofile option:
i) rust transforms files into LLVM IR and LLVM transforsm this IR in executable code
ii) when adding -Zprofile, the IR is instrumented on the LLVM side to add some counters on entry blocks and on branches between blocks (some gcno files are generated: they contain block & branch info). When the code is executed some gcda files are generated (they contain the values of the counters).
iii) rust (like a lot of languages) adds some extra code (for example some dtors are called at the end of a scope and often it explains why a closing '}' has a coverage (or not)).
iv) grcov just gets gcno & gcda to generate something readable.

A yellow line can be an interesting information:

fn first_is(&self, c: u8) -> bool {
        !self.buf.is_empty() && self.buf[0] == c
}

here this line is yellow because there is an extra check when getting self.buf[0] and of course we never fail and so never take the else branch of the check. This check is useless since we checked just before that the buf is not empty. So let's rewrite it to remove useless check and so have green lines:

fn first_is(&self, c: u8) -> bool {
        if let Some(cc) = self.buf.get(0) {
            *cc == c
        } else {
            false
        }
}

On an easy case like this, probably the optimizer in LLVM will be able to remove redundant checks, but sometimes it's pretty hard so it's probably better to try to write good code.
So finally, most of time a yellow line should be a green one (assert!(...) is often yellow because we don't test that the test itself is failing) but in some cases a yellow line is an interesting information we should take care about.

[ghost]

I have read the travis example and still can not make grov work for my local project (using cargo new --lib).
I can not figure out what ~/Documents/FD/mozilla-central/build is in my project. I tried to run grcov command with many path arguments: my project root folder, target/debug, target/debug/build and got the same error:

thread 'Producer' panicked at 'No input files found', \rustup.cargo\registry\src\github.com-1ecc6299db9ec823\grcov-0.5.0\src\producer.rs:501:5

note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.

[calixteman]

@Fubuchi, I added coverage for an other project:
https://github.com/mozilla/dump_syms/blob/master/.taskcluster.yml#L64-L82
The "env" section is important to have the correct flags to compile for ccov.
And the "command" section contains almost what you probably need.

[ghost]

@calixteman I tweak the command a bit to make it work on windows powershell and install WSL to get the genhtml tool.

[TomLouisKeller]

Took me a while to get this going, so I thought I'd share my progress.
This is an attempt at a minimal version with Rust, and I'm still not sure what all the flags mean.

Create a new project and go into the directory

cargo new project_name
cd project_name

Set the environment variables

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"

Build Rust project and run tests

cargo build
cargo test

Create a directory, in which we store the code coverage files

mkdir ccov

Add the needed files into a zip. (${PWD##*/} equates to the directory name)

zip -0 ccov/ccov.zip `find . \( -name "${PWD##*/}*.gc*" \) -print`

Generate the report.

grcov ccov/ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o ccov/lcov.info

Generate the html files. (genhtml is bundled with lcov, so you might have to install that)

genhtml -o ccov/ --show-details --highlight --ignore-errors source --legend ccov/lcov.info

Open the report in the browser.

firefox ccov/index.html

For your aliases.zsh

grcov-rust(){
  if [ ! $1 ]
  then
    project_name=$1
  else
    project_name=${PWD##*/}
  fi
  export CARGO_INCREMENTAL=0
  export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"
  cargo build
  cargo test
  mkdir ccov
  zip -0 ccov/ccov.zip `find . \( -name "$project_name*.gc*" \) -print`
  grcov ccov/ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore-dir "/*" -o ccov/lcov.info
  genhtml -o ccov/ --show-details --highlight --ignore-errors source --legend ccov/lcov.info
  firefox ccov/index.html
  unset CARGO_INCREMENTAL
  unset RUSTFLAGS
}

[calixteman]

You can try the new html output format: -t html and -o output_dir and give us some feedback on this output.

[ghost]

@calixteman is the html supported it in the 0.5.3 version? I install grcov for windows from cargo, result in version 0.5.0

[calixteman]

Yep it should be in 0.5.3:
7da9c4a

[calixteman]

@TomLouisKeller just a detail to add is that rust nightly is mandatory.

[ghost]

I have just tested the html output format, look very good 👍 . Hope the new version will come to crates.io soon

[marco-c]

I've released 0.5.1, 0.5.2, 0.5.3 and 0.5.4 (which changes --ignore-dir to just --ignore) on crates.io.

[TomLouisKeller]

@calixteman @marco-c Very nice, thank you =)

[gilescope]

These instructions are extremely helpful. Can someone please turn this issue into a documentation PR please?

[marco-c]

@gilescope can you? :)

[Nokel81]

When I run those instructions I get the following error from rustc (during the cargo build step)

  = note: Undefined symbols for architecture x86_64:
            "_CFMutableAttributedStringGetTypeID", referenced from:
                _$LT$core_foundation..attributed_string..CFMutableAttributedString$u20$as$u20$core_foundation..base..TCFType$GT$::type_id::hb12d0b020eb8a80d in libcore_foundation-bc4b5b818f36d2e0.rlib(core_foundation-bc4b5b818f36d2e0.core_foundation.5lhz0o74-cgu.0.rcgu.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

[calixteman]

@Nokel81, probably related to:
rust-lang/rust#63047

[Nokel81]

@calixteman Thanks, will try in a linux VM

[fgimian]

Hey guys, is there currently any way to print a report to the terminal? 😊

[gilescope]

@fgimian the top level is printed to the terminal when genhtml is run if that helps - E.g.:

Overall coverage rate:
  lines......: 73.5% (75 of 102 lines)
  functions..: 45.5% (10 of 22 functions)

[Nokel81]

grcov supports "covdir" which is a json format and one of the top level fields is coveragePercentage.

Also, it doesn't seem to be documented but grcov does also seen to be able to generate the html reports directly (without using genhtml)

[gilescope]

Nice - less dependencies makes happier users. Yep that -t html seems to work fine. Great.

[TriplEight]

grcov to codecov.io with GitLab CI:

codecov:
  stage:                           test
  <<:                              *docker-env
  variables:
    CARGO_INCREMENTAL:             0
    RUSTFLAGS:                     "-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off"
      script:
    - cargo clean
    - cargo test --verbose --all-features --no-fail-fast --workspace
    - cargo build --verbose --all-features --workspace
    - grcov ./target -s . -t coveralls+ --guess-directory-when-missing --llvm --branch --ignore-not-existing --ignore "/*"
        --token "$CODECOV_TOKEN" --commit-sha "$CI_COMMIT_SHA" --output-file ccov.json
    - bash <(curl -s https://codecov.io/bash) -t "$CODECOV_TOKEN" -X gcov -f ccov.json

According to doc we have to use these (CARGO_INCREMENTAL and RUSTFLAGS), however it failed tests with -Zno-landing-pads.
I've had significantly better lines discoverability with cargo-taurpalin more than 10% coverage difference than with grcov, but the former doesn't work within docker properly.

[gilescope]

Glad to here it works in docker. Could be methodology differences on whitespace treatment or maybe branch versus line? (I am just guessing)

_{Sent with GitHawk}

[marco-c]

You could try source-based coverage, I've recently added support for it in grcov (version 0.6.0). Short explanation at https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html, docs at https://github.com/mozilla/grcov#example-how-to-generate-source-based-coverage-for-a-rust-project, full example at https://github.com/marco-c/rust-code-coverage-sample.