Implement coverage reports using codecov #143
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cargo-llvm-cov is used for generating coverage reports.
The library supports different report types which all have their own pros and cons:
--lcov
- reports inlcov
format which are supported by Codecov. They support only line level coverage and will not provide any info on regions. Example report can be seen here.--codecov
- Codecov's custom format which somewhat supports region coverage but not fully. Example report can be seen here. Notice that some lines are colored in yellow - which means they are partially covered but exact regions within lines are not highlighted.--html
- html output which is provided byllvm-cov
. This output is the most informative as it highlights covered and uncovered regions. I pushed an example of this to Github Pages here. Unfortunately, Codecov does not support such outputs. We have an option to push these reports manually to some Github Pages repo during PR review. Or we can host these assets in Github Artifacts (GA). GA, however, will not make it easy to preview generated HTML - users will have to download the assets and open locally. More convenient preview in GA is blocked by this issue.Given the above limitations, I propose using
--codecov
based approach which this PR implements.I am not integrating reports provided by
cargo fuzz coverage
into this yet and planning to do it in a separate PR once we agree on the initial approach.cargo fuzz coverage
generates an output which is closer to--lcov
mentioned above. There is a hack that I think can be implemented to make it work with--codecov
reports.