An Elixir library that reports test coverage statistics, with the option to post to coveralls.io service. It uses Erlang's cover to generate coverage information, and posts the test coverage results to coveralls.io through the JSON API.
The following are example projects.
- coverage_sample is for Travis CI.
- github_coverage_sample is for GitHub Actions.
- circle_sample is for CircleCI .
- semaphore_sample is for Semaphore CI.
- excoveralls_umbrella is for umbrella project.
- gitlab_sample is for GitLab CI (using GitLab CI feature instead of coveralls.io).
- gitlab_parallel_sample is for GitLab CI (with parallel option).
- drone_sample is for Drone CI.
- excoveralls_post_sample is for coveralls.post usage example.
Add the following parameters.
test_coverage: [tool: ExCoveralls]for using ExCoveralls for coverage reporting.test_coverage: [tool: ExCoveralls, export: "cov"]for exporting data tocover/cov.coverdatapreferred_cli_env: [coveralls: :test]for runningmix coverallsin:testenv by default- It's an optional setting for skipping
MIX_ENV=testpart when executingmix coverallstasks.
- It's an optional setting for skipping
test_coverage: [test_task: "espec"]if you use Espec instead of default ExUnit.:excoverallsin the deps function.Application.put_env(:excoveralls, :base_path, "/bash/path")an optional config if you want to set the application root path explicitly. By default this is the directory that the mix.exs file is in.
def project do
[
app: :excoveralls,
version: "1.0.0",
elixir: "~> 1.0.0",
deps: deps(),
test_coverage: [tool: ExCoveralls],
preferred_cli_env: [
coveralls: :test,
"coveralls.detail": :test,
"coveralls.post": :test,
"coveralls.html": :test,
"coveralls.cobertura": :test
]
# if you want to use espec,
# test_coverage: [tool: ExCoveralls, test_task: "espec"]
]
end
defp deps do
[
{:excoveralls, "~> 0.18", only: :test},
]
endNote on umbrella application: If you want to use Excoveralls within an umbrella project, every apps must have
test_coverage: [tool: ExCoveralls] in the mix.exs of each app.
Note: If you're using earlier than elixir v1.3, MIX_ENV=test or preferred_cli_env may be required for running mix tasks. Refer to PR#96 for the details.
- ExCoveralls
- Settings
- Usage
- Mix Tasks
- [mix coveralls] Show coverage
- [mix coveralls.travis] Post coverage from travis
- [mix coveralls.github] Post coverage from GitHub Actions
- [mix coveralls.circle] Post coverage from circle
- [mix coveralls.semaphore] Post coverage from semaphore
- [mix coveralls.drone] Post coverage from drone
- [mix coveralls.post] Post coverage from any host
- [mix coveralls.detail] Show coverage with detail
- [mix coveralls.html] Show coverage as HTML report
- [mix coveralls.json] Show coverage as JSON report
- [mix coveralls.xml] Show coverage as XML report
- [mix coveralls.cobertura] Show coverage as Cobertura report
- [mix coveralls.lcov] Show coverage as lcov report (Experimental)
- coveralls.json
- Other Considerations
- Mix Tasks
- License
Run the MIX_ENV=test mix coveralls command to show coverage information on localhost.
This task locally prints out the coverage information. It doesn't submit the results to the server.
$ MIX_ENV=test mix coveralls
...
----------------
COV FILE LINES RELEVANT MISSED
100.0% lib/excoveralls/general.ex 28 4 0
75.0% lib/excoveralls.ex 54 8 2
94.7% lib/excoveralls/stats.ex 70 19 1
100.0% lib/excoveralls/poster.ex 16 3 0
95.5% lib/excoveralls/local.ex 79 22 1
100.0% lib/excoveralls/travis.ex 23 3 0
100.0% lib/mix/tasks.ex 44 8 0
100.0% lib/excoveralls/cover.ex 32 5 0
[TOTAL] 94.4%
----------------Specifying the --help option displays the options list for available tasks.
Usage: mix coveralls <Options>
Used to display coverage
<Options>
-h (--help) Show helps for excoveralls mix tasks
Common options across coveralls mix tasks
-o (--output-dir) Write coverage information to output dir.
-u (--umbrella) Show overall coverage for umbrella project.
-v (--verbose) Show json string for posting.
--subdir Git repo sub directory: This will be added to the the front of file path, use if your covered
file paths reside within a subfolder of the git repo. Example: If your source file path is
"test.ex", and your git repo root is one directory up making the file's relative path
"src/lib/test.ex", then the sub directory should be: "src/lib" (from coveralls.io)
--rootdir This will be stripped from the file path in order to resolve the relative path of this repo's
files. It should be the path to your git repo's root on your CI build environment. This is not
needed if your source file path is already relative. It's used to pull the source file from the
github repo, so must be exact. Example: If your source file path is "/home/runs/app/test.ex",
and your git repo resides in "app", then the root path should be: "/home/runs/app/" (from
coveralls.io)
--flagname Job flag name which will be shown in the Coveralls UI
--import-cover Directory from where '.coverdata' files should be imported and their results added to the report.
Coverdata is imported after tests are run.
Usage: mix coveralls.detail [--filter file-name-pattern]
Used to display coverage with detail
[--filter file-name-pattern] can be used to limit the files to be displayed in detail.
Usage: mix coveralls.travis [--pro]
Used to post coverage from Travis CI server.
Usage: mix coveralls.github
Used to post coverage from [GitHub Actions](https://github.com/features/actions).
Usage: mix coveralls.post <Options>
Used to post coverage from local server using token.
The token should be specified in the argument or in COVERALLS_REPO_TOKEN
environment variable.
<Options>
-t (--token) Repository token ('REPO TOKEN' of coveralls.io)
-n (--name) Service name ('VIA' column at coveralls.io page)
-b (--branch) Branch name ('BRANCH' column at coveralls.io page)
-c (--committer) Committer name ('COMMITTER' column at coveralls.io page)
-m (--message) Commit message ('COMMIT' column at coveralls.io page)
-s (--sha) Commit SHA (required when not using Travis)
--build Service number ('BUILDS' column at coveralls.io page)
--parallel coveralls.io 'parallel' option (See coveralls.io API Reference)Specify mix coveralls.travis as the build script in the .travis.yml and explicitly set the MIX_ENV environment to TEST.
This task submits the result to Coveralls when the build is executed on Travis CI.
language: elixir
elixir:
- 1.2.0
otp_release:
- 18.0
env:
- MIX_ENV=test
script: mix coveralls.travisIf you're using Travis Pro for a private
project, Use coveralls.travis --pro and ensure your coveralls.io
repo token is available via the COVERALLS_REPO_TOKEN environment
variable.
[mix coveralls.github] Post coverage from GitHub Actions
Specify mix coveralls.github as the build script in the GitHub action YML file and explicitly set the MIX_ENV environment to test and add GITHUB_TOKEN with the value of {{ secrets.GITHUB_TOKEN }}, this is required because is used internally by coveralls.io to check the action and add statuses.
The value of secrets.GITHUB_TOKEN is added automatically inside every GitHub action, so you not need to assign that.
This task submits the result to Coveralls when the build is executed via GitHub actions and add statuses in the checks of github.
on: push
jobs:
test:
runs-on: ubuntu-latest
name: OTP ${{matrix.otp}} / Elixir ${{matrix.elixir}}
strategy:
matrix:
otp: [21.3.8.10, 22.1.7]
elixir: [1.8.2, 1.9.4]
env:
MIX_ENV: test
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/[email protected]
- uses: erlef/setup-beam@v1
with:
otp-version: ${{matrix.otp}}
elixir-version: ${{matrix.elixir}}
- run: mix deps.get
- run: mix coveralls.githubSpecify mix coveralls.circle in the circle.yml.
This task is for submitting the result to the coveralls server when Circle-CI build is executed.
test:
override:
- mix coveralls.circleEnsure your coveralls.io repo token is available via the COVERALLS_REPO_TOKEN environment
variable.
Specify mix coveralls.semaphore in the build command prompt for instructions in semaphore.
This task is for submitting the result to the coveralls server when Semaphore-CI build is executed.
mix coveralls.semaphore
Ensure your coveralls.io repo token is available via the COVERALLS_REPO_TOKEN environment
variable.
Specify mix coveralls.drone in the .drone.yml.
This task is for submitting the result to the coveralls server when the Drone build is executed.
You will also need to add your coveralls repo token as a secret to the drone project:
drone secret add --repository=your-namespace/your-project --name=coveralls_repo_token --value=xyz
pipeline:
build:
secrets: [ coveralls_repo_token ]
commands:
- mix coveralls.droneAcquire the repository token of coveralls.io in advance, and run the mix coveralls.post command.
It is for submitting the result to coveralls server from any host.
The token can be specified as a mix task option (--token), or as an environment variable (COVERALLS_REPO_TOKEN).
MIX_ENV=test mix coveralls.post --token [YOUR_TOKEN] --branch "master" --name "local host" --committer "committer name" --sha "fd80a4c" --message "commit message"
....................................................................................................
Finished in 6.3 seconds (0.7s on load, 5.6s on tests)
100 tests, 0 failures
Randomized with seed 800810
Successfully uploaded the report to 'https://coveralls.io'.For the detailed option description, check mix coveralls --help task.
This task displays coverage information at the source-code level with colored text.
Green indicates a tested line, and red indicates lines which are not tested.
When reviewing many source files, pipe the output to the less program (with the -R option for color) to paginate the results.
$ MIX_ENV=test mix coveralls.detail | less -R
...
----------------
COV FILE LINES RELEVANT MISSED
100.0% lib/excoveralls/general.ex 28 4 0
...
[TOTAL] 94.4%
--------lib/excoveralls.ex--------
defmodule ExCoveralls do
@moduledoc """
Provides the entry point for coverage calculation and output.
This module method is called by Mix.Tasks.Test
...Also, displayed source code can be filtered by specifying arguments (it will be matched against the FILE column value). The following example lists the source code only for general.ex.
$ MIX_ENV=test mix coveralls.detail --filter general.ex
...
----------------
COV FILE LINES RELEVANT MISSED
100.0% lib/excoveralls/general.ex 28 4 0
...
[TOTAL] 94.4%
--------lib/excoveralls.ex--------
defmodule ExCoveralls do
@moduledoc """
Provides the entry point for coverage calculation and output.
This module method is called by Mix.Tasks.Test
...This task displays coverage information at the source-code level formatted as an HTML page.
The report follows the format inspired by HTMLCov from the Mocha testing library in JS.
Output to the shell is the same as running the command mix coveralls (to suppress this output, add "print_summary": false to your project's coveralls.json file). In a similar manner to mix coveralls.detail, reported source code can be filtered by specifying arguments using the --filter flag.
$ MIX_ENV=test mix coveralls.html
Output reports are written to cover/excoveralls.html by default, however, the path can be specified by overwriting the "output_dir" coverage option.
Custom reports can be created and utilized by defining template_path in coveralls.json. This directory should
contain an eex template named coverage.html.eex.
This task displays coverage information at the source-code level formatted as a JSON document.
The report follows a format supported by several code coverage services, including Codecov and Code Climate.
Output to the shell is the same as running the command mix coveralls (to suppress this output, add "print_summary": false to your project's coveralls.json file). In a similar manner to mix coveralls.detail, reported source code can be filtered by specifying arguments using the --filter flag.
Upload a coverage report to Codecov using their bash uploader or to Code Climate using their test-reporter.
Output reports are written to cover/excoveralls.json by default, however, the path can be specified by overwriting the "output_dir" coverage option.
This task displays coverage information at the source-code level formatted as a XML document.
The report follows a format supported by several code coverage services like SonarQube.
Output to the shell is the same as running the command mix coveralls (to suppress this output, add "print_summary": false to your project's coveralls.json file). In a similar manner to mix coveralls.detail, reported source code can be filtered by specifying arguments using the --filter flag.
Output reports are written to cover/excoveralls.xml by default, however, the path can be specified by overwriting the "output_dir" coverage option.
This task displays coverage information at the source-code level formatted as a Cobertura document.
The report follows a format supported by Gitlab code coverage visualization.
Output to the shell is the same as running the command mix coveralls (to suppress this output, add "print_summary": false to your project's coveralls.json file). In a similar manner to mix coveralls.detail, reported source code can be filtered by specifying arguments using the --filter flag.
Output reports are written to cover/cobertura.xml by default, however, the path can be specified by overwriting the "output_dir" coverage option.
This task displays coverage information at the line level formatted as a lcov.
The report follows a format supported by several code coverage services like VSCode extension(ryanluker.vscode-coverage-gutters).
Output to the shell is the same as running the command mix coveralls (to suppress this output, add "print_summary": false to your project's coveralls.json file). In a similar manner to mix coveralls.detail, reported source code can be filtered by specifying arguments using the --filter flag.
Output reports are written to cover/lcov.info by default, however, the path can be specified by overwriting the "output_dir" coverage option.
coveralls.json provides settings for excoveralls.
The default coveralls.json file is stored in deps/excoveralls/lib/conf, and custom coveralls.json files can be placed in the mix project root. The custom definition is prioritized over the default one (if definitions in the custom file are not found, then the definitions in the default file are used).
Stop words defined in coveralls.json will be excluded from the coverage calculation. Some kernel macros defined in Elixir are not considered "covered" by Erlang's cover library. It can be used for excluding these macros, or for any other reasons. The words are parsed as regular expression.
If you want to exclude/ignore files from the coverage calculation add the skip_files key in the coveralls.json file. skip_files takes an array of file paths, for example:
{
"skip_files": [
"folder_to_skip",
"folder/file_to_skip.ex"
]
}Path should contain a string that can be compiled to Elixir regex, you can test them running Regex.compile("your_path") in your iex shell.
Note that this doesn't work directly in an umbrella project. If you need to exclude files within an app, you should create a separate coveralls.json at the root of the app's folder and add a skip_files key to that file. Paths should be relative to that file, not the umbrella project.
When using in umbrella projects the default report may trim files names when viewing report in terminal.
If you want to change the column width used for file names add the file_column_width key to the terminal_options key in the coveralls.json, for example:
{
"terminal_options": {
"file_column_width": 40
}
}If you want to see only the total coverage without a table of each file, set the print_files option
to false:
{
"terminal_options": {
"print_files": false
}
}treat_no_relevant_lines_as_covered- By default, coverage for [files with no relevant lines] are displayed as 0% for aligning with coveralls.io behavior. But, if
treat_no_relevant_lines_as_coveredis set totrue, it will be displayed as 100%.
- By default, coverage for [files with no relevant lines] are displayed as 0% for aligning with coveralls.io behavior. But, if
output_dir- The directory which the HTML report will output to. Defaulted to
cover/.
- The directory which the HTML report will output to. Defaulted to
template_path- A custom path for html reports. This defaults to the htmlcov report in the excoveralls lib.
minimum_coverage- When set to a number greater than 0, this setting causes the
mix coverallsandmix coveralls.htmltasks to exit with a status code of 1 if test coverage falls below the specified threshold (defaults to 0). This is useful to interrupt CI pipelines with strict code coverage rules. Should be expressed as a number between 0 and 100 signifying the minimum percentage of lines covered.
- When set to a number greater than 0, this setting causes the
html_filter_full_covered- A boolean, when
truefiles with 100% coverage are not shown in the HTML report. Default tofalse.
- A boolean, when
floor_coverage- A boolean, when
falsecoverage values are ceiled instead of floored, this means that a project with some lines that are not covered can still have a total 100% coverage. Default totrue.
- A boolean, when
Example configuration file:
{
"default_stop_words": [
"defmodule",
"defrecord",
"defimpl",
"def.+(.+\/\/.+).+do"
],
"custom_stop_words": [
],
"coverage_options": {
"treat_no_relevant_lines_as_covered": true,
"output_dir": "cover/",
"template_path": "custom/path/to/template/",
"minimum_coverage": 90,
"xml_base_dir": "custom/path/for/xml/reports/",
"html_filter_full_covered": true
}
}Use comments coveralls-ignore-start and coveralls-ignore-stop to ignore certain lines from code coverage calculation.
defmodule MyModule do
def covered do
end
# coveralls-ignore-start
def ignored do
end
# coveralls-ignore-stop
endUse comment coveralls-ignore-next-line to ignore only the next line.
defmodule MyModule do
def covered do
# coveralls-ignore-next-line
"ignored"
"covered"
end
endTo remove OTP warnings about modules or specific logging, you can copy the cover.erl file under src/ of your Elixir project and modify it to remove the warnings, as a tentative solution.
-
Remove the 2 lines below to remove the "WARNING: Module already imported from ..." log https://github.com/erlang/otp/blob/131398b54cca5f1ae95ed268274936d2efde8c39/lib/tools/src/cover.erl#L1553-L1554
-
If you do not want the imported info cluttering your test output, replace the function in https://github.com/erlang/otp/blob/131398b54cca5f1ae95ed268274936d2efde8c39/lib/tools/src/cover.erl#L1520-L1525 with
imported_info(_Text,_Module,_Imported) ->
ok.
ExCoveralls can include .coverdata files in the result of the current test run through the --import-cover flag. This can be used to include coverage data from partitioned tests or integration tests that may run in a subprocess, for instance.
Coverage data is generated when running mix test --cover, optionally with the --export-coverage flag to specify an output name.
$ mix test --only integration --cover --export-coverage integration-coverage
Excluding tags: [:test]
Including tags: [:integration]
... test run omitted ...
# Coverage data written to cover/integration-coverage.coverdata
# Report coverage, do not run integration tests
$ mix coveralls --exclude integration
Excluding tags: [:integration]
... test run omitted ...
----------------
COV FILE LINES RELEVANT MISSED
...
[TOTAL] 80.2% # <-- This result does not include coverage from integration tests
----------------
# Report coverage, do not run integration tests, but include previously written coverdata
$ mix coveralls --exclude integration --import-cover cover
Excluding tags: [:integration]
... test run omitted ...
----------------
COV FILE LINES RELEVANT MISSED
...
[TOTAL] 95.3% # <-- This result now includes coverage from integration tests
----------------Coverage data is imported after tests are run.
See the mix test Coverage documentation for more information on .coverdata.
You can customize the HTTP options used by :httpc when posting results. The example below shows how to specify a custom cacertfile:
config :excoveralls,
http_options: [
timeout: 10_000,
ssl: [
# Refer to the secure coding guide:
# https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/inets
verify: :verify_peer,
depth: 2,
customize_hostname_check: [
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
],
cacertfile: to_charlist(System.fetch_env!("TEST_COVERAGE_CACERTFILE"))
]
By default, ExCoveralls uses the cacertfile from castore when the dependency is installed. If it's not available and you're running Erlang 25 or later, the system will attempt to use the OS certificates via :public_key.cacerts_load/0.
- If mock library is used, it will show some warnings during execution.
- In case Erlang clashes at
mix coveralls, executingmix testin advance might avoid the error. - When erlang version 17.3 is used, an error message
(MatchError) no match of right hand side value: ""can be shown. Refer to issue #14 for the details.
- It might not work well on projects which handle multiple project (Mix.Project) files.
- Needs improvement on file-path handling.
This source code is licensed under the MIT license. Copyright (c) 2013-present, parroty.