Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code coverage reports with covimerage #1586

Merged
merged 2 commits into from
Dec 7, 2017
Merged

Code coverage reports with covimerage #1586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3c782c8
Code coverage reports with covimerage
arp242 Nov 26, 2017
e860c37
Small improvements based on blueyed's feedback
arp242 Nov 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .coveragerc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[run]
plugins = covimerage
data_file = .coverage.covimerage
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
doc/tags
.DS_Store
/doc/tags
/.coverage.covimerage
/coverage.xml
8 changes: 4 additions & 4 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ notifications:
email: false
matrix:
include:
- env: SCRIPT=test VIM_VERSION=vim-7.4
- env: SCRIPT=test VIM_VERSION=vim-8.0
- env: SCRIPT=test VIM_VERSION=nvim
- env: SCRIPT="test -c" VIM_VERSION=vim-7.4
- env: SCRIPT="test -c" VIM_VERSION=vim-8.0
- env: SCRIPT="test -c" VIM_VERSION=nvim
- env: SCRIPT=lint VIM_VERSION=vim-8.0
install:
- ./scripts/install-vim $VIM_VERSION
- pip install --user vim-vint
- pip install --user vim-vint covimerage codecov
script:
- ./scripts/$SCRIPT $VIM_VERSION
27 changes: 22 additions & 5 deletions scripts/run-vim
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ set -euC
vimgodir=$(cd -P "$(dirname "$0")/.." > /dev/null && pwd)
cd "$vimgodir"

coverage=0
while getopts "c" option; do
case "$option" in
c) coverage=1; ;;
esac
done
shift $((OPTIND - 1))

if [ -z "${1:-}" ]; then
echo "unknown version: '${1:-}'"
echo "First argument must be 'vim-7.4', 'vim-8.0', or 'nvim'."
Expand All @@ -23,11 +31,20 @@ if [ ! -f "$dir/bin/vim" ]; then
exit 1
fi

$dir/bin/vim --noplugin -u NONE -N \
+"set shm+=WAFI rtp=$dir/share/vim/vimgo packpath=$dir/share/vim/vimgo,$vimgodir" \
+'filetype plugin indent on' \
+'packloadall!' \
"$@"
if [ $coverage -eq 1 ]; then
covimerage -q run --report-file /tmp/vim-go-test/cov-profile.txt --append \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw: why do you use -q?
I think not making it quiet should result in sensible output, and e.g. warnings in case some dict functions cannot be found etc - not sure if this suppressed with one -q only already.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general rule, I dislike programs that output a lot of informational messages. Basically, the "rule of silence" from The Art of Unix Programming.

Without the -q flag it adds a lot of output:

$ ./scripts/test -c vim-8.0
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/autoload/go/tags_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.j5euwlyg' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.j5euwlyg.
Writing coverage file .coverage.covimerage.
ok   tags_test.vim              0.171777s / 2 tests
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/autoload/go/fillstruct_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.d2794fk1' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.d2794fk1.
Writing coverage file .coverage.covimerage.
ok   fillstruct_test.vim        0.742317s / 1 tests
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/autoload/go/tool_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.dy8bfk6j' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.dy8bfk6j.
Writing coverage file .coverage.covimerage.
ok   tool_test.vim              0.025166s / 2 tests
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/autoload/go/impl_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.rnk3t2wt' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.rnk3t2wt.
Writing coverage file .coverage.covimerage.
ok   impl_test.vim              0.056763s / 2 tests
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/autoload/go/def_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.dxp6b0f7' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.dxp6b0f7.
Writing coverage file .coverage.covimerage.
ok   def_test.vim               0.007708s / 2 tests
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/autoload/go/fmt_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.z1leipmj' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.z1leipmj.
Writing coverage file .coverage.covimerage.
ok   fmt_test.vim               0.050436s / 3 tests
Running cmd: /tmp/vim-go-test/vim-8.0-install/bin/vim --noplugin -u NONE -N '+set shm+=WAFI rtp=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo packpath=/tmp/vim-go-test/vim-8.0-install/share/vim/vimgo,/home/martin/.vim/pack/plugins/start/vim-go' '+filetype plugin indent on' '+packloadall!' -e '+silent e /home/martin/.vim/pack/plugins/start/vim-go/test/gopath_test.vim' '+let g:test_verbose=0' -S ./scripts/runtest.vim --cmd 'profile start /tmp/covimerage.profile.rjn2eh0c' --cmd 'profile! file ./*' (in /home/martin/.vim/pack/plugins/start/vim-go)
Parsing profile file /tmp/covimerage.profile.rjn2eh0c.
Writing coverage file .coverage.covimerage.
ok   gopath_test.vim            0.294958s / 2 tests

All tests PASSED

We don't have a lot of tests (yet), but this should hopefully increase by a lot soon.

I know this is only run on the CI, but still – it makes it harder to see what failed, and what output is from the test/code, and what is from the tooling, especially for contributors who do not regularly contribute and are unfamiliar with the testing system.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see logger.INFO is only used for a handful of informational messages, by the way: https://github.com/Vimjas/covimerage/search?utf8=%E2%9C%93&q=LOGGER.info&type=

So this should be fine, I think.

Copy link

@blueyed blueyed Nov 28, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see.
I think using warning as the default level would make sense then - but it would change the meaning of -q then.
I'll consider adding an option -l / --loglevel to give this explicitly.
Vimjas/covimerage#25

$dir/bin/vim --noplugin -u NONE -N \
+"set shm+=WAFI rtp=$dir/share/vim/vimgo packpath=$dir/share/vim/vimgo,$vimgodir" \
+'filetype plugin indent on' \
+'packloadall!' \
"$@"
else
$dir/bin/vim --noplugin -u NONE -N \
+"set shm+=WAFI rtp=$dir/share/vim/vimgo packpath=$dir/share/vim/vimgo,$vimgodir" \
+'filetype plugin indent on' \
+'packloadall!' \
"$@"
fi
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to have the base command just once, and then prepend the covimerage part in case of coverage=1 - might help with keeping it in sync.

You could use ${coverage:+covimerage -q run --report-file /tmp/vim-go-test/cov-profile.txt --append} - and then have $coverage be empty/unset without -c.



# vim:ts=2:sts=2:sw=2:et
23 changes: 18 additions & 5 deletions scripts/test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,25 @@ vimgodir=$(cd -P "$(dirname "$0")/.." > /dev/null && pwd)
cd "$vimgodir"

_usage() {
echo "Usage: ${0##*/} [-v] [-r file] vim_version"
echo "Usage: ${0##*/} [-hvc] [-r file] vim_version"
echo
echo "Options:"
echo " -h Show this help"
echo " -v Enable verbose output"
echo " -r Run only the tests from this file"
echo " -c Generate and submit code coverage reports"
echo
}

verbose=0
run=""
while getopts "hvr:" option; do
coverage=""
while getopts "hvcr:" option; do
case "$option" in
h) _usage; exit 0 ;;
v) verbose=1; ;;
r) run=$OPTARG ;;
c) coverage="-c" ;;
*)
echo "error: unknown option '$option'"
_usage
Expand Down Expand Up @@ -54,14 +57,16 @@ fi
### Run tests.
##############
# Clean stale log file.
[ -f '/tmp/vim-go-test/test.log' ] && rm '/tmp/vim-go-test/test.log'
[ -f '/tmp/vim-go-test/FAILED' ] && rm '/tmp/vim-go-test/FAILED'
[ -f '/tmp/vim-go-test/test.log' ] && rm '/tmp/vim-go-test/test.log'
[ -f '/tmp/vim-go-test/FAILED' ] && rm '/tmp/vim-go-test/FAILED'
[ -f '/tmp/vim-go-test/cov-profile.txt' ] && rm '/tmp/vim-go-test/cov-profile.txt'
[ -f '/tmp/vim-go-test/cov-report.txt' ] && rm '/tmp/vim-go-test/cov-report.txt'

# Run the actual tests.
find "$vimgodir" -name '*_test.vim' | while read test_file; do
[ -n "$run" -a "$(basename "$test_file")" != "$run" ] && continue

"$vimgodir/scripts/run-vim" $vim -e \
"$vimgodir/scripts/run-vim" $coverage $vim -e \
+"silent e $test_file" \
+"let g:test_verbose=$verbose" \
-S ./scripts/runtest.vim || (
Expand All @@ -85,4 +90,12 @@ if [ -f "/tmp/vim-go-test/FAILED" ]; then
fi
echo 2>&1 "All tests PASSED"

# Submit coverage reports
if [ -n "$coverage" ]; then
coverage xml --omit '*_test.vim'
codecov -X search gcov pycov -f coverage.xml --required \
--flags "$(echo "$vim" | sed -s 's/[-.]//g')"
rm coverage.xml
fi

# vim:ts=2:sts=2:sw=2:et