Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#stdout coverage format #480

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ Session.vim
*~
images
*.fuel
coverage/
3 changes: 2 additions & 1 deletion .smalltalk.ston
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ SmalltalkCISpec {
],
#testing : {
#coverage : {
#categories : [ 'SmalltalkCI-*' ]
#categories : [ 'SmalltalkCI-*' ],
#format: [#stdout, #lcov, #coveralls]
},
#exclude : {
#classes : [ #SCIExcludedTests ]
Expand Down
18 changes: 14 additions & 4 deletions docs/COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ It natively supports uploading coverage to [Coveralls][coveralls] from GitHub Ac

Other CI services/Coverage reporters are supported by smalltalkCI with the [LCOV output format](#uploading-with-different-ci-services%2Fcoverage-reporters).

smalltalkCI is also able to print its own coverage summary to stdout, where all methods are grouped by classes.

## Configuring coverage testing

To enable coverage testing behavior, add a `#coverage` dictionary to the `#testing` slot of your `.smalltalk.ston`:
Expand All @@ -16,7 +18,7 @@ SmalltalkCISpec {
#coverage : {
#packages : [ 'Packages-To-Cover.*' ],
#classes : [ #ClassToCover, #'ClassToCover class' ],
#categories : [ 'Categories-To-Cover*' ]
#categories : [ 'Categories-To-Cover*' ],
#format : #coveralls
}
}
Expand All @@ -31,7 +33,8 @@ The `#coverage` dictionary can contain the following options:
- Measure coverage for all classes' methods as well as their meta classes' methods
- `#format` (defaults to `#coveralls`)
- The output format of the Coverage data
- May be either `#coveralls` or `#lcov`
- May be either `#coveralls`,`#lcov`, or `#stdout`
- May also be a list containing multiple output formats

When running smalltalkCI on TravisCI or AppVeyor with the `#coveralls` coverage format, the results will be uploaded to [Coveralls][coveralls] automatically.
Make sure your repository is [added to Coveralls][coveralls_new].
Expand All @@ -49,15 +52,22 @@ Writing LCOV coverage info to: /path/to/coverage/lcov.info
Most coverage services already support uploading coverage in the LCOV format with uploader scripts.

For the most common usecases, see these instructions:
- [Inspecting coverage locally](#inspecting-coverage-locally)
- [Coverage report on stdout](#coverage-report-on-stdout)
- [Inspecting coverage locally](#generating-a-coverage-report-locally)
- [Coveralls](#coveralls)
- [Travis CI](#coveralls-%26-travis-ci)
- [GitHub actions](#coveralls-%26-github-actions)
- [CodeCov](#codecov)
- [Travis CI](#codecov-%26-travisci)
- [GitHub actions](#codecov-%26-github-actions)

### Inspecting coverage locally
### Coverage report on stdout
smalltalkCI can optionally print a coverage report directly to stdout in the `Exporting Coverage...` step.
To enable this feature, add `#stdout` to the `#format` coverage option.

The coverage report will contain a list of classes. Fully covered classes are marked with a green checkmark and listed first, followed by classes that are only partially covered, listing which methods are missing coverage.

### Generating a coverage report locally
On Linux distributions, LCOV is available as a set of tools that can generate a coverage report as HTML/CSS files.
First, make sure you have LCOV coverage enabled in your `.smalltalk.ston`.
Then navigate to your project directory (the directory containing your `.smalltalk.ston`), run `bin/smalltalkci` and generate the LCOV report.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
helpers
exportCoverage: anSCICodeCoverage in: aDirectory

"Coverage is not enabled"
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
helpers
printlnToStdout: msg ansiColor: ansiColor
"Compared to printToStdout:ansiColor, only print a single line without empty line in front"
self stdout nextPutAll: ansiColor, msg, self ansiReset; cr; flush
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
helpers
run: aBlock spec: aSmalltalkCISpec in: aProjectDirectory
"No coverage testing enabled"
^ aBlock value
aBlock value.
^ nil
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
executing
basicTest
| runner |
| runner coverage |
self ensureSpecIsValid.
self class
stage: 'Testing project...'
id: 'test_project'
block: [
self prepareForTesting.
self executionClass
coverage := self executionClass
run: [ runner := self runTests ]
spec: self spec
in: self projectDirectory.
self finishUpAfterTesting ].
coverage ifNotNil: [
self class printToStdout: 'Exporting coverage...' ansiColor: self class ansiBlue, self class ansiBold.
self executionClass exportCoverage: coverage in: self projectDirectory].
self reportResults: runner.
runner isSuccessful
ifFalse: [ SCIError signal: '[test failure]' ]
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
"ensureNoExistingBuildStatusFile" : "fn 11/20/2017 11:55",
"escape" : "fn 9/26/2016 16:12",
"explicitTestKeys" : "fn 11/13/2016 22:23",
"exportCoverage:in:" : "smalltalkCI 12/8/2020 15:42",
"failAndQuit:" : "smalltalkCI 8/27/2020 15:25",
"fileExists:" : "fn 11/22/2017 14:19",
"fold:block:" : "fn 10/7/2016 21:26",
Expand Down Expand Up @@ -74,6 +75,7 @@
"printToStdout:" : "fn 9/27/2016 20:37",
"printToStdout:ansiColor:" : "fn 7/9/2017 00:47",
"printWarningToStdout:" : "fn 7/9/2017 00:47",
"printlnToStdout:ansiColor:" : "smalltalkCI 10/4/2020 19:07",
"promptToProceed" : "LM 9/4/2020 14:47",
"promptToProceedImpl" : "LM 9/4/2020 14:48",
"quitImage" : "fn 10/11/2016 18:42",
Expand All @@ -85,7 +87,7 @@
"resolve:with:" : "fn 6/22/2016 09:42",
"resolveAll:with:" : "fn 6/22/2016 09:42",
"resolvedPackageNames:" : "fn 11/23/2016 11:16",
"run:spec:in:" : "fn 11/30/2017 17:01",
"run:spec:in:" : "smalltalkCI 10/12/2020 17:17",
"saveAndQuitImage" : "fn 10/11/2016 21:34",
"saveImage" : "fn 7/2/2016 17:21",
"setAuthor:" : "fn 11/3/2016 18:22",
Expand All @@ -110,7 +112,7 @@
"workingDirectory" : "fn 10/16/2016 16:28" },
"instance" : {
"basicLoad" : "fn 11/21/2017 11:10",
"basicTest" : "fn 11/30/2017 13:47",
"basicTest" : "smalltalkCI 12/8/2020 15:42",
"compatibleConfigurationsFor:" : "fn 11/11/2017 15:03",
"compatibleSpecs" : "fn 11/11/2017 15:03",
"ensureSpecIsValid" : "fn 11/21/2017 11:10",
Expand Down
6 changes: 3 additions & 3 deletions repository/SmalltalkCI-Core.package/monticello.meta/version
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
(name 'SmalltalkCI-Core-fn.258' message 'Introduce SmalltalkCI class>>#isColorful' id '9790a636-679f-4585-86ce-a2db1457f78d' date '26 November 2020' time '12:23:37.055938 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.257' message 'Change default of promptToProceed to keep image open' id '35024ec0-98d8-4551-b86e-bd59db86e080' date '4 September 2020' time '2:48:56.328475 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.256' message 'Delegate promptToProceed to promptToProceedImpl to avoid infinite recursion' id '1a4ba816-d3fb-4b4f-8654-af752fcf17e9' date '4 September 2020' time '2:48:10.934365 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.255' message 'Delegate promptToProceed to platformclass' id '9bb2347e-0b0b-4d62-93e1-16f79625f6f7' date '4 September 2020' time '2:43:45.730905 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.254' message 'Add promptToProceed to SmalltalkCI' id 'c9c0b396-e8c2-4d04-a320-8377e0c16bec' date '3 September 2020' time '12:07:21.192529 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.253' message 'Close Progressbars before prompting to proceed' id '4ceddd02-20fb-4888-bb8b-fd9e5bff8671' date '3 September 2020' time '11:34:57.661483 am' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.252' message 'Add promptToProceed helper function' id 'dc0d9a8e-b8d4-4ccc-bd41-7519db135042' date '3 September 2020' time '11:17:47.176847 am' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.251' message 'Add missing period in failAndQuit:' id 'ddfd0c64-51d6-48cb-a977-776e3b081e93' date '27 August 2020' time '3:26:18.498778 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.250' message 'Use correct platformClass for self calls in failAndQuit:' id 'b316831d-b0cb-4690-a465-2ae18d4747e0' date '26 August 2020' time '2:35:45.846989 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.249' message 'Add reference to image folder in saveAndQuit:' id '7d3e7080-d695-4a63-932d-38cf21e4a9ec' date '26 August 2020' time '2:32:45.92929 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.248' message 'Add failAndQuit: helper' id '922f314e-28b7-4738-90a4-e6844dc35ea0' date '26 August 2020' time '2:16:40.79477 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.247' message 'Fix unit in fold:on:block:' id '2d7dd736-99ac-4f39-936f-20828deba5c6' date '25 August 2020' time '1:06:34.562412 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-fn.246' message 'Do nothing by default in SCITestRunner>>serializeError:of: (#440).' id '5d923b61-521c-4d39-aab6-65bd3631e33f' date '26 September 2019' time '3:27:58.152573 pm' author 'fn' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.245' message 'extend ignoreImage api to react to MetacelloIgnorePackageLoaded.
This will force the load of a package already in image even if is in the ancestor list.' id 'a71d4650-a333-0d00-9a56-d9c206468e0a' date '2 October 2018' time '6:40:16.696915 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.244' message 'expose #ignoreImage metacello api' id 'aa61a3ae-a133-0d00-be56-b23706db7c03' date '2 October 2018' time '4:43:29.93288 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-fn.243' message 'Use classVariableNames instead of instanceVariableNames on class side (fixes #395).' id '60d9f141-3bd4-4585-b65e-14af44c01c81' date '2 October 2018' time '9:18:40.024 am' author 'fn' ancestors ((name 'SmalltalkCI-Core-fn.242' message 'Merge commit (240+241); apply #onConflict/#onUpgrade only when values are not nil.' id 'f287589a-5208-4534-b637-9cca010ca578' date '2 October 2018' time '8:51:04.357 am' author 'fn' ancestors ((name 'SmalltalkCI-Core-JohanBrichau.241' message 'Change option disableSerializeError to serializeError' id '7256b7b0-0a32-0d00-8794-11e9012935e3' date '12 September 2018' time '11:09:53.105902 am' author 'JohanBrichau' ancestors ((name 'SmalltalkCI-Core-JohanBrichau.240' message 'Change option disableSerializeError to serializeError' id 'd39938a7-0a32-0d00-8792-5dcd012935e3' date '12 September 2018' time '11:07:13.80521 am' author 'JohanBrichau' ancestors ((name 'SmalltalkCI-Core-fn.239' message 'Update Metacello repository URL.' id '10ec3dcf-d1fe-4fc3-9193-8affb4be053b' date '7 December 2017' time '11:34:47.321 am' author 'fn' ancestors () stepChildren ())) stepChildren ())) stepChildren ())(name 'SmalltalkCI-Core-EstebanLorenzano.240' message 'add #onConflict:/#onUpgrade: metacello logic' id 'be7e2fe1-4f33-0d00-9fd9-e2a50f2ac847' date '28 September 2018' time '3:07:50.658062 pm' author 'EstebanLorenzano' ancestors ((id '10ec3dcf-d1fe-4fc3-9193-8affb4be053b')) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())
(name 'SmalltalkCI-Core-smalltalkCI.259' message 'Refactor coverage exporting for GemStone compatibility' id '488b0eba-f388-4151-8591-0ac600c260a2' date '8 December 2020' time '3:43:34.938909 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-fn.258' message 'Introduce SmalltalkCI class>>#isColorful' id '9790a636-679f-4585-86ce-a2db1457f78d' date '26 November 2020' time '12:23:37.055938 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.257' message 'Change default of promptToProceed to keep image open' id '35024ec0-98d8-4551-b86e-bd59db86e080' date '4 September 2020' time '2:48:56.328475 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.256' message 'Delegate promptToProceed to promptToProceedImpl to avoid infinite recursion' id '1a4ba816-d3fb-4b4f-8654-af752fcf17e9' date '4 September 2020' time '2:48:10.934365 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.255' message 'Delegate promptToProceed to platformclass' id '9bb2347e-0b0b-4d62-93e1-16f79625f6f7' date '4 September 2020' time '2:43:45.730905 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.254' message 'Add promptToProceed to SmalltalkCI' id 'c9c0b396-e8c2-4d04-a320-8377e0c16bec' date '3 September 2020' time '12:07:21.192529 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.253' message 'Close Progressbars before prompting to proceed' id '4ceddd02-20fb-4888-bb8b-fd9e5bff8671' date '3 September 2020' time '11:34:57.661483 am' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.252' message 'Add promptToProceed helper function' id 'dc0d9a8e-b8d4-4ccc-bd41-7519db135042' date '3 September 2020' time '11:17:47.176847 am' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.251' message 'Add missing period in failAndQuit:' id 'ddfd0c64-51d6-48cb-a977-776e3b081e93' date '27 August 2020' time '3:26:18.498778 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.250' message 'Use correct platformClass for self calls in failAndQuit:' id 'b316831d-b0cb-4690-a465-2ae18d4747e0' date '26 August 2020' time '2:35:45.846989 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.249' message 'Add reference to image folder in saveAndQuit:' id '7d3e7080-d695-4a63-932d-38cf21e4a9ec' date '26 August 2020' time '2:32:45.92929 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.248' message 'Add failAndQuit: helper' id '922f314e-28b7-4738-90a4-e6844dc35ea0' date '26 August 2020' time '2:16:40.79477 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-smalltalkCI.247' message 'Fix unit in fold:on:block:' id '2d7dd736-99ac-4f39-936f-20828deba5c6' date '25 August 2020' time '1:06:34.562412 pm' author 'smalltalkCI' ancestors ((name 'SmalltalkCI-Core-fn.246' message 'Do nothing by default in SCITestRunner>>serializeError:of: (#440).' id '5d923b61-521c-4d39-aab6-65bd3631e33f' date '26 September 2019' time '3:27:58.152573 pm' author 'fn' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.245' message 'extend ignoreImage api to react to MetacelloIgnorePackageLoaded.

This will force the load of a package already in image even if is in the ancestor list.' id 'a71d4650-a333-0d00-9a56-d9c206468e0a' date '2 October 2018' time '6:40:16.696915 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-EstebanLorenzano.244' message 'expose #ignoreImage metacello api' id 'aa61a3ae-a133-0d00-be56-b23706db7c03' date '2 October 2018' time '4:43:29.93288 pm' author 'EstebanLorenzano' ancestors ((name 'SmalltalkCI-Core-fn.243' message 'Use classVariableNames instead of instanceVariableNames on class side (fixes #395).' id '60d9f141-3bd4-4585-b65e-14af44c01c81' date '2 October 2018' time '9:18:40.024 am' author 'fn' ancestors ((name 'SmalltalkCI-Core-fn.242' message 'Merge commit (240+241); apply #onConflict/#onUpgrade only when values are not nil.' id 'f287589a-5208-4534-b637-9cca010ca578' date '2 October 2018' time '8:51:04.357 am' author 'fn' ancestors ((name 'SmalltalkCI-Core-JohanBrichau.241' message 'Change option disableSerializeError to serializeError' id '7256b7b0-0a32-0d00-8794-11e9012935e3' date '12 September 2018' time '11:09:53.105902 am' author 'JohanBrichau' ancestors ((name 'SmalltalkCI-Core-JohanBrichau.240' message 'Change option disableSerializeError to serializeError' id 'd39938a7-0a32-0d00-8792-5dcd012935e3' date '12 September 2018' time '11:07:13.80521 am' author 'JohanBrichau' ancestors ((name 'SmalltalkCI-Core-fn.239' message 'Update Metacello repository URL.' id '10ec3dcf-d1fe-4fc3-9193-8affb4be053b' date '7 December 2017' time '11:34:47.321 am' author 'fn' ancestors () stepChildren ())) stepChildren ())) stepChildren ())(name 'SmalltalkCI-Core-EstebanLorenzano.240' message 'add #onConflict:/#onUpgrade: metacello logic' id 'be7e2fe1-4f33-0d00-9fd9-e2a50f2ac847' date '28 September 2018' time '3:07:50.658062 pm' author 'EstebanLorenzano' ancestors ((id '10ec3dcf-d1fe-4fc3-9193-8affb4be053b')) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())) stepChildren ())
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
instance creation
empty

^ SCICodeCoverage new
coveredMethods: OrderedCollection new;
includedMethods: OrderedCollection new;
excludedMethods: OrderedCollection new;
yourself
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
executing
exportCoverage: anSCICodeCoverage in: aDirectory

anSCICodeCoverage ifNotNil: [anSCICodeCoverage exportResultsIn: aDirectory]
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ run: aBlock spec: aSmalltalkCISpec in: projectDirectory
^ self new
spec: aSmalltalkCISpec;
run: aBlock;
exportResultsIn: projectDirectory
yourself
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
accessing
coverageWrappers

^ coverageWrappers

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
exporting
coverageWriterClasses

^ self spec coverageDictionary
at: #format
ifPresent: [:formatOption | |formats|
formats := formatOption.
"The user is allowed to only provide a single format for
convenience, so convert it to a collection"
(formats isSymbol or: [formats isCollection not]) ifTrue: [formats := {formats}].
formats collect: [:format |
SCICoverageWriter writerClassFor: format]]
ifAbsent: [SCICoverallsWriter asOrderedCollection].
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
exporting
coverageWriters

^ self coverageWriterClasses collect: [:clazz |
clazz new
spec: spec;
yourself]
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
accessing
coveredMethods: aCollection

coveredMethods := aCollection.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
accessing
excludedMethods: aCollection

excludedMethods := aCollection.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
exporting
exportResultsIn: projectDirectory

self coverageWriter export: self in: projectDirectory.
self coverageWriters do: [:each | each export: self in: projectDirectory.]
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
accessing
includedMethods: aCollection

includedMethods := aCollection.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
as yet unclassified
initialize

self
includedMethods: OrderedCollection new;
coveredMethods: OrderedCollection new;
excludedMethods: OrderedCollection new.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
testing
isFullyCovered

^ self includedMethods size = self coveredMethods size
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
helpers
splitByClass

| classDictionary |
classDictionary := Dictionary new.
self coveredMethods do: [:methodReference |
(classDictionary at: methodReference actualClass ifAbsentPut: [SCICodeCoverage empty])
coveredMethods addLast: methodReference].
self includedMethods do: [:methodReference |
(classDictionary at: methodReference actualClass ifAbsentPut: [SCICodeCoverage empty])
includedMethods addLast: methodReference].
self excludedMethods do: [:methodReference |
(classDictionary at: methodReference actualClass ifAbsentPut: [SCICodeCoverage empty])
excludedMethods addLast: methodReference].
^ classDictionary

Loading