-
Notifications
You must be signed in to change notification settings - Fork 5
/
README.Rmd
356 lines (289 loc) · 13.1 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
---
title: "autotest"
output:
md_document:
variant: gfm
rmarkdown::html_vignette:
self_contained: no
---
# autotest <a href='https://docs.ropensci.org/autotest'><img src='man/figures/autotest.png' align="right" height=210 width=182></a>
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r setup, include=FALSE}
knitr::opts_chunk$set (
collapse = TRUE,
warning = TRUE,
message = TRUE,
width = 120,
comment = "#>",
fig.retina = 2,
fig.path = "README-"
)
```
<!-- badges: start -->
[](https://github.com/ropensci-review-tools/autotest/actions?query=workflow%3AR-CMD-check)
[](https://codecov.io/gh/ropensci-review-tools/autotest)
[](https://www.repostatus.org/#concept)
<!-- badges: end -->
Automatic mutation testing of R packages. Mutation in the sense of mutating
inputs (parameters) to function calls. `autotest` primarily works by scraping
documented examples for all functions, and mutating the parameters input to
those functions.
## Installation
The easiest way to install this package is via the associated
[`r-universe`](https://ropensci-review-tools.r-universe.dev/ui#builds). As
shown there, simply enable the universe with
```{r options, eval = FALSE}
options (repos = c (
ropenscireviewtools = "https://ropensci-review-tools.r-universe.dev",
CRAN = "https://cloud.r-project.org"
))
```
And then install the usual way with,
```{r install, eval = FALSE}
install.packages ("autotest")
```
Alternatively, the package can be installed by running one of the following
lines:
```{r gh-installation, eval = FALSE}
# install.packages("remotes")
remotes::install_git ("https://git.sr.ht/~mpadge/autotest")
remotes::install_bitbucket ("mpadge/autotest")
remotes::install_gitlab ("mpadge/autotest")
remotes::install_github ("ropensci-review-tools/autotest")
```
The package can then be loaded the usual way:
```{r load-fakey, eval = FALSE}
library (autotest)
```
```{r load, echo = FALSE, message = FALSE}
devtools::load_all (".", export_all = FALSE)
```
## Usage
The simply way to use the package is
```{r autotest-example, eval = FALSE}
x <- autotest_package ("<package>")
```
The main argument to the [`autotest_package()`
function](https://docs.ropensci.org/autotest/reference/autotest_package.html)
can either be the name of an installed package, or a path to a local directory
containing the source for a package. The result is a `data.frame` of errors,
warnings, and other diagnostic messages issued during package `auotest`-ing.
The function has an additional parameter, `functions`, to restrict tests to
specified functions only.
By default,
[`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html)
returns a list of all tests applied to a package without actually running them.
To implement those tests, set the parameter `test` to `TRUE`. Results are only returned for tests
in which functions do not behave as expected, whether through triggering
errors, warnings, or other behaviour as described below. The ideal behaviour of
`autotest_package()` is to return nothing (or strictly, `NULL`), indicating
that all tests passed successfully. See the [main package
vignette](https://docs.ropensci.org/autotest/articles/autotest.html) for an
introductory tour of the package.
## What is tested?
The package includes a function which lists all tests currently implemented.
```{r autotest_types}
autotest_types ()
```
That functions returns a [`tibble`](https://tibble.tidyverse.org) describing
`r nrow(autotest_types())` unique tests. The default behaviour of
[`autotest_package()`](https://docs.ropensci.org/autotest/reference/autotest_package.html)
with `test = FALSE` uses these test types to identify which tests will be
applied to each parameter and function. The table returned from
[`autotest_types()`](https://docs.ropensci.org/autotest/reference/autotest_types.html)
can be used to selectively switch tests off by setting values in the `test`
column to `FALSE`, as demonstrated below.
## How Does It Work?
The package works by scraping documented examples from all `.Rd` help files,
and using those to identify the types of all parameters to all functions. Usage
therefore first requires that the usage of all parameters be demonstrated in
example code.
As described above, tests can also be selectively applied to particular
functions through the parameters `functions`, used to nominate functions to
include in tests, or `exclude`, used to nominate functions to exclude from
tests. The following code illustrates.
```{r stats-var-no-test, fig.show = "hide"}
x <- autotest_package (package = "stats", functions = "var", test = FALSE)
print (x)
```
Testing the `var` function also tests `cor` and `cov`, because these are all
documented within a single `.Rd` help file. Typing `?var` shows that the help
topic is `cor`, and that the examples include the three functions, `var`,
`cor`, and `cov`. That result details the `r nrow (x)` tests which would be
applied to the `var` function from the `stats` package. These `r nrow (x)`
tests yield the following results when actually applied:
```{r stats-var-test}
y <- autotest_package (package = "stats", functions = "var", test = TRUE)
print (y)
```
And only `r nrow (y)` of the original `r nrow (x)` tests produced unexpected
behaviour. There were in fact only `r length (unique (y$operation))` kinds of
tests which produced these `r nrow (y)` results:
```{r unique-operations}
unique (y$operation)
```
One of these involves conversion of a vector to a list-column representation
(via `I(as.list(<vec>))`). Relatively few packages accept this kind of input,
even though doing so is relatively straightforward. The following lines
demonstrate how these tests can be switched off when `autotest`-ing a package.
The `autotest_types()` function, used above to extract information on all types
of tests, also accepts a single argument listing the `test_name` entries of any
tests which are to be switched off.
```{r stats-var-test-switch}
types <- autotest_types (notest = "vector_to_list_col")
y <- autotest_package (
package = "stats", functions = "var",
test = TRUE, test_data = types
)
print (y)
```
Those tests are still returned from `autotest_package()`, but with `test =
FALSE` to indicate they were not run, and a `type` of "no_test" rather than the
previous "diagnostic".
## Can `autotest` automatically create tests in my `tests` directory?
Not yet, but that should be possible soon. In the meantime, there are
[`testthat`](https://testthat.r-lib.org) expectations, listed in the [main
package
functions](https://docs.ropensci.org/autotest/reference/index.html),
which enable `autotest` to be used in a package's test suite.
## Prior work
1. The
[`great-expectations`](https://github.com/great-expectations/great_expectations)
framework for python, described in [this medium
article](https://medium.com/@expectgreatdata/down-with-pipeline-debt-introducing-great-expectations-862ddc46782a).
2. [`QuickCheck`](https://hackage.haskell.org/package/QuickCheck) for Haskell
3. [`mutate`](https://github.com/mbj/mutant) for ruby
4. [`mutant`](https://github.com/ropensci/mutant) for mutation of R code itself
## Code of Conduct
Please note that this package is released with a [Contributor Code of
Conduct](https://ropensci.org/code-of-conduct/). By contributing to this
project, you agree to abide by its terms.
## Contributors
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
All contributions to this project are gratefully acknowledged using the [`allcontributors` package](https://github.com/ropenscilabs/allcontributors) following the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!
### Code
<table>
<tr>
<td align="center">
<a href="https://github.com/mpadge">
<img src="https://avatars.githubusercontent.com/u/6697851?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/commits?author=mpadge">mpadge</a>
</td>
<td align="center">
<a href="https://github.com/helske">
<img src="https://avatars.githubusercontent.com/u/1560448?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/commits?author=helske">helske</a>
</td>
<td align="center">
<a href="https://github.com/maelle">
<img src="https://avatars.githubusercontent.com/u/8360597?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/commits?author=maelle">maelle</a>
</td>
<td align="center">
<a href="https://github.com/simpar1471">
<img src="https://avatars.githubusercontent.com/u/65285181?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/commits?author=simpar1471">simpar1471</a>
</td>
</tr>
</table>
### Issue Authors
<table>
<tr>
<td align="center">
<a href="https://github.com/noamross">
<img src="https://avatars.githubusercontent.com/u/571752?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Anoamross">noamross</a>
</td>
<td align="center">
<a href="https://github.com/njtierney">
<img src="https://avatars.githubusercontent.com/u/6488485?u=3eacd57f61342d1c3cecd5c8ac741b1c4897e1de&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Anjtierney">njtierney</a>
</td>
<td align="center">
<a href="https://github.com/JeffreyRStevens">
<img src="https://avatars.githubusercontent.com/u/51374986?u=aaa90c8bc7973213aa29cbf69a821bf18ee9bedf&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3AJeffreyRStevens">JeffreyRStevens</a>
</td>
<td align="center">
<a href="https://github.com/bbolker">
<img src="https://avatars.githubusercontent.com/u/78918?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Abbolker">bbolker</a>
</td>
<td align="center">
<a href="https://github.com/mattfidler">
<img src="https://avatars.githubusercontent.com/u/514778?u=9b4cbf14caf7dca48a1696eefbdc2ac50a6f6507&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Amattfidler">mattfidler</a>
</td>
<td align="center">
<a href="https://github.com/kieranjmartin">
<img src="https://avatars.githubusercontent.com/u/15383022?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Akieranjmartin">kieranjmartin</a>
</td>
<td align="center">
<a href="https://github.com/statnmap">
<img src="https://avatars.githubusercontent.com/u/21193866?u=073dc3920b91762de2543317110d7f2a319e433a&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Astatnmap">statnmap</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/vgherard">
<img src="https://avatars.githubusercontent.com/u/62836440?u=8ea1b4592c7304c0e8c50cd7f1ae132ebfe7e9c9&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Avgherard">vgherard</a>
</td>
<td align="center">
<a href="https://github.com/christophsax">
<img src="https://avatars.githubusercontent.com/u/1390827?u=ce6363f6da758d1bb85987d021cacc34a81c8837&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Achristophsax">christophsax</a>
</td>
<td align="center">
<a href="https://github.com/joelnitta">
<img src="https://avatars.githubusercontent.com/u/13459362?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Ajoelnitta">joelnitta</a>
</td>
<td align="center">
<a href="https://github.com/santikka">
<img src="https://avatars.githubusercontent.com/u/8639149?v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Asantikka">santikka</a>
</td>
<td align="center">
<a href="https://github.com/gilbertocamara">
<img src="https://avatars.githubusercontent.com/u/6058509?u=22d1d5486b5df88b23ead0cd3d43bbf64c3c4138&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+author%3Agilbertocamara">gilbertocamara</a>
</td>
</tr>
</table>
### Issue Contributors
<table>
<tr>
<td align="center">
<a href="https://github.com/schneiderpy">
<img src="https://avatars.githubusercontent.com/u/77991319?u=4242d4c5942fced6368dd5c68221e6618092cbf8&v=4" width="100px;" alt=""/>
</a><br>
<a href="https://github.com/ropensci-review-tools/autotest/issues?q=is%3Aissue+commenter%3Aschneiderpy">schneiderpy</a>
</td>
</tr>
</table>
<!-- markdownlint-enable -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->