[REVIEW]: wbacon: Weighted BACON algorithms for multivariate outlier nomination (detection) and robust linear regression

[whedon]

Submitting author: @tobiasschoch (Tobias Schoch)
Repository: https://github.com/tobiasschoch/wbacon
Version: v0.5
Editor: @fboehm
Reviewer: @msalibian, @aalfons
Archive: 10.5281/zenodo.4895167

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/6218ee213e8b00273645233145e96cfa"><img src="https://joss.theoj.org/papers/6218ee213e8b00273645233145e96cfa/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/6218ee213e8b00273645233145e96cfa/status.svg)](https://joss.theoj.org/papers/6218ee213e8b00273645233145e96cfa)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@msalibian & @aalfons, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @fboehm know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @msalibian

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@tobiasschoch) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @aalfons

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@tobiasschoch) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @msalibian, @aalfons it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.23 s (190.0 files/s, 37362.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TeX                              4            280            173           3276
C                                8            302            696           1307
R                               15            115            122            850
HTML                             1             95              5            739
Markdown                         3             39              0            139
C/C++ Header                     9             23             17            132
Rmd                              1            119            159             39
YAML                             1              1              0             18
DOS Batch                        1              0              2              2
SVG                              1              0              0              1
-------------------------------------------------------------------------------
SUM:                            44            974           1174           6503
-------------------------------------------------------------------------------


Statistical information for the repository 'a9b8cecb1778454a282290cc' was
gathered on 2021/05/04.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Tobias Schoch                   36          6003           3526          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Tobias Schoch              2477           41.3          2.4               31.09

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/S0167-9473(99)00101-2 is OK
- 10.1145/567806.567807 is OK
- 10.1137/1.9780898719604 is OK
- 10.1016/j.csda.2004.09.009 is OK
- 10.1080/01621459.1990.10474920 is OK
- 10.1198/jcgs.2009.0005 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[fboehm]

@msalibian and @aalfons - we're now starting the review. You can see above your review checklist. Please check the boxes as you proceed through the review. For any boxes that you can't check right now, please leave comments in this thread. You may also choose to open issues in the package repository. If you do open issues, please be sure to reference them in this thread. Please let me know if you have any questions as you work through the review. Thanks again!

[fboehm]

@msalibian and @aalfons - how is the review going? Is there anything that I can assist with? Please feel free to check off the boxes above as you proceed through the review. Thanks!

[msalibian]

This is a nice contribution (both package and manuscript). I have some minor suggestions about the text, and noted that some items from the checklist above are missing.

• In the text, when referring to "large datasets", please be specific if this is regarding the number of cases (rows), variables (columns), or both. In the robustness literature, computational effort ("complexity" in an informal way) usually grows exponentially with the number of columns, while a larger number of cases does not generally increase the cost faster than linearly.
• There are references to the "BACON algorithms", it may be helpful for the reader if the Summary mentioned that the two algorithms correspond to either a multivariate location/scatter model, or a linear regression one.
• line 12: "superior" is a relative term that requires a comparison (superior to what?), I suggest simply mentioning the method's popularity instead.
• line 15: I believe the preferred term in R is "package" (rather than "library").
• line 18: "etc.", I suggest listing all the implemented diagnostic methods / tools instead.
• "Example usage": currently missing. It would be very useful to include an example illustrating the usage of the package (including how to set tuning parameters), and use it also to compare its performance against that of other BACON implementations.
• "State of the field" and "Performance": although the author describes how this software compares to other implementations, a detailed example would be very useful (as I mention above). In particular, it is not currently clear if the "large datasets" for which this package is geared are "large" in terms of the number of cases (rows), the number of variables (columns), or both.
• "Community guidelines" are missing.

[tobiasschoch]

@msalibian - Thank you for the very helpful remarks. Before I’m going to address the points raised, I have some questions for the editor.

@fboehm - I have the following questions:

1. I’m planning to do the following (while working through the issues raised by the reviewers):

• Make changes in the documents/ code (in the repository)
• Add a new version tag to the repository
• Call [at]whedon set [new version] as version
• Call [at]whedon generate pdf
• Is that right?

2. “Example usage”: The package has a vignette. In the vignette, the methods are applied to example data. Should I include the examples or some of the examples in the paper as well? Or should I just note in the paper that there is a vignette?

3. “Community guidelines”: These guidelines can be found in the README.md on https://github.com/tobiasschoch/wbacon. I put them there such that they show up on the GitHub site (this is the place where most people will “land” while browsing/ searching the internet). Should I include the “community guidelines” in the paper as well?

4. "Performance":

• I test my implementation against (the existing code) robustX::BACON; see folder test (focus: does my implementation give the same results?).
• In addition, there are some benchmarks; see test/benchmark (used to “see” the impact of the OpenMP parallelization).
• However, there are yet no benchmarks, where I compare my implementation against robustX::BACON in terms of performance/ speed. I’m planning to add such a benchmark. Then, I note in the paper that such benchmark is available. Is that ok?

[fboehm]

@tobiasschoch - Thanks for your questions. Below I try to answer them.

1. Don't worry about making the tagged release or interacting with whedon. I'll guide you through that when it's time. For now, just concentrate on your first bullet point. If you want to see how the pdf looks after your changes are made, you can tell whedon to "generate pdf", but don't create a new release until I ask you to do so.

2. I'm not sure about this one. I think the paper.md might be better off with a mention of the vignette. You probably don't need to include examples in the paper.md file. @msalibian - do you have thoughts on this?

3. I think that including guidelines in the README.md is sufficient. However, you might also want to add a CONTRIBUTING.md file to the repository. For example, see this: https://github.com/tidyverse/dplyr/blob/master/.github/CONTRIBUTING.md. You don't need to include community guidelines in the paper.md. There is a function in the R package usethis that can create such a file from a template: usethis::use_tidy_contributing. You might want to edit the resulting file.

4. Yes, I think that mentioning in paper.md that you have the tests and benchmarks in the package is sufficient. You might also include unit tests, possibly by using the R package testthat. @msalibian - do you agree? Do you have more specific comments to add on this point?

[fboehm]

@aalfons - how is the review going? Please let me know if you encounter any difficulties.

[msalibian]

@fboehm @tobiasschoch About the above:

2 - I don't have a strong opinion on this. A reference to the vignette would probably be enough. Although in that case I think it would be helpful to include in the paper a few sentences describing / quantifying the performance gain that can be expected with this implementation. Something like "When the sample size is larger than XXX, using this implementation often results in a speed gain of YYY% compared with such and such..."

4 - Agreed.

[aalfons]

@fboehm The only difficulty is finding time. I'll get to it by the end of the week. My apologies for the delay, I was not aware that it needs to be done so quickly.

[tobiasschoch]

@fboehm @msalibian @aalfons

• I'm greatful for all the hints and comments. Now I understand how the review process works at JOSS.
• Frederick, I don't mind if the review takes more time than usual. The reviewers are busy professors (so am I). I prefer a detailed review to a quick review. We should give aalfons more time if he needs it.
• Next, I'm going to address the points raised by msalibian.

[fboehm]

@aalfons - I apologize if I implied that I expected the review to be done by now. That wasn't my intention. I merely wanted to ask if you'd gotten started or encountered difficulties. Sometimes reviewers are unsure how to start the review, especially since we use a nontraditional format at JOSS. If any questions arise, please let me know. Thanks again!!

[fboehm]

@tobiasschoch - that all sounds good. Please feel free to address the issues that @msalibian raised as the next step. Thanks!!

[whedon]

👋 @msalibian, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @aalfons, please update us on how your review is going (this is an automated reminder).

[aalfons]

The package seems to provide useful functionality and is well documented, and the manuscript describes the functionality and the need for it nicely. I still have some comments, some of which I realized afterwards are also brought up by @msalibian.

• Installation instructions: I received the following error message when trying to install the package:

In file included from fitwls.c:20:
./fitwls.h:5:10: fatal error: 'omp.h' file not found

The latest version of R for Mac now rely only on the compilers shipped with Apple's XCode developer platform, and unfortunately it seems that Apple has disabled OpenMP support there by default, see https://mac.r-project.org/openmp/ . I was not aware of that.

I have of course fixed OpenMP support now as per instructions on the above website, and the package installed as per the documentation. But there may be other users who run into the same issue and give up on your package.

I believe the recommended way to include OpenMP in the C++ header files is the following:

#ifdef _OPENMP
#include <omp.h>
#endif

This is a simple fix, so please go ahead and implement it. While users without OpenMP installed then do not have access to parallel computing when using your package, at least they can install and use it.

• Performance: line 36: What size of the data set (number of observations, number of variables) is approximately necessary for implementation inefficiencies to matter?

• Automated tests: There are some scripts in the tests/ directory that seem to reproduce the results of a paper and contain some speed tests of the computational performance, but I did not find any unit tests that the functions produce the correct output and so on. On the other hand, the documentation contains some examples, and it can be automatically checked whether these produce warnings or errors via R CMD check. So in that sense, there would be some minimal automated testing of the functionality. @fboehm Could you please provide some guidance on this point?

• State of the field: The authors mention only other packages that implement the same algorithms. Maybe it would be good to also mention some other popular packages that provide other algorithms for anomaly detection? (Although this could quickly go out of hand, as there are many algorithms for this purpose.)

General comments:

• line 12: the statement that the BACON algorithms are "superior" would imply that they are always better than other anomaly detection algorithms. Please rephrase this.

• lines 15-17: Even though the function to load a package is called library(), which causes some confusion about the terminology, the preferred term is "package" in R. Please change this throughout the paper.

• line 66/67: How should one proceed if the diagnostic tools indicate that the "good" observations violate the structure that is required by the algorithm?

[tobiasschoch]

I modified the paper according to @msalibian suggestions (I'm going to address the points raised by @aalfons later)

Issue 1

• Reviewer: In the text, when referring to "large datasets", please be specific if this is regarding the number of cases (rows), variables (columns), or both. In the robustness literature, computational effort ("complexity" in an informal way) usually grows exponentially with the number of columns, while a larger number of cases does not generally increase the cost faster than linearly.
• Answer: In the revised manuscript, I now point out that the time complexity of the algorithms is dominated by the number of variables/ columns. Therefore, the implemented parallelization over the columns is meaningful. See also the new chapter on benchmarking.

Issue 2

• Reviewer: There are references to the "BACON algorithms", it may be helpful for the reader if the Summary mentioned that the two algorithms correspond to either a multivariate location/scatter model, or a linear regression one.
• Answer: Yes, I now mention the two methods.

Issue 3

• Reviewer: line 12: "superior" is a relative term that requires a comparison (superior to what?), I suggest simply mentioning the method's popularity instead.
• Answer: Yes, I deleted the ambiguous term “superior”.

Issue 4

• Reviewer: line 15: I believe the preferred term in R is "package" (rather than "library").
• Answer: Yes, I now talk about “package” not “library”.

Issue 5

• Reviewer: line 18: "etc.", I suggest listing all the implemented diagnostic methods / tools instead.
• Answer: I chose to stick with the (rather unspecific) “etc.” instead of listing all methods as this would have made the summary bulky. However, I now added a chapter called “Illustration” where I show an application of some of the methods and refer the reader to the vignette to learn more.

Issue 6

• Reviewer: "Example usage": currently missing. It would be very useful to include an example illustrating the usage of the package (including how to set tuning parameters), and use it also to compare its performance against that of other BACON implementations.
• Answer: I added a small example on robust regression. This is meant as a kind of teaser to get readers interested in the vignette.

Issue 7

• Reviewer: "State of the field" and "Performance": although the author describes how this software compares to other implementations, a detailed example would be very useful (as I mention above). In particular, it is not currently clear if the "large datasets" for which this package is geared are "large" in terms of the number of cases (rows), the number of variables (columns), or both.
• Answer: I added a chapter on “benchmarking”, where I compare my implementation with the one in the robustX package (reference) in terms of computation time for data sets in various sizes. From the benchmarking, we can learn that my implementation outperforms the reference on medium to large data sets.

Issue 8

• Reviewer: "Community guidelines" are missing.
• Answer: I already had the file “CONTRIBUTING.md” in the base directory of the package. Now I added a chapter on "community guidelines" to the paper.

[tobiasschoch]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[tobiasschoch]

@aalfons - Thank you for the very helpful remarks. Your comments reached me while I was responding to the points raised by msalibian. I will now address the points you have raised.

Issue 1

Reviewer: Installation instructions: I received the following error message when trying to install the package [...].

Answer: Yes, indeed I forgot to add the (conditional) include guards. This is now fixed. I added the include guards in the C header files.

Issue 2

Reviewer: Performance: line 36: What size of the data set (number of observations, number of variables) is approximately necessary for implementation inefficiencies to matter?

Answer: In the mean time I have updated the paper. In the "Benchmarking" section, I compare my implementation with the reference implementation robustX in terms of computation time for data sets in various sizes. From the benchmarking, we can learn that my implementation outperforms the reference on medium to large data sets.

Issue 3

Reviewer: Automated tests: There are some scripts in the tests/ directory that seem to reproduce the results of a paper and contain some speed tests of the computational performance, but I did not find any unit tests that the functions produce the correct output and so on. On the other hand, the documentation contains some examples, and it can be automatically checked whether these produce warnings or errors via R CMD check. So in that sense, there would be some minimal automated testing of the functionality. @fboehm Could you please provide some guidance on this point?

Answer: I am a believer in test-driven development. I distinguish between two (idealized) types of tests:

• Unit test: Test what the code is. The purpose of unit testing is to isolate the smallest testable parts of the code base and verify whether they function properly in isolation.
• Functional test: Test what the code does. A tester is not concerned with the actual code, rather s/he wants to verify the output with the expected output.

As a pragmatic programmer I take the following view: Some functions are suitable for unit testing, others less so.

• I do have unit tests for the functions wquantile and wselect. Because these functions are also used in other R packages, they reside in their own GitHub repo called "wquantile". There, you can find unit tests based on the C language CHEAT test library of Guillermo Freschi and Sampsa Kiiskinen; see https://github.com/tobiasschoch/wquantile/blob/master/tests/test.c.

• The vast majority of code in the "wbacon" repo is tested under the "functional testing" paradigm for the following reason: The BACON algorithms are based on the basic "building blocks": mean, covariance matrix, linear regression, etc. In my package, these building blocks are computed with the help of the LAPACK and BLAS subroutines. More precisely,

  • the covariance matrix is based on dsyrk => computes the lower triangular scatter matrix (see wbacon.c).
  • the regression coefficients are computed with dgels; the residuals are computed with dgemv (see fitwls.c).
  • ...

We could, in principle, follow the unit-test paradigm and write separate unit tests for the covariance matrix, regression, and so on. However, this would ultimately boil down to testing the subroutines in BLAS and LAPACK. Neither is my package the place to test these subroutines, nor would such a testing strategy increase confidence in my functions. Of course, that does not rule out the possibility that I mixed up the order of the arguments when calling the subroutines. But the compiler (and valgrind) would have told me if I had actually done it. So I decided to take a "functional test" approach and test the ensemble of building blocks. In total, the BACON algorithms are tested on 159 different (real) test sets to match the results of the (reference implementation) in the robustX package.

Issue 4

Reviewer: State of the field: The authors mention only other packages that implement the same algorithms. Maybe it would be good to also mention some other popular packages that provide other algorithms for anomaly detection? (Although this could quickly go out of hand, as there are many algorithms for this purpose.)

Answer: Yes, this could indeed go out of hand quickly... I focused on the BACON algorithms and did not intend to write a review article on multivariate outlier detection. @fboehm: Shall I discuss other methods?

Issue 5

Reviewer: line 12: the statement that the BACON algorithms are "superior" would imply that they are always better than other anomaly detection algorithms. Please rephrase this.

Answer: Yes, this has been fixed; see response to points raised by msalibian.

Issue 6

Reviewer: lines 15-17: Even though the function to load a package is called library(), which causes some confusion about the terminology, the preferred term is "package" in R. Please change this throughout the paper.

Answer: Yes, this has been fixed; see response to points raised by msalibian.

Issue 7

Reviewer: line 66/67: How should one proceed if the diagnostic tools indicate that the "good" observations violate the structure that is required by the algorithm?

Answer: Well... all I can do (see also vignette) is to quote the developpers of the BACON algorithms: “Although the algorithms will often do something reasonable even when these assumptions are violated, it is hard to say what the results mean.” Billor et al. (2000, p. 290). It is better to be "safe than sorry" and apply another method. I think I have made it sufficiently clear when the method can and cannot be used; see vignette.

[whedon]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands

[fboehm]

@whedon set v0.5 as version

[whedon]

OK. v0.5 is the version.

[fboehm]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/S0167-9473(99)00101-2 is OK
- 10.1002/0470055464 is OK
- 10.1145/567806.567807 is OK
- 10.1137/1.9780898719604 is OK
- 10.1002/9781119214656 is OK
- 10.1016/j.csda.2004.09.009 is OK
- 10.1080/01621459.1990.10474920 is OK
- 10.1198/jcgs.2009.0005 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[fboehm]

@whedon set 10.5281/zenodo.4895167 as archive

[whedon]

OK. 10.5281/zenodo.4895167 is the archive.

[fboehm]

@whedon accept

[whedon]

To recommend a paper to be accepted use @whedon recommend-accept

[fboehm]

@whedon recommend-accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#2356

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#2356, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/S0167-9473(99)00101-2 is OK
- 10.1002/0470055464 is OK
- 10.1145/567806.567807 is OK
- 10.1137/1.9780898719604 is OK
- 10.1002/9781119214656 is OK
- 10.1016/j.csda.2004.09.009 is OK
- 10.1080/01621459.1990.10474920 is OK
- 10.1198/jcgs.2009.0005 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[fboehm]

@tobiasschoch - I just recognized that the title of the archive doesn't match the title of the manuscript. Can you please fix this?

[tobiasschoch]

@fboehm - Sorry... I have fixed it.

[Kevin-Mattheus-Moerman]

@tobiasschoch

• Can you ensure the author information of the Zenodo archive version is complete and matches that of the paper? Please use your full name and you should be able to add your ORCID id there too.
• In the paper when using the / it appears the authors sometimes follow it with a space and sometimes not e.g. variables/ columns and intercept/constant, I propose you remove the additional spaces when used.

[tobiasschoch]

@Kevin-Mattheus-Moerman

1. I have corrected the author information of the Zenodo archive, and I have added my ORCID.
2. I have removed the additional spaces after / in the paper.

[Kevin-Mattheus-Moerman]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Kevin-Mattheus-Moerman]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.03238 joss-papers#2364
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.03238
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[Kevin-Mattheus-Moerman]

Congratulations @tobiasschoch on getting this work published in JOSS!

Thanks @fboehm for editing this work!

Thanks also to @msalibian, @aalfons for the great job reviewing this work!

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.03238/status.svg)](https://doi.org/10.21105/joss.03238)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.03238">
  <img src="https://joss.theoj.org/papers/10.21105/joss.03238/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.03238/status.svg
   :target: https://doi.org/10.21105/joss.03238

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss