[JOSS REVIEW] Main feedback

[humanfactors]

Submission: openjournals/joss-reviews#3236
Title: statsExpressions: R Package for Tidy Dataframes and Expressions with Statistical Details
Review Outcome: Minor Revisions

Overview

statsExpressions is an R package for quickly running various forms of statistical tests using a single framework (e.g.,
consistent syntax) and offers users "pre-formatted statistical expressions" for annotating plots, as well as consistent
dataframe outputs. The package covers a comprehensive range of commonly used statistical tests (at least in the
behavioral sciences).

I commend the author on their effort, the package appears to be well written and documented, with comprehensive tests and exhaustive use of continuous integration. I installed the software successfully, and ran a few statistical tests myself. I could see myself using this software in the future.

At this stage, there are a number of relatively minor revisions that should be made to the package and manuscript. Below, I list the changes which would allow me to recommend acceptance.

Manuscript and Documentation

I've made some minor typographical corrections and commentary to the manuscript in this attached PDF (10.21105.joss.03236_WithComments.pdf). Below are more general or substantial issues to be addressed.

Package summary and need statement

There is currently inconsistency across sources regarding the exact purpose and scope of the package which needs to be resolved.

[Statement of Need] The aim of this package is to provide an approachable and intuitive syntax to carry out common statistical tests across diverse statistical approaches.

[Summary] The statsExpressions package is designed to facilitate producing dataframes with rich statistical details for the most common types of statistical approaches and tests

Although the primary focus of this package is to get expressions containing statistical results, one can also use it to extract dataframes containing these details. https://indrajeetpatil.github.io/statsExpressions/articles/stats_details.html

These various sources emphasize somewhat different package features. For instance, the statement of need does not
mention expressions or consistent dataframe outputs, while the vignette description downplays what appears to be a key selling point of the package: consistent syntax and outputs.

There is a need for these various descriptions to become more homogeneous (consistent) with each other. This will
require updating in documentation also (README and Vignettes). I foresee that a hybrid of all three descriptions that clearly explains the purpose of the package is warranted.

Note that I have seen mixed answers regarding whether an explicit statement of need is required if it is already
covered in the summary (in the case you experience a lot of overlap). @mikldk

You should mention the specific user-base (target audience) this package would appeal to in the statement of need (if applicable).

I also want to flag that it was not immediately clear to me what an "expression" was, and I initially had to read
further into the README to know what the term "expression" was referring to. I recommend placing some added description in parentheses when first introducing the term expression (in both manuscript and README). For instance, "expression (i.e., a pre-formatted in-text statistical result)".

Comparison to Other Packages

I had a few issues with the contents of this section which require revision.

1. Only the last paragraph (L43-46) is directly discussing comparisons to other packages. The remaining content could be moved into a separate subsection, or alternatively, have a new header (e.g., "Package overview").

2. The language in L23-36 is somewhat speculative and encroaching informality in tone, particularly from L32. I suggest revising for objective tone, noting the following:

   • L23-31: I believe providing a single concrete example would be powerful. For instance, of the packages/functions statsExpressions covers, which of these have an object that return different data types for similar statistical tests?
   • L28-31: If you are suggesting that in order to access the statistical details of objects from some packages, there are many data types, I think this is a reasonable statement — but you do need to specify this: e.g., "Statistical test objects returned by functions in packages often have information stored in multiple types (e.g., package X, package Y). By contrast, statsExpressions …. "
   • I suggest removing or completely redeveloping L32-36, it does not add to the manuscript substantially.

Tidy Dataframes from Statistical Analysis

In the Tidy section starting from L47, the examples demonstrate that effectively only a single string needs to be changed in the statsExpressions::two_sample_test function to use a parametric/robust t-test. This section can be simplified and reduced by removing the code examples (which verge on being API examples), and stating directly in-line exactly what the functionality is. An example replacement could be: "A user can simply modify one argument in the statsExpressions::two_sample_test function to change between a robust, parametric or Bayesian analysis.".

The output code is not needed in the manuscript. If you believe it is crucial to show somewhere the dataframe format, I would dedicate a specific section and description to it, rather than show it for all functions by default.

You do not need full reproducible examples (setting seed; library loading) in the paper, particularly as these are already contained on the README.

I would also specify the dplyr grouped example in text (i.e., say it can be done, mention the function used), and point the reader to the vignettes or README where further details can be obtained.

Expressions for Plots

Figure 2 appears very useful and important, and immediately makes clear what the package offers. I suggest it appear first, with a description of the philosophy (i.e., gold standard), and then show the plot afterwards as an example output. You also can probably remove the generating code here, given the only unique argument is one string df$expression[1].

References

The APA reference is appearing incorrectly, you can address this LaTeX such that it treats American Psychological Association as a full singular name by wrapping in {{name}}. Also ensuring you specify which APA version (6 or 7), rather than the original text from 1985.

Other Feedback (related to package)

• The consistent handling of missing data is a clear strength of the package. However, it is not made clear in outputs
  if or how much data was omitted due to this. I did wonder if it would be beneficial to have this information in the
  dataframe (or warning), as it seems particularly important for grouped analyses. That said, I also do not see this as a concern precluding publication. I leave this to the digression of the author to decide.
• I was curious as to why the test for support of missing data is omitted if R version < 3.6? Just curious.
• I also noticed that the package has incomplete support for … notation, meaning additional arguments to the underlying packages are not supported. This itself is not an issue, but I do think it would be worthwhile mentioning in the manuscript or documentation that some arguments (likely only to be used by for specific/advanced reasons) are not supported in the package (as a footnote perhaps).

---

Please let me know if you have any questions or require clarification. Thank you for inviting me to review this work.

[humanfactors]

Edited to be clear that Only the last paragraph (L43-46) was what compared packages.

[IndrajeetPatil]

Wow, thank you so much @humanfactors for your comprehensive review! This was super-helpful for me to reorganize my thoughts: both on the paper and the package itself. :)

I am providing point-by-point responses below: All of my comments are in bolditalics and as a comment so that they stand out.

Overview

statsExpressions is an R package for quickly running various forms of statistical tests using a single framework (e.g.,
consistent syntax) and offers users "pre-formatted statistical expressions" for annotating plots, as well as consistent
dataframe outputs. The package covers a comprehensive range of commonly used statistical tests (at least in the
behavioral sciences).

I commend the author on their effort, the package appears to be well written and documented, with comprehensive tests and exhaustive use of continuous integration. I installed the software successfully, and ran a few statistical tests myself. I could see myself using this software in the future.

At this stage, there are a number of relatively minor revisions that should be made to the package and manuscript. Below, I list the changes which would allow me to recommend acceptance.

Manuscript and Documentation

I've made some minor typographical corrections and commentary to the manuscript in this attached PDF (10.21105.joss.03236_WithComments.pdf). Below are more general or substantial issues to be addressed.

Thanks! I have absorbed all of your comments in the revised version of the manuscript.

Package summary and need statement

There is currently inconsistency across sources regarding the exact purpose and scope of the package which needs to be resolved.

• [Statement of Need] The aim of this package is to provide an approachable and intuitive syntax to carry out common statistical tests across diverse statistical approaches.

• [Summary] The statsExpressions package is designed to facilitate producing dataframes with rich statistical details for the most common types of statistical approaches and tests

• Although the primary focus of this package is to get expressions containing statistical results, one can also use it to extract dataframes containing these details. https://indrajeetpatil.github.io/statsExpressions/articles/stats_details.html

These various sources emphasize somewhat different package features. For instance, the statement of need does not
mention expressions or consistent dataframe outputs, while the vignette description downplays what appears to be a key selling point of the package: consistent syntax and outputs.

There is a need for these various descriptions to become more homogeneous (consistent) with each other. This will
require updating in documentation also (README and Vignettes). I foresee that a hybrid of all three descriptions that clearly explains the purpose of the package is warranted.

This was unfortunately true. I have now included copies of the modified "Summary" and "Statament of Need" sections from the paper in README and vignette. They are consistent and homogeneous now and explain the key aims and its necessity now.

Note that I have seen mixed answers regarding whether an explicit statement of need is required if it is already
covered in the summary (in the case you experience a lot of overlap). @mikldk

You should mention the specific user-base (target audience) this package would appeal to in the statement of need (if applicable).

I am hesitant to do this because I do not want to pigeonhole the scope of this package to a particular field. But given that the package supports the most basic statistical tests - which are used across a wide number of disciplines - I think it is useful for most research fields.

I also want to flag that it was not immediately clear to me what an "expression" was, and I initially had to read
further into the README to know what the term "expression" was referring to. I recommend placing some added description in parentheses when first introducing the term expression (in both manuscript and README). For instance, "expression (i.e., a pre-formatted in-text statistical result)".

Good point. I have clarified this now, both in README and in the paper.

Comparison to Other Packages

I had a few issues with the contents of this section which require revision.

1. Only the last paragraph (L43-46) is directly discussing comparisons to other packages. The remaining content could be moved into a separate subsection, or alternatively, have a new header (e.g., "Package overview").

Agreed. I have now moved this to "Tidy Dataframes from Statistical Analysis" section, which is the correct place for it.

2. The language in L23-36 is somewhat speculative and encroaching informality in tone, particularly from L32. I suggest revising for objective tone, noting the following:

   • L23-31: I believe providing a single concrete example would be powerful. For instance, of the packages/functions statsExpressions covers, which of these have an object that return different data types for similar statistical tests?

   • L28-31: If you are suggesting that in order to access the statistical details of objects from some packages, there are many data types, I think this is a reasonable statement — but you do need to specify this: e.g., "Statistical test objects returned by functions in packages often have information stored in multiple types (e.g., package X, package Y). By contrast, statsExpressions …. "

**For both of these points, I am hesitant to do this because I do not want to single out a particular function or a package or. a data type. The argument is that there is diversity among all of the things you have mentioned and this statement is trivially true. So I don't think it is necessary to provide specific examples of such differences since the users must have already observed this in their prior experience. But, if you insist, I can reconsider this. **

  - I suggest removing or completely redeveloping L32-36, it does not add to the manuscript substantially.

Agreed. I have removed these lines.

Tidy Dataframes from Statistical Analysis

In the Tidy section starting from L47, the examples demonstrate that effectively only a single string needs to be changed in the statsExpressions::two_sample_test function to use a parametric/robust t-test. This section can be simplified and reduced by removing the code examples (which verge on being API examples), and stating directly in-line exactly what the functionality is. An example replacement could be: "A user can simply modify one argument in the statsExpressions::two_sample_test function to change between a robust, parametric or Bayesian analysis.".

Agreed and done!

The output code is not needed in the manuscript. If you believe it is crucial to show somewhere the dataframe format, I would dedicate a specific section and description to it, rather than show it for all functions by default.

I would prefer to have the code. I believe it gives the necessary context to show the consistency of the syntax, which is the biggest selling point for this package.

You do not need full reproducible examples (setting seed; library loading) in the paper, particularly as these are already contained on the README.

Agreed. I've removed these details.

I would also specify the dplyr grouped example in text (i.e., say it can be done, mention the function used), and point the reader to the vignettes or README where further details can be obtained.

I have removed this example.

Expressions for Plots

Figure 2 appears very useful and important, and immediately makes clear what the package offers. I suggest it appear first, with a description of the philosophy (i.e., gold standard), and then show the plot afterwards as an example output. You also can probably remove the generating code here, given the only unique argument is one string df$expression[1].

Done. But I do not want to remove the code. I want the reader to realize just how easy it is to do this and the only way they can appreciate that is by looking at the simplicity of the code.

References

The APA reference is appearing incorrectly, you can address this LaTeX such that it treats American Psychological Association as a full singular name by wrapping in {{name}}. Also ensuring you specify which APA version (6 or 7), rather than the original text from 1985.

Thanks, I have fixed this citation.

Other Feedback (related to package)

• The consistent handling of missing data is a clear strength of the package. However, it is not made clear in outputs
  if or how much data was omitted due to this. I did wonder if it would be beneficial to have this information in the
  dataframe (or warning), as it seems particularly important for grouped analyses. That said, I also do not see this as a concern precluding publication. I leave this to the digression of the author to decide.

I do not wish to complicate the current API to accommodate this. Note that this behavior is also consistent with the behavior of underlying packages (like stats) that silently remove mssing data.

• I was curious as to why the test for support of missing data is omitted if R version < 3.6? Just curious.

I think all tests ideally should be skipped for that version of R because the behavior of set.seed changed in that version.

• I also noticed that the package has incomplete support for … notation, meaning additional arguments to the underlying packages are not supported. This itself is not an issue, but I do think it would be worthwhile mentioning in the manuscript or documentation that some arguments (likely only to be used by for specific/advanced reasons) are not supported in the package (as a footnote perhaps).

I am a bit hesitant to do this because I might in the future end up supporting passing the ... to the underlying functions. But note that the package manual does and will always make clear what this argument does, so the user should never have trouble figuring it out.

---

Please let me know if you have any questions or require clarification. Thank you for inviting me to review this work.

[IndrajeetPatil]

Making one minor change: Adding dplyr example and code for Figure 2 in "Appendix" instead of completely removig it.

[humanfactors]

Hi @IndrajeetPatil, great job on addressing the comments above. I still have a minor issue remaining with the manuscript.

In referencing to providing a concrete example on L13-21, I think your reasoning is sensible and examples can be omitted (see your response below).

[L29-31] Furthermore, if someone wishes to utilize the objects returned by these packages downstream in their workflow, this is not straightforward either because even functions from the same package can return a list, a matrix, an array, a dataframe, etc., depending on the function.

[@IndrajeetPatil] For both of these points, I am hesitant to do this because I do not want to single out a particular function or a package or. a data type. The argument is that there is diversity among all of the things you have mentioned and this statement is trivially true. So I don't think it is necessary to provide specific examples of such differences since the users must have already observed this in their prior experience. But, if you insist, I can reconsider this.

However, I still believe the manuscript would benefit from addressing my second point ("If you are suggesting that in order to..."). To clarify this point, I was not necessarily stating that you need explicit examples; but that the text on L29-31 is unclear. My assumed interpretation is that: In many packages, the statistical test objects returned by the test functions
do not have all required information (e.g., degrees of freedom, significance, Bayes factor) accessible in a consistent data type. Depending on the specific test object and statistic in question, details may returned as a list, a matrix, an array, or a dataframe.

If my explanation above is correct, this is a different meaning than what is implied by saying that "functions from the same package can return different types", with the type "depending on the function", which implies either multiple function return values or multiple functions for a test object.

I believe it is important to revise and make this small sentence clear, because it is major strength of your package!

---

Further information

Basically I am assuming what you are getting at, is that for a single object (let's take BFoneSample from BayesFactor), to even begin to extract the required details, your workflow would being looking like below, where there many data types required to get all object information:

data(sleep)
myobject = BayesFactor::ttestBF(x = sleep$extra[sleep$group==1], y = sleep$extra[sleep$group==2], paired=TRUE)
bayes_factor = BayesFactor::extractBF(myobject)
sample_size = nrow(attributes(myobject)$data)
attributes(myobject)$bayesFactor[["bf"]]
ps_median = parameters::model_parameters(myobject)$Median

And once you factor in switching between multiple approaches (robust/parametric), it becomes completely unmanageable.

[IndrajeetPatil]

Thanks, Michael, for pushing me to be clearer here and for you suggested rephrasing. This is what I have changed the paragraph to. Do you find this more reasonable?

Statistical packages exhibit substantial diversity in terms of their syntax and
expected input and output data type. For example, some functions expect vectors
as inputs, while others expect dataframes. Depending on whether it is a repeated
measures design or not, functions from the same package might expect data to be
in wide or tidy format. Some functions can internally omit missing values, while
other functions error in their presence. Furthermore, the statistical test
objects returned by the test functions might not have all required information
(e.g., degrees of freedom, significance, Bayes factor, etc.) accessible in a
consistent data type. Depending on the specific test object and statistic in
question, details may be returned as a list, a matrix, an array, or a dataframe.
This diversity can make it difficult to easily access all needed information for
hypothesis testing and estimation, and to switch from one statistical approach
to another.

[humanfactors]

Perfect @IndrajeetPatil !