[PRE REVIEW]: AutoRA: Automated Research Assistant for Closed-Loop Empirical Research

[editorialbot]

Submitting author: @musslick (Sebastian Musslick)
Repository: https://github.com/AutoResearch/autora-paper
Branch with paper.md (empty if default branch): main
Version: v4.0.0
Editor: @jbytecode
Reviewers: @seandamiandevine, @szorowi1
Managing EiC: Chris Vernon

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e"><img src="https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e/status.svg)](https://joss.theoj.org/papers/be6d470033fbe5bd705a49858eb4e21e)

Author instructions

Thanks for submitting your paper to JOSS @musslick. Currently, there isn't a JOSS editor assigned to your paper.

@musslick if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.03 s (1508.4 files/s, 286008.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                        21            451              0           1177
Jupyter Notebook                 8              0           5853            887
YAML                            11             35             27            455
TOML                             1             16              0            118
TeX                              1             12              0             81
Python                           3             28             29             45
HTML                             1              1              0             38
JSON                             2              0              0             22
JavaScript                       1              1              0             15
-------------------------------------------------------------------------------
SUM:                            49            544           5909           2838
-------------------------------------------------------------------------------

Commit count by author:

  1028	John Gerrard Holland
   284	benwandrew
   281	Sebastian Musslick
   202	TheLemonPig
   183	Roger Guimera
   100	George Dang
    84	Younes Strittmatter
    50	dependabot[bot]
    50	listar2000
    46	Chad C Williams
    11	Ben Andrew
     8	musslick
     7	seeslab
     5	Marina Dubova
     4	chadcwilliams
     3	Admin
     3	Hannah Even
     2	Ioana
     2	rguimera
     1	blinodelka
     1	hollandjg
     1	ioanam25
     1	kphan20

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1549

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- No DOI given, and none found for title: Bayesian machine scientist for model discovery in ...
- No DOI given, and none found for title: An evaluation of experimental sampling strategies ...
- 10.31222/osf.io/ysv2u may be a valid DOI for title: Against theory-motivated experimentation in scienc...
- 10.1016/j.jbef.2017.12.004 may be a valid DOI for title: Prolific. ac—A subject pool for online experiments
- No DOI given, and none found for title: Scikit-learn: Machine learning in python
- No DOI given, and none found for title: Deep symbolic regression: Recovering mathematical ...
- No DOI given, and none found for title: A unified framework for deep symbolic regression
- No DOI given, and none found for title: Discovering symbolic models from deep learning wit...
- 10.1126/sciadv.aav6971 may be a valid DOI for title: A Bayesian machine scientist to aid in the solutio...

INVALID DOIs

- https://doi.org/10.31234/osf.io/c2ytb is INVALID because of 'https://doi.org/' prefix

[crvernon]

👋 @musslick - thank you for your submission to JOSS. It seems the main branch is missing the actual autora package. Can you confirm please? Thanks!

[crvernon]

@editorialbot invite @jbytecode as editor

👋 @jbytecode want to take this one on as editor?

Note: see my comment above about the missing package. I checked their other branches and it seems accidental on their part. We will rerun the code specs after they update it.

[editorialbot]

Invitation to edit this submission sent!

[jbytecode]

@editorialbot assign me as editor

[editorialbot]

Assigned! @jbytecode is now the editor

[jbytecode]

@musslick - Thank you for submitting your work to JOSS. I am the handling editor of this submission. As our track editor @crvernon states, the main branch of the repo contains a test file and some other Python files for documenting. It is a better approach to host the source code in main branch and possibly carrying the paper markdown in a separate branch, e.g., joss-paper. The other thing, adding the folders created by editors, e.g., .vscode to .gitignore file would be beneficial just because they are user-specific.

Could you please address these issues and ping me again?

Thank you in advance.

[musslick]

@jbytecode @crvernon Thank you for handling this so quickly!

We apologize for the confusion about the location of our code. The problem is that our code is intentionally distributed across multiple Python packages with a shared namespace. We made this design choice for two reasons. For users, we wanted to minimize time and storage required for individual installations of autora (a single installation of autora may only require a small subset of packages within the namespace). For contributors, we wanted to minimize dependency conflicts that they have to deal with. autora lists all of its packages (our total code contribution) in the pyproject.tml.

However, we do have two required dependencies (autora-core) and (autora-synthetic). autora-core contains most of the workflow mechanic for autora and can be found here: https://github.com/AutoResearch/autora-core. If you prefer the review to happen within a single repository, then we could use this one.

Please let us know about your preference. We are happy to move the paper (and review of the code) to autora-core which is a required dependency of autora and hosts the code for the main workflow mechanics. I could then host the paper in a separate branch named "joss-paper" in the repository for autora-core. Alternatively, we can keep the paper in the "main" autora package and I supply a list with the repositories of all relevant subpackages. Either way, I created a separate branch "joss-paper" on the autora repository for all future work on the paper.

Our apologies again for this complication! We appreciate your guidance on how to proceed.

[jbytecode]

@musslick - We talked about this situation with our track editor. Additionally, one of our submissions resembled this situation, we can use it as an example to continue. Please take a look the repository here:

https://github.com/PdIPS/CBX

This repository hosts a JOSS submission, e.g., paper and bibtex. It provides a clean explanation that the submission links to other repositories in which the source code is hosted. So I suggest

• Opening a third repository that holds the submission only
• Please make a clean explanation that the repository hosts a JOSS submission and links to the source code repositories in README.md.

After finishing this procedure, please let me know the new repository so I can change the submission address in the pre-review issue. In your situation, we will have a repository for submission, a repository for autora-core, and a repository for autora-synthetic, respectively, with last two linked to the first one. We will ask our reviewers to examine and review both of them.

Please ping me if there is anything unclear or if you get into any trouble.

Thank you in advance.

[musslick]

@jbytecode

Thank you for pointing us in the right direction.

We opened a third repository for the submission only: https://github.com/AutoResearch/autora-paper

In this repository, we link the two main packages autora-core and autora-synthetic, as well as the main package autora which provides a common namespace for installation of and documentation for all sub packages.

The AutoRA framework includes a number of optional dependencies that we also listed in the linked submission, primarily to illustrate the scope of the code contribution. However, we agree that reviewing autora-core and autora-synthetic would be most appropriate (potentially together with the autora repository).

We believe that our submission falls within the expertise of the following reviewers:

• Sean Devine (seandamiandevine)
• Benedikt Ehinger (behinger)

Thanks again for your help!

[jbytecode]

@editorialbot set https://github.com/AutoResearch/autora-paper as repository

[editorialbot]

Done! repository is now https://github.com/AutoResearch/autora-paper

[jbytecode]

@editorialbot check repository

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.01 s (524.1 files/s, 35115.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                         2             33              0            100
TeX                              1             14              0             90
YAML                             1              1              5             25
-------------------------------------------------------------------------------
SUM:                             4             48              5            215
-------------------------------------------------------------------------------

Commit count by author:

    11	Sebastian Musslick
     3	musslick
     2	Younes Strittmatter

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1549

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[jbytecode]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.31222/osf.io/ysv2u is OK
- 10.1016/j.jbef.2017.12.004 is OK
- 10.48550/arXiv.1912.04871 is OK
- 10.48550/arXiv.2006.11287 is OK
- 10.1126/sciadv.aav6971 is OK
- 10.31234/osf.io/c2ytb is OK

MISSING DOIs

- Errored finding suggestions for "Bayesian machine scientist for model discovery in ...", please try later
- Errored finding suggestions for "An evaluation of experimental sampling strategies ...", please try later
- No DOI given, and none found for title: Scikit-learn: Machine learning in python
- No DOI given, and none found for title: A Unified Framework for Deep Symbolic Regression

INVALID DOIs

- None

[jbytecode]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[jbytecode]

👋👋👋 Dear @seandamiandevine, @behinger 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled:

AutoRA: Automated Research Assistant for Closed-Loop Empirical Research

You can find more information at the top of this Github issue (#6769).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

Thank you in advance!

[seandamiandevine]

@jbytecode

Yes, I can review this article. Happy to help!

[jbytecode]

@editorialbot add @seandamiandevine as reviewer

[editorialbot]

@seandamiandevine added to the reviewers list!

[jbytecode]

@seandamiandevine - Thank you for accepting our invitation. Whenever we set a second reviewer, the review will start in a separate thread. I will provide the instructions there.

[jbytecode]

@musslick - Do you have any suggestions for additional potential reviewers? You can use the list provided in https://reviewers.joss.theoj.org/lookup and filter reviewers by their preferred language and expertise.

[musslick]

@jbytecode Sure, we can recommend the following additional reviewers based on their expertise:

• Sam Zorowitz (szorowi1)
• Haoxue Fan (haoxue-fan)
• Chang Huan Lo (lochhh)

Happy to provide more if needed!

[jbytecode]

👋👋👋 Dear @szorowi1 👋👋👋

Would you be willing to assist in reviewing this submission for JOSS (Journal of Open Source Software)?

JOSS publishes articles about open source research software. The submission I'd like you to review is titled:

AutoRA: Automated Research Assistant for Closed-Loop Empirical Research

You can find more information at the top of this Github issue (#6769).

The review process at JOSS is unique: it takes place in a GitHub issue, is open, and author-reviewer-editor conversations are encouraged. If you have any questions please let me know.

This is the pre-review issue. After setting at least 2 reviewers we will start the review process in a separate thread. In that thread, there will be 23 check items for each single reviewer.

Thank you in advance!

[szorowi1]

Hi all, hope everyone is well! I am on vacation for the next two weeks, but I'd be happy to review when I return. If someone else is available before then, then that's also ok by me!

[jbytecode]

@szorowi1 - Thank you for the response. It would be great if you could find a time to review this submission after the vacation.
I am now adding you as a reviewer. The review process will start in a separate thread (GitHub issue).

[jbytecode]

@editorialbot add @szorowi1 as reviewer

[editorialbot]

@szorowi1 added to the reviewers list!

[jbytecode]

@editorialbot start review

[editorialbot]

OK, I've started the review over in #6839.