[REVIEW]: ExoTiC-ISM: A Python package for marginalised exoplanet transit parameters across a grid of systematic instrument models

[whedon]

Submitting author: @ivalaginja (Iva Laginja)
Repository: https://github.com/hrwakeford/ExoTiC-ISM
Version: v2.0.0
Editor: @xuanxu
Reviewer: @hayesla, @mattpitkin
Archive: 10.5281/zenodo.3923986

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/8ffda44e5537e748cbe8f1c668613e24"><img src="https://joss.theoj.org/papers/8ffda44e5537e748cbe8f1c668613e24/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/8ffda44e5537e748cbe8f1c668613e24/status.svg)](https://joss.theoj.org/papers/8ffda44e5537e748cbe8f1c668613e24)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@hayesla & @mattpitkin, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @xuanxu know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @hayesla

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@ivalaginja) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @mattpitkin

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@ivalaginja) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @hayesla, @mattpitkin it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[xuanxu]

@ivalaginja, @hayesla, @mattpitkin : this is the review thread for the paper. All of our communications will happen here from now on.

Both reviewers have checklists at the top of this thread with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#2281 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks, but that's not a hard deadline. Please let me know if any of you require some more time. We can also use Whedon (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@xuanxu) if you have any questions/concerns.

[xuanxu]

@whedon check references

[whedon]

Reference check summary:

OK DOIs

- 10.1038/nature16068 is OK
- 10.1088/0004-637X/814/1/66 is OK
- 10.3847/1538-3881/ab7b78 is OK
- 10.3847/1538-3881/aa9e4e is OK
- 10.1126/science.aah4668 is OK
- 10.1038/s41586-018-0067-5 is OK
- 10.3847/0004-637X/819/1/10 is OK
- 10.1093/mnras/stu1975 is OK
- 10.1086/345520 is OK
- 10.1051/0004-6361/200913675 is OK
- 10.1051/0004-6361/201423804 is OK
- 10.5281/zenodo.2573885 is OK
- 10.1117/12.447161 is OK
- 10.1109/MCSE.2007.55 is OK
- 10.5281/zenodo.2893252 is OK
- 10.5281/zenodo.3644238 is OK
- 10.1109/MCSE.2011.37 is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.5281/zenodo.3378022 is OK
- 10.1088/1538-3873/aa65b0 is OK
- 10.5281/zenodo.157363 is OK
- 10.5281/zenodo.35265 is OK
- 10.1086/683602 is OK
- 10.1088/1538-3873/aaf5ad is OK

MISSING DOIs

- https://doi.org/10.25080/majora-92bf1922-00a may be missing for title: Data structures for statistical computing in python

INVALID DOIs

- None

[mattpitkin]

@ivalaginja The installation instructions work well. I'd just recommend explicitly adding how to activate the conda environment before running setup.py:

conda activate exoticism
python setup.py develop

[mattpitkin]

This isn't a requirement, but are there plans to have continuous integration (e.g., through Travis) that can run your test suite? Do you know how much of the main functionality of the code the current test suite covers?

[mattpitkin]

For some references they don't seem to render the journal. I think this is due to the paper.bib file containing the AAS macros (e.g., \mnras, \aap) for several papers - these macros should be replaced with the full or abbreviated journal name.

There are also a couple that need the DOI added:

• Claret, A
• McKinnet, W the DOI is 10.25080/Majora-92bf1922-00a

[mattpitkin]

The functionality of the code, and required inputs in the configuration file, are well documented in the tutorial notebook, so I have ticked the Functionality documentation check box. There is no specific documentation for the API, so in the future I would recommend there is an autogenerated doc page that does contain the API, but this is not required for this review.

[mattpitkin]

Once the references are fixed I'm happy to sign-off the review.

Just to summarise and slightly add to what I've written above, I've a couple of general recommendations but these are not requirements for the review:

• add some continuous integration for the tests (maybe run the described example as a test and compare the output to the expected output)
• add some autogenerated documentation that includes documentation for the API

One very minor typo fix for the paper: in the first sentence change "has" to "have".

I should just note that I ran the example on a remote machine (my laptop's running low on space, so I had to ssh into my work desktop to clone the repo there!). When I tried running the example it performed the first stage of fitting, but hung when it got to the second. I got things to work by changing to plotting = False in the config file, after which it ran through everything ok.

[ivalaginja]

Thanks for all your comments @mattpitkin! To address your questions:

1. I have updated the Quickstart section to include the activation of the environment, thanks for that!
2. We currently use GitHub Actions to run our test suite in every open PR (e.g. here where I merged a PR based on your review comments), alternatively this can be run locally with pytest. Our current test coverage is very thin and probably in the lower single digits percent-wise. However, we aim at improving on that in the future which is why we already set up CI, so that all that's left is actually to write more senseful tests. We have an open issue for this here.
3. We have an issue open for auto-generated documentation here, we just decided to punt it to later since we ran into some problems in the first trial to set that up.
4. I changed the word in the first sentence of the paper.
5. I took out all the AAS macros in the paper.bib
6. I noted the problem you ran into on the remote machine and created an issue for it here.
7. I added the DOI for the pandas paper but can partout not find the DOI for the Claret 2000 paper. I have reached out to an author who recently cited it to ask whether they know the DOI, but I am not sure this will help. I will ask @hrwakeford (co-author) if she knows more. What would be our alternative if we can't find the DOI?

[ivalaginja]

I just saw that Doe et al. also doesn't have a DOI, facing the same problem here.

[mattpitkin]

Thanks for your detailed responses and for opening the issue.

I think you're right that there just aren't DOIs for Claret or Doe et al, so the references are fine without them. I'll tick of that last box for my review.

[mattpitkin]

@whedon check references

[whedon]

Reference check summary:

OK DOIs

- 10.1038/nature16068 is OK
- 10.1088/0004-637X/814/1/66 is OK
- 10.3847/1538-3881/ab7b78 is OK
- 10.3847/1538-3881/aa9e4e is OK
- 10.1126/science.aah4668 is OK
- 10.1038/s41586-018-0067-5 is OK
- 10.3847/0004-637X/819/1/10 is OK
- 10.1093/mnras/stu1975 is OK
- 10.1086/345520 is OK
- 10.1051/0004-6361/200913675 is OK
- 10.1051/0004-6361/201423804 is OK
- 10.5281/zenodo.2573885 is OK
- 10.1117/12.447161 is OK
- 10.1109/MCSE.2007.55 is OK
- 10.5281/zenodo.2893252 is OK
-  10.25080/Majora-92bf1922-00a  is OK
- 10.5281/zenodo.3644238 is OK
- 10.1109/MCSE.2011.37 is OK
- 10.1051/0004-6361/201322068 is OK
- 10.3847/1538-3881/aabc4f is OK
- 10.5281/zenodo.3378022 is OK
- 10.1088/1538-3873/aa65b0 is OK
- 10.5281/zenodo.157363 is OK
- 10.5281/zenodo.35265 is OK
- 10.1086/683602 is OK
- 10.1088/1538-3873/aaf5ad is OK

MISSING DOIs

- None

INVALID DOIs

- None

[hrwakeford]

The reference to Claret 2000 has two sources an A&A paper and a catalog in VizieR
Paper - https://ui.adsabs.harvard.edu/abs/2000A%26A...363.1081C/abstract
VizieR - http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/A+A/363/1081

The Sherpa package lists the following link as having the correct DOI for citation but I struggled to navigate to where that might be but this is where to start
https://zenodo.org/record/3631574#.XvC_bi-ZNhE

[ivalaginja]

https://zenodo.org/record/3631574#.XvC_bi-ZNhE

You're right about Sherpa, however this is only for the package itself, for which we already added the DOI; what's missing is the DOI for the paper they want us to cite.

As for Claret, it really seems like there's no DOI to be found :P

[mattpitkin]

I've checked my last tick box, so my sign off is complete.

[hayesla]

my apologies for my delay - I will work on this today!

[hayesla]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[hayesla]

I think the pandas 2020 reference should be

@software{jeff_reback_2020_3644238,
  author       = {Jeff Reback and
                  Wes McKinney and
                  jbrockmendel and
                  Joris Van den Bossche and
                  Tom Augspurger and
                  Phillip Cloud and
                  gfyoung and
                  Sinhrks and
                  Adam Klein and
                  Matthew Roeschke and
                  Jeff Tratner and
                  Chang She and
                  Simon Hawkins and
                  William Ayd and
                  Terji Petersen and
                  Jeremy Schendel and
                  Andy Hayden and
                  Marc Garcia and
                  MomIsBestFriend and
                  Vytautas Jancauskas and
                  Pietro Battiston and
                  Skipper Seabold and
                  chris-b1 and
                  h-vetinari and
                  Stephan Hoyer and
                  Wouter Overmeire and
                  alimcmaster1 and
                  Mortada Mehyar and
                  Kaiqi Dong and
                  Christopher Whelan},
  title        = {pandas-dev/pandas: Pandas 1.0.1},
  month        = feb,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {v1.0.1},
  doi          = {10.5281/zenodo.3644238},
  url          = {https://doi.org/10.5281/zenodo.3644238}
}

[hayesla]

Overall looks good to me. The Quickstart guide worked well, and I was able to go through the example notebook easily after this and play around with the codebase.

Once the reference (above) is fixed I have nothing blocking and just some suggested comments

• I agree with @mattpitkin, a specific description of the API, or even just an overview in the README would be good to make it clear what is available. And of course something like an automated documentation of the API, maybe https://readthedocs.org/ is something to consider for the future.

• For the Community Guidelines I would suggest adding a few sentences to the README explicitly mention where users can report issues or problems seek support, maybe within another heading?

• The tutorial notebook is great and is well documented to describe the functionality of the code and apply it to some nice examples. I would certainly suggest moving this to some documentation somewhere else in the future also.

• Future continuous integration functionality would be great to see.

[ivalaginja]

@hayesla thank you for your comments! I fully agree with you on all you said, and we will work in the near future to get these things set up; issues for this already exist in the repository.

As for the reference, thanks for pointing it out. I will for sure update the author list, however, the DOI and the version wouldn't work for our paper since we used pandas version v0.24.2, for which there is no explicit Zenodo entry. My idea was to stick in the concept DOI instead, which will always point to the most recent release since there is no DOI for the particular version we use. And I could adjust the version number in the Bibtex manually.

Would that be an acceptable solution?

[whedon]

OK. 10.5281/zenodo.3923986 is the archive.

[xuanxu]

Thanks @hayesla and @mattpitkin for your reviews!

[xuanxu]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1522

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1522, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

👋 @ivalaginja - here are some suggested changes for the paper:

• "with the goal to identify" <- "with the goal of identifying"
• "For these giant, close-in exoplanets the most" <- "For these giant, close-in exoplanets, the most"
• "to be well mixed" <- "to be well-mixed"
• "systematic models which yields" <- "systematic models, which yields"
• "There are a number of Python solutions to create and fit transiting planet light curves, however ExoTiC-ISM" <- "While there are a number of Python solutions to create and fit transiting planet light curves, ExoTiC-ISM"
• "(Wakeford et al., 2016), however, can be adapted to other instruments." <- "(Wakeford et al., 2016). However, this can be adapted to other instruments."
• "Freeman et al., 2001) as well as" <- "Freeman et al., 2001), as well as"
• "The development in Python" <- "Development in Python"

In addition, please fix the cases in the references, for example
"Astronomical data analysis software and systems xvi, Astronomical society of the pacific conference series"
You will need to do this by protecting (with {}s) cases in the .bib file. A bunch of journals also need to be fixed, as we don't know or expand standard astro abbreviations, such as aap, apj, apjl, mnras, etc.

once you've done this, please issue @whedon generate pdf (and iterate if needed), then let me know it's ready to go.

[danielskatz]

@ivalaginja - this is just a reminder that we are waiting on the changes to the paper and bib as requested in the issue above, then your paper can be published

[ivalaginja]

Thanks for the reminder! I did not forget, as a matter of act it's on the top of my work todo list, I was just relocating this week so I am a little short on time. I should be able to get to it this weekend or Monday!

[ivalaginja]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[ivalaginja]

@danielskatz this should be ready now

[danielskatz]

Thanks - looks good!

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1539

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1539, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02281 joss-papers#1540
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02281
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[danielskatz]

Thanks to @hayesla, @mattpitkin for reviewing!
And @xuanxu for editing!

Congratulations to @ivalaginja (Iva Laginja) and co-author!!

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02281/status.svg)](https://doi.org/10.21105/joss.02281)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02281">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02281/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02281/status.svg
   :target: https://doi.org/10.21105/joss.02281

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[ivalaginja]

Thanks to everyone!!