[REVIEW]: Fear Learning and Anxiety Response (FLARe): Open-source software for remote human fear conditioning experiments.

[whedon]

Submitting author: @McGregor14 (Thomas McGregor)
Repository: https://github.com/flare-kcl/flare-app
Version: v1.0.0
Editor: @oliviaguest
Reviewer: @peircej, @KirstensGitHub, @EleonoraVagnoni
Archive: Pending

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/293219216840edc601e494c896f183c1"><img src="https://joss.theoj.org/papers/293219216840edc601e494c896f183c1/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/293219216840edc601e494c896f183c1/status.svg)](https://joss.theoj.org/papers/293219216840edc601e494c896f183c1)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@peircej & @KirstensGitHub & @EleonoraVagnoni, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @oliviaguest know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @peircej

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@McGregor14) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @KirstensGitHub

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@McGregor14) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @EleonoraVagnoni

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@McGregor14) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @peircej, @KirstensGitHub, @EleonoraVagnoni it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.actpsy.2007.11.007 is OK
- 10.1016/j.brat.2019.103475 is OK
- 10.1101/2020.12.18.423504 is OK

MISSING DOIs

- 10.1002/da.23146 may be a valid DOI for title: Large-scale remote fear conditioning: demonstration of associations with anxiety using the FLARe smartphone app

INVALID DOIs

- None

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.99 s (157.5 files/s, 52248.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                            11              4              0          40685
TypeScript                      93            786            461           7092
XML                             14              9              3            421
Java                             7             67             30            389
YAML                             3             83             20            370
Markdown                         4             50              0            181
JavaScript                       8             22             16            159
Gradle                           3             29            120            150
Bourne Shell                     1             21             36            126
Objective-C                      3             27              4             99
DOS Batch                        1             25              2             76
Swift                            1             14             19             64
TeX                              1              4              0             46
Starlark                         1              2              1             16
C/C++ Header                     3              4              0             10
TOML                             1              0              0              7
ProGuard                         1              1              9              0
-------------------------------------------------------------------------------
SUM:                           156           1148            721          49891
-------------------------------------------------------------------------------


Statistical information for the repository '14b4b09d77e42563046ec3c0' was
gathered on 2021/05/10.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Nathan                           3           286              3           39.43
Nathan Horrigan                 18           429             15           60.57

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Nathan Horrigan             697          162.5          2.3                6.60

[whedon]

Failed to discover a valid open source license.

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[oliviaguest]

@McGregor14 can you add a licence, please?

[oliviaguest]

Hi all! 👋 Thank you so much @peircej, @KirstensGitHub, @EleonoraVagnoni for accepting to review this. Please read the instructions above. Any questions, feedback on the paper, etc., please post here. Any very code-specific questions, suggestions, etc., please use the issues in the code repo and link to them from this thread, so we can all keep track of them. 🌸

For examples of how this process plays out feel free to skim previous reviews, such as: #2285 and #2348. ☺️

[McGregor14]

Hi @oliviaguest,

Sorry for the delay, I am writing up my thesis at the moment so a little bit distracted!

We have now added a license for the app repository (https://github.com/flare-kcl/flare-app/blob/develop/LICENSE) and will add one to the portal repository ASAP.

[whedon]

👋 @peircej, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @EleonoraVagnoni, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @KirstensGitHub, please update us on how your review is going (this is an automated reminder).

[oliviaguest]

@McGregor14 thanks and keep us posted! ☺️

[McGregor14]

@oliviaguest portal license is up! https://github.com/flare-kcl/flare-portal/blob/main/LICENSE

[oliviaguest]

@whedon check repository

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.22 s (708.6 files/s, 235054.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                            11              4              0          40669
TypeScript                      93            786            461           7092
XML                             14              9              3            421
Java                             7             67             30            389
YAML                             3             83             20            370
Markdown                         4             50              0            181
JavaScript                       8             22             16            159
Gradle                           3             29            120            151
Bourne Shell                     1             21             36            126
Objective-C                      3             27              4             99
DOS Batch                        1             25              2             76
Swift                            1             14             19             64
TeX                              1              4              0             46
Starlark                         1              2              1             16
C/C++ Header                     3              4              0             10
TOML                             1              0              0              7
ProGuard                         1              1              9              0
-------------------------------------------------------------------------------
SUM:                           156           1148            721          49876
-------------------------------------------------------------------------------


Statistical information for the repository '719f12b5720c3efe6a119aeb' was
gathered on 2021/06/05.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Nathan                           3           286              3           39.43
Nathan Horrigan                 18           429             15           60.57

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Nathan Horrigan             697          162.5          2.3                6.60

[oliviaguest]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[oliviaguest]

@McGregor14 thanks! Can you clarify if you are happy with explaining to users/reviewers in your documentation how to install this software? I got an email from @EleonoraVagnoni that this is not clear enough at the moment. I personally think this should be accessible (as users to collect data) for people who might not be familiar with the process of installing, e.g., the required background software. What do you think? ☺️

[McGregor14]

Nathan and I will work on it this afternoon. Apologies again for the delay.

[McGregor14]

@oliviaguest just checking, would you like this added to the readme or the paper.md document?

[KirstensGitHub]

Also, as a small note, where the documentation suggests running "fab up" I had to enter "fab start"

[KirstensGitHub]

@McGregor14 can you advise?

[McGregor14]

Hello @KirstensGitHub, sorry for the late reply.

I have nudged @nathanhorrigan by email, so hopefully he will get back shortly.

@oliviaguest

I think it would be worth me briefly explaining the current situation and roles of the authors for future reference, too.

FLARe is a King’s College London project that has been going for around 5 years, basically creating and using different versions of the app for research. I have been the scientific lead on the 3rd and 4th versions (which includes the one currently under review). KLP was the scientific lead prior to me (versions 1 and 2) and continues to provide significant input on the project. TCE and GB have led the project since its inception and KSY is an expert on the fear conditioning paradigm who has given us a lot of advice throughout the latest round of development.

Having previously worked with a different industry partner, for the most recent version of the app we went through a full rebuild with Torchbox and have made the software open source, hence our submission to JOSS. Nathan was the primary developer on the project, working on both the app and portal. MC initially developed the portal before Nathan took over both elements towards the end. BH was the project manager overseeing our day-to-day work with Torchbox. PV was in charge of strategy, helping ensure FLARe met the expectations of our team and our funders.

Nathan has since left Torchbox and moved to a new company that works on US time, as well as completing his undergraduate degree here in the UK. I think this has, understandably, made it difficult for him to respond promptly. As you can see from the roles outlined above, our team (at King’s College London) have a lot of expertise in using the product to conduct research and the science underlying it, but are not best placed to advise on the technical issues being addressed so far, here.

Hope this all makes sense and let me know if you have any questions.

[Morsey187]

Hi @KirstensGitHub,

It looks like the authentication ("must_accept_terms" decorator) is set to compare against a field ("researcher_terms_updated_at") that doesn't have a value set by default, I presume this was to prevent people from signing up on the production site before any terms and conditions were set, however this should be handled automatically for local development or mentioned in the setup instructions at least.

I've written a quick fix below, but in the meantime I'll update the code, so we can avoid having to carry out this step for local builds.

Fix:

Within fab sh run ./manage.py shell, then paste the code snipppet below and press enter. Then try accepting the terms and condtions again and you should be redirected to the dashboard.

from flare_portal.site_config.models import SiteConfiguration
from datetime import datetime
config = SiteConfiguration.get_solo()
config.researcher_terms_updated_at = datetime.now()
config.save()

exit()

I've also had a look at the Portal and App repositories today and have made some notes on the documentation, so hopefully later this afternoon I can open some pull requests to address issues like "fab up" being the wrong command as well as add some further details to make setting up the FLARe app easier in relation to andriod and ios.

Thanks,
Ben

Edit: forgot to add config.save() to the script

[KirstensGitHub]

thanks so much @Morsey187 I'll try that out and let you know how it goes!

[EleonoraVagnoni]

Hello,

I am sorry, I think I am stuck. I think I do not have enough coding expertise to proceed.

[Morsey187]

Hi @EleonoraVagnoni,

If I had to hazzard a guess based off the previous comments in this thread and the picture provided, I'd assume you're not running the commands in the correct folder directory. In order to run the project's commands you will have to be in same directory of the package.json file they are refering to, in this case the project's root directory.

If you run ls in your terminal and find yourself not within the correct folder directory. you should try running terminal commands i.e. cd flare-app or cd downloads/flare-app. If that's difficult you I believe you can drag and drop fodlers into terminals on OSX to get their folder path, if you do this and press enter that should also take you to the correct folder.

If thats not the issue could you provide me with a bit more info such as what's outputted when you run node -v in your terminal and what any of the following commands output for you?

• npm i
• npx @react-native-community/cli doctor
• cd ios && pod install
• npm run ios

[EleonoraVagnoni]

This is what I get if I run node -v

[Morsey187]

Hi @EleonoraVagnoni,
It looks like you need to install the missing dependencies listed in that screenshot, for IOS that would be watchman which you can do by following the instructions here, or if you have homebrew installed you can do this by running brew install watchman.
Similarly with the Andriod build, you'll have to install the listed dependencies that are crossed out.

Thanks,
Ben

[McGregor14]

Hello @oliviaguest,

It has been a little while since I have heard anything on this and I just wondered if there is anything we can do to help things get moving again or what the next steps are?

All the best,

Tom

[peircej]

For my part, I'm stuck ticking the boxes for the following:

• Example usage: I couldn't see any documentation showing usage.
• Functionality documentation. As above. For the phone app I couldn't see any signs of documentation other than how to build/install. For the portal there is a mention that, after installing the portal docs can be viewed at localhost but it would surely be worth providing documentation online. If the authors want users to get involved then being able to see what they're doing before committing to a substantial install process is important I think
• Community guidelines. No sign of these. The issues lists for the 2 projects are essentially unused. No suggestion of a forum/discussion site for users or contributors
• State of the field. I couldn't see any real discussion of the alternative options. There are many platforms out there with simpler processes for the user to run online behavioural studies which presumably include, but would not be limited to, fear conditioning. For instance, lab.js, jsPsych or PsychoPy can all generate a very wide range of behavioural studies and deploy them to online servers to run large numbers of participants at a time.

Overall I was left with a sense that this is a project built for the existing group that wrote and uses the software, and therefore doesn't require the documentation, rather than a one that expects a broader audience or community. To become an active package with wider appeal I would suggest working on publishing documentation about the software so that users can decide if it's the right package for them. Cross-linking the different sites will help that. Currently there's a webpage at https://www.kcl.ac.uk/research/flare but it doesn't point to the repositories or any documentation, and the readme files don't point back to that page.

[McGregor14]

Thanks for the feedback @peircej. I will see what we can do about these issues and report back.

[EleonoraVagnoni]

Hello,
sorry for the late reply. I have tried to run brew install watchman. I am still stuck.
I think it is too difficult for a person with limited coding experience to download the software. I have been looking online for the errors I got but I have been copying and pasting lines without really understanding what I was doing. Even with your step-by-step guidance, I was not able to download it and use it. The documentation available is not enough for a person with limited coding experience.

[oliviaguest]

@EleonoraVagnoni this sounds like a very important point — thanks for bringing it up.

[McGregor14]

Yes, thanks all for the feedback. I think I was too focused on the author submission guidelines and should have spent time checking the reviewer checklist to see where the software stood and what was practically required to be classed as open-source. I will do my best to work with the team to fix the issues you have raised and will get back to you once I feel they have been addressed.

Thanks again for all your time so far.

[oliviaguest]

@McGregor14 no problem! Do you have any guess as to how long you might take? 😊

[McGregor14]

@oliviaguest some of the team are already off for Christmas, so I will wait until the new year to raise this with everyone. I think we could probably estimate how long the work will take and when it can start by the end of January and I will provide you with an update then if that's ok?

[oliviaguest]

@McGregor14 OK, when you are all back let us know! 😊

[McGregor14]

Hello @oliviaguest,

I have discussed the issues outlined above with my supervisor and, unfortunately, we have decided that it is not currently feasible for us to address them all in the foreseeable future. This is due to a number of reasons, mainly funding and capacity within our team (which is currently a bit stretched due to people moving on). I do apologise for the time this has taken you and the reviewers. I am a big fan of what the journal is doing and I appreciate how much work goes into any review, let alone that of potentially unfamiliar code/software. Hopefully, at some point, we can gain funding to work specifically on the software in order to make it more accessible to a broader audience, rather than just providing access to the codebase. Maybe then we can try another submission to JOSS.

Thank you and all the best,

Tom

[danielskatz]

Given this, I think we should mark this submission as withdrawn.

@McGregor14 and @oliviaguest, do you agree with this plan?

[McGregor14]

@danielskatz yes I think that makes sense.

Thanks

[danielskatz]

Thanks to @oliviaguest for her work editing, and to @peircej, @KirstensGitHub, and @EleonoraVagnoni for their work in reviewing

[danielskatz]

@whedon withdraw

[whedon]

Paper withdrawn.

[peircej]

Sorry to see it came out that way (for now) but I think this is probably the right call. @McGregor14 the work that's been done on the package is great and clearly very substantial but I think these requirements and expectations are there to get packages over that next line to becoming sustainable community projects. It's really just documentation that's missing here. Hopefully you'll be able to add the additional pieces and resubmit later.

[oliviaguest]

Sorry for not replying sooner here. I somehow missed that I must have read this and not replied. Apologies. Indeed, agreed with all above.