Provide a test reporting, or document how the project is tested

[kinow]

Hi,

It looks like BrowserStack is used to test the mobile app. Is there any dashboard or report that is available to users? They may be interested to know if there are any failures for a build/version, or if there is any information on what was tested.

Do you have test coverage anywhere?

What about the backend app? I think that's PHP, from what I understand so far (waiting for #18 to build the environment and test locally). Are there any tests for the backend?

From: openjournals/joss-reviews#2506

Cheers
Bruno

[V69dxz]

Hi Bruno,

Yes, I use BrowserStack. When you send me your email address, I will grant you access to the test environment, and send you the login credentials for both the developer iOS and Android version. Ok?

Meanwhile, I think you also suggest to set up an automatic testing procedure that generates reports automatically? Until now, we (: me, my PhD students who use the app in their research, and end users) have tested the app only manually (: e.g., whether the measurement schedule is rendered properly, whether the data is correctly stored on the server, whether push notifications are received). I think I need some time to set up automatic testing procedures as this is very new to me :).

With regard to the backend app. It is not so fancy yet. I set up phpmyadmin panel for the researchers (which is shield by an additional password and can only be accessed through an eduroam vpn connection [used by higher education institutes]). I don't think I need any tests here, but if you have suggestions how to achieve this, I would be happy to hear (and implement).

Best,
Peter

[kinow]

Hi Peter,

I think it would be useful if developers/users could run the tests. Are you using Selenium, Nightwatch, Cypress, or any other test system for testing with the browser at BrowserStack? Or is it fully manual?

The backend doesn't need a lot of tests, but I think it would be important to offer a minimum set of tests. The project homepage has a card/panel about security, for example. There could be a test to verify that a password is salted, or an error message is given when the user doesn't provide user and password, etc.

Or even simpler unit tests.

This is related to this item from my checklist, under Documentation:

Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

I think we can leave it unchecked if there are no tests for now. Not sure if that's an issue or not, I think after I finish the review the editor will take a look at the checklist and comment.

Thanks
Bruno

[V69dxz]

Hi Bruno,

The app = It is fully manual now, but will set up such an automatic testing procedure. Let you know when I have it running.

The backend = I will have another look at this, will let you know when I have implemented some tests here too :)

Best,
Peter

[kinow]

Thanks Peter!

[V69dxz]

Hi Bruno,

Automatic backend testing
Today I have written a simple php script to perform some basic checks to test whether a survey is correctly programmed by the researcher (following the required settings, see also the new Wiki page).

The script definitely needs some improvements (e.g. checking other elements of the backend + rendering in html). I will work on this after my vacation. Your feedback is very much appreciated.

(A better approach would be to write a small program that helps researchers to set up a survey correctly, but that is something for next year 😄.)

Automatic app testing
Will have a closer look at this after my vacation. I will see how far I get.

Manual steps for testing
Regardless of the degree to which I succeed in implementing some automatic tests, I will write down some manual steps for testing in line with the JOSS guideliness. Will start working on this when you succeed in installing the backend + app (hopefully after my vacation too 😄).

Thanks!

Peter

[kinow]

Hi Peter,

If you have a PHP script, maybe that could be converted to a PHPUnit test?

Either way, if the container can be set up so that running the script works inside or outside the container, it would be easy to automate that with GitHub actions or Travis CI, to run with every commit, preventing regressions.

Enjoy your vacation! We can discuss more about that later 🍹

Thank you!
Bruno

[kinow]

@pmkruyen build done. I've played with DearScholar, and it reminds me of an application we have at NIWA for Citizen Science. In our application, researchers are able to create custom forms, like in SurveyMonkey or Lime Survey, but with more functionalities (e.g. some GIS information to use maps, questions referencing questions, etc).

The functionality appears to be OK.

For my JOSS review, I think the only part missing now is this issue. Once we have tests, I can announce in the review thread that my review is done with no issues :-)

[V69dxz]

Hi Bruno,

it reminds me of an application we have at NIWA for Citizen Science.

Is it somewhere in the public domain? Is there a link? I will refer to it in the paper.

Either way, if the container can be set up so that running the script works inside or outside the container, it would be easy to automate that with GitHub actions or Travis CI, to run with every commit, preventing regressions.

Good idea. Will start working on this after my vacation (including the phpunit test idea).

Enjoy your vacation! We can discuss more about that later 🍹

🍹🍹🍹

Thank you

[kinow]

Hi @pmkruyen

Hi Bruno,

it reminds me of an application we have at NIWA for Citizen Science.

Is it somewhere in the public domain? Is there a link? I will refer to it in the paper.

The app is in the Android & iOS application stores. The best link to use in the references would be https://citizenscience.niwa.co.nz/. The source code is not open at the moment.

Either way, if the container can be set up so that running the script works inside or outside the container, it would be easy to automate that with GitHub actions or Travis CI, to run with every commit, preventing regressions.

Good idea. Will start working on this after my vacation (including the phpunit test idea).

Great! That should be easy to review.

Enjoy your vacation! We can discuss more about that later 🍹

🍹🍹🍹

Thank you

Thanks!
Bruno

[V69dxz]

Hi Bruno,

After a couple of frustrating evenings (I could not let it go during my vacation), I managed to get the basics of phpunit (and object oriented programming) to test changes in dearscholar.php. I have to admit, this new testing (and programming) knowledge is really helpful.

You can find the files needed for testing in the phpunit folder. Actually, I decided to write 22 integration tests instead of unit tests (as I learned the vocabulary when working on it) because I had difficulties with mocking mysql databases when I started with this "sub-project".

To run the tests on your machine:

1. start the docker (+ populate the database + move dearscholar.php in the app folder created by the docker.
   2. in phpmyadmin change in the PINtable the code column for user test3 to REVOKE.
2. navigate to the phpunit folder, run php composer.phar update and ./vendor/bin/phpunit ... and the tests should be executed.

Hope it works 😃 . All feedback is very welcome!

What's next?
What I would like to do now, is to automate these php tests on uploading new commits to GitHub. As you can see on the README page, I have started to use Travis CI, which now only tests whether the app is successfully build in Android and iOS.

From your comments at #18, I understood that you compiled a new docker that includes the dearscholar.php file and a prepopulated database. Is this correct? Could you please share with me the code you used to build this docker? I need to tweak the Docker a bit to update the dearscholar.php file (I changed some parts of the code based on the testing sessions <-- as I said really helpful new knowledge :) ) and change in the PINtable the code column for user test3 to REVOKE. Sorry that I have not found the time to explore your docker image yet during my vacation! In case you did not build the docker, I am sorry that I misunderstood you and try to build one myself.

UPDATE: The docker image works. Thanks a lot. Interesting to see your code. Added in the PINtable the code column for user test3 to REVOKE and the PhPUnit test works (however I also see now an issue with actually changing the database in a test, on the second attempt the test for the authentication` table fails. Need to adjust the code a bit here :) (and update the installation instruction too).

And further

• When I have managed to automate the php unit tests, I will start with writing a script to test the javascript files and try to automate the app testing in browserstack (and then I'm done :)).
• With regard to the later, Travis CI is (still) unable to push the app to BrowserStack automatically due to issues with the encrypted key, the BrowserStack people are looking at it.

Best,
Peter

PS. Thanks for the reference to your app, looks cool! Will delve further in it when I start working on the last version of the paper.

[V69dxz]

Hi Bruno,

From your comments at #18, I understood that you compiled a new docker that includes the dearscholar.php file and a prepopulated database. Is this correct? Could you please share with me the code you used to build this docker? I need to tweak the Docker a bit to update the dearscholar.php file (I changed some parts of the code based on the testing sessions <-- as I said really helpful new knowledge :) ) and change in the PINtable the code column for user test3 to REVOKE. Sorry that I have not found the time to explore your docker image yet during my vacation! In case you did not build the docker, I am sorry that I misunderstood you and try to build one myself.

I found and added your docker composer. Thanks! Will start with my attempt to automate the unitphp testing in travis ci (and also update the installation instructions).

Thanks again, I learn a lot from your help!

Peter

[kinow]

Hi @pmkruyen !

Welcome back!

I struggle to disconnect from projects when I have interesting problems. I hope it didn't distract you too much from your vacation, but that at the same time you had some fun with phpunit and Docker :-)

Tests working for me already! I found one minor issue with the Dockerfile, as it is using installation instead of Installation when referencing the local directory. I'm sending a PR for that later.

But once I fixed that, started the container, and then ran the tests, everything worked with no issues! 🚀

kinow@ranma:~/Development/javascript/workspace/dearscholar/testing/phpunit$ ./vendor/phpunit/phpunit/phpunit 
PHPUnit 9.3.7 by Sebastian Bergmann and contributors.

......................                                            22 / 22 (100%)

Time: 00:01.610, Memory: 6.00 MB

OK (22 tests, 31 assertions)

Looks like you will be a bit more busy with tests, but for the JOSS paper review, we already have what we needed IMO. I'll update the review ticket now, and keep lurking here.

I learned a lot working with you too @pmkruyen ! And hope to keep having fun working with you in the future, either with this or other JOSS related projects, or in different projects 🤓

Cheers
Bruno

[V69dxz]

Hi Bruno,

Haha, no worries!

Thanks for all help. Super cool. And yes, I hope we stay connected!

Obviously, I keep you updated... certainly let you know when I managed to put everything in Travis.

Peter

[kinow]

Or GitHub actions if you prefer. So far the only difference I have found between Travis and GitHub is that GitHub doesn't support some platforms like ARM. But if you only need a place to run Docker, PHP, JS, etc... either should work fine 👍

[V69dxz]

Hi Bruno,

I updated the Wiki page (installation and testing). As you can see, I managed to do automatic testing of the php code in Travis (thanks to your Docker). Now, moving on to automatic testing of the javascript functions :)

Cheers,
Peter

[kinow]

Hi Peter,

You even included the badge already. Great job! Another useful resource is code coverage, showing how much of the code was tested. You should be able to use phpunit to get the value (may need another package installed).

But probably an enhancement for later :-)

Cheers
Bruno

[V69dxz]

Hi Bruno,

Sorry for the delayed response, I'm back again. Updated Cordova to the latest release (there is no need to have a seperate Android26 app anymore, the only issue I have now is a a black status bar on Android, haven't figured out how to fix this yet, but anyway).

Yes to have code coverage is a good idea, will implement it after I have implemented the javascript testing.

Best,
Peter

[kinow]

Hi Peter

Updated Cordova to the latest release (there is no need to have a seperate Android26 app anymore, the only issue I have now is a a black status bar on Android, haven't figured out how to fix this yet, but anyway).

Great! Now the building of DearScholar is even simpler!

Yes to have code coverage is a good idea, will implement it after I have implemented the javascript testing.

👍 I've just finished adding coverage for a Python project with python-coverage + pytest and Codacy to store the reports in another projects. Let me know if you need any help with testing, review, etc.