Initial notebook testing framework

[jklaise]

Testing how we can use either nbmake or jupytext for executing notebooks under pytest.

Working on two new Github Actions workflows:

• test_all_notebooks - "batch" notebook execution, intended to run manually and at regular (e.g. weekly) intervals
• test_changed_notebooks - select notebook execution, intended to be triggered by any changes to existing notebooks or new notebooks.

Current version uses nbmake with default options (so 300 second timeout which we may want to increase for certain notebooks or re-design the notebooks to take less time). There is also an example script in testing/test_notebooks.py if we wanted to use jupytext to execute notebooks ourselves. This could be more flexible in the future as it would allow slightly more configuration as well as applying static checkers onto scripts generated by jupytext if we wished and/or papermill to allow passing parameters to notebooks (e.g. this may be more useful for alibi-detect to execute both True and False paths for the load_pretrained_detector variable).

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

Merging #480 (af82207) into master (90ff57e) will not change coverage.
The diff coverage is n/a.

❗ Current head af82207 differs from pull request most recent head d611589. Consider uploading reports for the commit d611589 to get more accurate results

@@           Coverage Diff           @@
##           master     #480   +/-   ##
=======================================
  Coverage   82.32%   82.32%           
=======================================
  Files          76       76           
  Lines       10316    10316           
=======================================
  Hits         8493     8493           
  Misses       1823     1823           

[ascillitoe]

@jklaise this looks like a nice setup in terms of the github actions and general approach to testing i.e. weekly batch vs testing when changed etc.

Is the plan going forward to leave this here as a PR until we've figured out how to speed execution up with jupytext/papermill?

Also, purely out of curiosity, what sort of checks are you thinking in terms of static checks on jupytext output?

[jklaise]

@ascillitoe my preference would be to merge this (for both libraries) and then start solving issues with the failing notebooks which may involve using other tools such as papermill. Without merging it's more difficult to check the whole notebook test suite as it would need to be run locally, whereas after merging we can trigger the workflow manually on CI.

Purely speculative, but one could run a reduced set of flake8 on the output of jupytext, when I did it for an example it flagged up a lot of irrelevant stuff (e.g. line length for Markdown cells), but also caught some relevant things (e.g. unused imports).

[ascillitoe]

@jklaise, got you, makes total sense on both points 👍🏻