Use nbmake or testbook to automate notebook testing

[jonmaddock]

One of these two tools could make testing notebooks much more straightforward, avoiding the duplicate .py scripts. The tools need to be investigated and compared with Bluemira's strategy (which I can forward). One option should be implemented to ensure that our notebooks are well-tested in a sustainable manner.

[clmould]

There were 3 options available for automating notebook testing - nbmake, testbook and jupytext (used by Bluemira). After discussion with @timothy-nunn we decided that testbook would be the best option for PROCESS

• jupytext is used by Bluemira and uses python files instead of ipynb files and this has some advantages such as python files being easier to read as a diff, however we want to keep the example notebooks as ipynb files instead of converting them to python files (see below), so decided this approach would not be best for what we want in PROCESS
• nbmake doesn't seem to allow the writing of modular tests and so we did not want to use this
• testbook has an API that most aligns with our current tests, and so this is the tool that we have decided to use for testing the PROCESS example notebooks

We want to keep the examples as notebooks because, whilst running notebooks requires more packages to be installed on first running PROCESS and also debugging errors that occur in notebooks is harder, there are the advantages that running notebooks in vscode is easier than the python script version and also PROCESS docs don't render the examples yet so storing the notebooks with output is ideal

[alex-treebeard]

hi @clmould thanks for considering nbmake (I develop it).

How have you found testbook after choosing it? I'm quite open to feature requests if a pythonic interface could add something to the community here!

[timothy-nunn]

Hi @alex-treebeard,

Thank you for getting in touch, and for your contribution to the open source community. My main reason for choosing testbook over nbmake was the Pythonic interface. Running the notebook via a Python function required minimal changes to our workflow: our modelling team do not need to remember an --nbmake flag when running the test suite.

In general, nbmake seems more appropriate for a project more heavily reliant on notebooks/ use notebooks for documentation. In PROCESS, we do not utilise notebooks in that way yet. A more explicit Python API (it may even exist, just not be documented?) would have put nbmake on-par with testbook for us, as we would not be leveraging the full potential of either project--we just want to check our notebooks don't throw an error.

I hope this answered your question, and thank you again for your question and work on open source projects,
Timothy