Treat exceptions raised outside of notebook cells as errors

[hodgestar]

Is your feature request related to a problem? Please describe.
When executing a notebook cell, exceptions may be raised after the cell executes. This can happen, for example, when plotting a matplotlib figure using %matplotlib inline and an invalid label such as r"\\tau" (the double-slash is an incorrect escape). These exceptions raised after the cell completes are not picked up by nbmake (and also don't halt notebook execution in the Jupyter kernel).

Describe the solution you'd like
It would be nice to report these errors as failures. I don't know enough about how nbmake is implemented to know how difficult this would be.

Describe alternatives you've considered

• Detect and fail notebooks with these errors by default.
• Add a command-line option that toggles this behaviour on and off.
• Add a notebook metadata option that toggles this behaviour on and off.

Additional context
Thank you for maintaining nbmake -- it is an excellent tool.

[alex-treebeard]

Thanks for the bug report @hodgestar, and for using nbmake!

I'm going to take a look at this one in the next week.

e: If you are able to link me to an open source notebook then this can help me repro the issue.

[hodgestar]

@alex-treebeard Apologies for the delay. Below is an example of three cells that demonstrate the issue for me:

%matplotlib inline

import matplotlib.pyplot as plt

fig, ax = plt.subplots(1, 1)
ax.plot([0, 1, 2], [0, 1, 2])
ax.set_xlabel(r"$\\tau$");

print("Notebook execution continued despite the error!")

The final cell executes despite the error printed by the second cell, which for me ends wth:

File ~/miniconda3/envs/qutip/lib/python3.10/site-packages/matplotlib/_mathtext.py:2239, in Parser.parse(self, s, fonts_object, fontsize, dpi)
   2237     result = self._expression.parseString(s)
   2238 except ParseBaseException as err:
-> 2239     raise ValueError("\n".join(["",
   2240                                 err.line,
   2241                                 " " * (err.column - 1) + "^",
   2242                                 str(err)])) from err
   2243 self._state_stack = None
   2244 self._em_width_cache = {}

ValueError: 
\\tau
^
Unknown symbol: \, found '\'  (at char 0), (line:1, col:1)

<Figure size 432x288 with 1 Axes>

[alex-treebeard]

hi @hodgestar , this has been fixed in 1.3.4 -- please reopen if any issues

[hodgestar]

@alex-treebeard That is awesome. Thank you!