Return code of executing notebooks in papermill

[vinaykumar80]

Hi All,

Is there a way to capture return code of executing notebooks in papermill API library or CLI methods? When any notebook cell execution encounters an error, it should return an error return code AND when all notebook cells execute successfully it should return a success return code.

Thanks!

[MSeal]

It does this today unless I'm missing something from the question. You get a zero exit code on success and non-zero on failure from the CLI call -- or an exception if you're in Python calling the library.

[willingc]

@vinaykumar80 The reference section of the documentation provides more information. Here's an example of an exception: https://papermill.readthedocs.io/en/latest/reference/papermill-workflow.html#papermill.execute.raise_for_execution_errors

[BoPeng]

I see the same problem here:

$ papermill test.ipynb test_error.ipynb
Executing:  14%|█                         | 10/73 [00:08<00:52,  1.19cell/s]
$ echo $?
0

It turns out that the kernel I am using (sos) returns execute_reply with status: 'error', but does not send an error message to iopub.

Because the raise_for_execution_errors function of papermill

papermill/papermill/execute.py

Line 154 in 0ee0da7

def raise_for_execution_errors(nb, output_path):

looks for output of type error, which can only be generated by a message of type error. If the kernel returns execute_reply without sending an error message, papermill would not be able to detect the error.

However, returning code 0 when papermill could not execute all the cells does look like a bug to me. If papermill actually executes the notebook, should not it respond to an execute_reply message with status error? Even if execute_reply is not accessible, not reaching 100% looks like a legitimate reason to return a non-zero code.

[BoPeng]

I would further argue that the existence of an error message does not mean the cell has failed. For example, the SoS super kernel has an %env --expect-error magic that executes the cell (in SoS or a subkernel such as R) and returns an ok status when the subkernel returns an execute_reply message with status error. So where as this notebook can be reproduced in full successfully through "Run all cells", papermill will stop at the error cell because of the error message.

Basically I believe that only an execute_reply with an error status is a proper indication of the failure of the cell.

@MSeal What do you think?

[MSeal]

Sorry for the late reply @BoPeng , catching back up from the holidays.

Unfortunately this is a shortcoming in the API spec for jupyter. The execute_reply and the error message patterns are intermixed by a lot of kernels as it's confusing what's used for what. The result is that different interfaces have made changes to work for specific kernel patterns which isn't necessarily in the spec.

From the api spec docs: https://jupyter-client.readthedocs.io/en/stable/messaging.html#messages-on-the-shell-router-dealer-channel interfaces should behave the way your describing, relying on execute_reply['content']['status'] only while cell error outputs are for display / exception gathering purposes.

Upon completion of the execution request, the kernel always sends a reply, with a status code indicating what happened and additional data depending on the outcome.

The actual code we need to touch is in nbconvert https://github.com/jupyter/nbconvert/blob/master/nbconvert/preprocessors/execute.py#L475-L479, captured by

papermill/papermill/preprocess.py

Line 64 in 0ee0da7

except CellExecutionError as ex:

in papermill. This means the change is more of an nbconvert issue (soon to be nbexecute). We should probably re-author the issue there and get a change in for the upcoming 6.0 release. The code handling this predates my involvement with nbconvert some, so I'd like to get some earlier folks' comments, though the code as-is reads as making an assumption that execute_reply will follow an error message always as the error pattern, even though the spec in jupyter_client doesn't say that's required. This is probably a result of implementing nbconvert for use with the ipython kernel originally, which happens to behave this way. Would you mind making an issue there, referencing this one? I can help get the change made and merged so long as there isn't some expected behavior I'm not aware of that also needs tackling.

Also this change in behavior is pending for the next papermill release: #449 -- it adds checks for the type of exit so that user exit(0) can be respected as a successful exit. This might complicate the proposed check here, though I don't think they're incompatible.

[BoPeng]

Since jupyter/nbconvert#1163 has been merged, can we re-open this ticket and see what can be done on the papermill side?

[MSeal]

I believe there is noting to do in papermill. It just needs the dependency update and it should follow nbconvert's behavior. This will happen once I finish jupyter/nbconvert#821 (which I am actively working on this week) and papermill 2.0 releases.

[rijobro]

If it helps, you can grep the output for "status": "failed" (obviously you'll run into trouble if your notebook contains that anyway):

out=$(echo "$notebook" | papermill --progress-bar)
success=$?
if [[ ${success} -ne 0 || "$out" =~ "\"status\": \"failed\"" ]]; then
    print_style_fail_msg
    exit ${success}
fi

[davidxia]

@BoPeng is there a way to catch any error in any cell?

When does papermill execution of a notebook return output of type error? I have

import papermill as pm

nb_node = pm.execute_notebook(
   'input.ipynb',
   'output.ipynb',
)
pm.execute.raise_for_execution_errors(nb_node, "output.ipynb")

input.ipynb has one cell that's just 9/0.

The output.output_type is display_data. It's the same when I just raise Exception("test") in input.ipynb.

output
{'data': {'text/html': '<pre '
                       ...ZeroDivisionError: '
                        ...
                        '\x1b[31m│\x1b[0m \x1b[3;31m[Errno 2] No such file or '
                        'directory: '
                        '\x1b[0m                                                     '
                        '\x1b[31m│\x1b[0m\n'
                        '\x1b[31m│\x1b[0m '
                        "\x1b[3;31m'/var/folders/sr/ryl6yttx739282tn918mf3l00000gn/T/ipykernel_6465/303056122.py'\x1b[0m            "
                        '\x1b[31m│\x1b[0m\n'
                        '\x1b[31m╰───────────────────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n'
                        '\x1b[1;91mZeroDivisionError: \x1b[0mdivision by '
                        'zero\n'},
 'metadata': {},
 'output_type': 'display_data'}

[BoPeng]

This is kernel dependent. Obviously this kernel decides to output an error message in HTML format instead of sending an actual error message.

[davidxia]

I see. What kernels are compatible with this method?

[davidxia]

Hi, sorry, bumping to see if anyone knows the answer to my last question. 🙏

[BoPeng]

I see. What kernels are compatible with this method?

ipykernel (Python) should work.

[davidxia]

I'm using ipykernel (see .metadata.kernelspec below). I wonder why it's not sending an error message.

cat output.ipynb
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e8c439cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T20:52:41.779622Z",
     "iopub.status.busy": "2022-12-04T20:52:41.779398Z",
     "iopub.status.idle": "2022-12-04T20:52:42.015130Z",
     "shell.execute_reply": "2022-12-04T20:52:42.014679Z"
    },
    "papermill": {
     "duration": 0.23956,
     "end_time": "2022-12-04T20:52:42.016056",
     "exception": true,
     "start_time": "2022-12-04T20:52:41.776496",
     "status": "failed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800000; text-decoration-color: #800000\">╭──────────────────────────── </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Traceback </span><span style=\"color: #bf7f7f; text-decoration-color: #bf7f7f; font-weight: bold\">(most recent call last)</span><span style=\"color: #800000; text-decoration-color: #800000\"> ────────────────────────────╮</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/var/folders/sr/ryl6yttx739282tn918mf3l00000gn/T/ipykernel_39928/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">303056122.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">&lt;cell </span> <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">line: 1&gt;</span>                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000; font-style: italic\">[Errno 2] No such file or directory: </span>                                                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000; font-style: italic\">'/var/folders/sr/ryl6yttx739282tn918mf3l00000gn/T/ipykernel_39928/303056122.py'</span>           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">╰───────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">ZeroDivisionError: </span>division by zero\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[31m╭─\u001b[0m\u001b[31m─────────────────────────── \u001b[0m\u001b[1;31mTraceback \u001b[0m\u001b[1;2;31m(most recent call last)\u001b[0m\u001b[31m ───────────────────────────\u001b[0m\u001b[31m─╮\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/var/folders/sr/ryl6yttx739282tn918mf3l00000gn/T/ipykernel_39928/\u001b[0m\u001b[1;33m303056122.py\u001b[0m:\u001b[94m1\u001b[0m in \u001b[92m<cell \u001b[0m \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[92mline: 1>\u001b[0m                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[3;31m[Errno 2] No such file or directory: \u001b[0m                                                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[3;31m'/var/folders/sr/ryl6yttx739282tn918mf3l00000gn/T/ipykernel_39928/303056122.py'\u001b[0m           \u001b[31m│\u001b[0m\n",
       "\u001b[31m╰───────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
       "\u001b[1;91mZeroDivisionError: \u001b[0mdivision by zero\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "9/0"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('hray')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 1.644183,
   "end_time": "2022-12-04T20:52:42.133595",
   "environment_variables": {},
   "exception": true,
   "input_path": "input.ipynb",
   "output_path": "output.ipynb",
   "parameters": {},
   "start_time": "2022-12-04T20:52:40.489412",
   "version": "2.4.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "c5c9ad77933d0544b86fa7b931ab2bf7b54ec43e82f16ada22862c0e19c5f883"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}