guides/cron/README.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "id": "ca934856",
            "metadata": {
                "papermill": {
                    "duration": 0.049223,
                    "end_time": "2022-03-12T19:09:02.782991",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:02.733768",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "To run this locally, [install Ploomber](https://docs.ploomber.io/en/latest/get-started/quick-start.html) and execute: `ploomber examples -n guides/cron`\n",
                "\n",
                "[![binder-logo](https://raw.githubusercontent.com/ploomber/projects/master/_static/open-in-jupyterlab.svg)](https://binder.ploomber.io/v2/gh/ploomber/binder-env/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252Fploomber%252Fprojects%26urlpath%3Dlab%252Ftree%252Fprojects%252Fguides/cron%252FREADME.ipynb%26branch%3Dmaster)\n",
                "\n",
                "Found an issue? [Let us know.](https://github.com/ploomber/projects/issues/new?title=guides/cron%20issue)\n",
                "\n",
                "Questions? [Ask us on Slack.](https://ploomber.io/community/)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "3e13e3b1",
            "metadata": {
                "papermill": {
                    "duration": 0.041039,
                    "end_time": "2022-03-12T19:09:02.862739",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:02.821700",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "# Scheduling with cron\n",
                "\n",
                "This guide shows how to schedule Ploomber pipelines using `cron`.\n",
                "\n",
                "**Note:** `cron` is only available on macOS and Linux.\n",
                "\n",
                "## Pre-requisites\n",
                "\n",
                "Ensure cron is installed:"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "bc2f1cab",
            "metadata": {
                "papermill": {
                    "duration": 0.037729,
                    "end_time": "2022-03-12T19:09:02.940515",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:02.902786",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```sh\n",
                "crontab -l\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "a6cac96d",
            "metadata": {
                "papermill": {
                    "duration": 0.045923,
                    "end_time": "2022-03-12T19:09:03.026574",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:02.980651",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "If you don't see a \"command not found error\", you have `cron` installed and can continue.\n",
                "\n",
                "## Get the example"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "12bed96a",
            "metadata": {
                "papermill": {
                    "duration": 0.039559,
                    "end_time": "2022-03-12T19:09:03.106888",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.067329",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```sh\n",
                "pip install ploomber\n",
                "ploomber examples -n guides/cron -o cron\n",
                "cd cron\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "567069ce",
            "metadata": {
                "papermill": {
                    "duration": 0.044075,
                    "end_time": "2022-03-12T19:09:03.191974",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.147899",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "## Setup\n",
                "\n",
                "Configure virtual environment:"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "4d2faa97",
            "metadata": {
                "papermill": {
                    "duration": 0.038337,
                    "end_time": "2022-03-12T19:09:03.270925",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.232588",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```sh\n",
                "ploomber install --create-env --use-venv\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "ece78ab8",
            "metadata": {
                "papermill": {
                    "duration": 0.04095,
                    "end_time": "2022-03-12T19:09:03.352933",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.311983",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Activate environment:"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "a6f77aea",
            "metadata": {
                "papermill": {
                    "duration": 0.038889,
                    "end_time": "2022-03-12T19:09:03.429870",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.390981",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```sh\n",
                "source venv-cron/bin/activate\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "56ebdfe3",
            "metadata": {
                "papermill": {
                    "duration": 0.043446,
                    "end_time": "2022-03-12T19:09:03.512014",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.468568",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "## The code\n",
                "\n",
                "The pipeline has two stages: load, and plot:"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "b8489e05",
            "metadata": {
                "lines_to_next_cell": 2,
                "papermill": {
                    "duration": 0.040839,
                    "end_time": "2022-03-12T19:09:03.592240",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.551401",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```yaml\n",
                "# Content of pipeline.yaml\n",
                "tasks:\n",
                "  - source: scripts/load.py\n",
                "    product:\n",
                "      nb: products/{{timestamp}}/load.html\n",
                "      data: products/{{timestamp}}/load.csv\n",
                "\n",
                "  - source: scripts/plot.py\n",
                "    product:\n",
                "      nb: products/{{timestamp}}/plot.html\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "3e5a1224",
            "metadata": {
                "papermill": {
                    "duration": 0.040496,
                    "end_time": "2022-03-12T19:09:03.671763",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.631267",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Notice that all the products are parametrized prefixed by: `products/{{timestamp}}`; this will allow us to store the products of each run depending on the runtime timestamp.\n",
                "\n",
                "## Development\n",
                "\n",
                "When modifying the pipeline, you can call the following command:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "id": "b2460656",
            "metadata": {
                "execution": {
                    "iopub.execute_input": "2022-03-12T19:09:03.760960Z",
                    "iopub.status.busy": "2022-03-12T19:09:03.760251Z",
                    "iopub.status.idle": "2022-03-12T19:09:13.642901Z",
                    "shell.execute_reply": "2022-03-12T19:09:13.643445Z"
                },
                "papermill": {
                    "duration": 9.933065,
                    "end_time": "2022-03-12T19:09:13.643821",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:03.710756",
                    "status": "completed"
                },
                "tags": []
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Loading pipeline...\n",
                        "name    Ran?      Elapsed (s)    Percentage\n",
                        "------  ------  -------------  ------------\n",
                        "load    True          1.90359       39.4266\n",
                        "plot    True          2.9246        60.5734\n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "/Users/Edu/miniconda3/envs/projects/lib/python3.9/site-packages/papermill/iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.\n",
                        "  from pyarrow import HadoopFileSystem\n",
                        "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:62: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
                        "  warnings.warn(res)\n",
                        "\r",
                        "  0%|          | 0/2 [00:00<?, ?it/s]\r",
                        "Building task 'load':   0%|          | 0/2 [00:00<?, ?it/s]\n",
                        "\r",
                        "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
                        "\r",
                        "Executing:  25%|██▌       | 1/4 [00:01<00:03,  1.30s/cell]\u001b[A\r",
                        "Executing: 100%|██████████| 4/4 [00:01<00:00,  2.67cell/s]\n",
                        "\r",
                        "Building task 'load':  50%|█████     | 1/2 [00:01<00:01,  1.91s/it]\r",
                        "Building task 'plot':  50%|█████     | 1/2 [00:01<00:01,  1.91s/it]\n",
                        "\r",
                        "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
                        "\r",
                        "Executing:  25%|██▌       | 1/4 [00:02<00:06,  2.10s/cell]\u001b[A\n",
                        "\r",
                        "Executing: 100%|██████████| 4/4 [00:02<00:00,  2.11cell/s]\u001b[A\r",
                        "Executing: 100%|██████████| 4/4 [00:02<00:00,  1.54cell/s]\n",
                        "\r",
                        "Building task 'plot': 100%|██████████| 2/2 [00:04<00:00,  2.51s/it]\r",
                        "Building task 'plot': 100%|██████████| 2/2 [00:04<00:00,  2.42s/it]\n",
                        "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:98: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
                        "  warnings.warn(res)\n"
                    ]
                }
            ],
            "source": [
                "%%sh\n",
                "ploomber build"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "6bf0a6f1",
            "metadata": {
                "papermill": {
                    "duration": 0.045847,
                    "end_time": "2022-03-12T19:09:13.735306",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:13.689459",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Note that products will go to `output/dev/`.\n",
                "\n",
                "For scheduling the workflow, we need to tell Ploomber to use a different configuration file:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "id": "07580831",
            "metadata": {
                "execution": {
                    "iopub.execute_input": "2022-03-12T19:09:13.826630Z",
                    "iopub.status.busy": "2022-03-12T19:09:13.825907Z",
                    "iopub.status.idle": "2022-03-12T19:09:23.486023Z",
                    "shell.execute_reply": "2022-03-12T19:09:23.486932Z"
                },
                "papermill": {
                    "duration": 9.709503,
                    "end_time": "2022-03-12T19:09:23.487151",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:13.777648",
                    "status": "completed"
                },
                "tags": []
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Loading pipeline...\n",
                        "name    Ran?      Elapsed (s)    Percentage\n",
                        "------  ------  -------------  ------------\n",
                        "load    True          1.99751       40.9529\n",
                        "plot    True          2.88008       59.0471\n"
                    ]
                },
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "/Users/Edu/miniconda3/envs/projects/lib/python3.9/site-packages/papermill/iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.\n",
                        "  from pyarrow import HadoopFileSystem\n",
                        "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:62: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
                        "  warnings.warn(res)\n",
                        "\r",
                        "  0%|          | 0/2 [00:00<?, ?it/s]\r",
                        "Building task 'load':   0%|          | 0/2 [00:00<?, ?it/s]\n",
                        "\r",
                        "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
                        "\r",
                        "Executing:  25%|██▌       | 1/4 [00:01<00:04,  1.41s/cell]\u001b[A\r",
                        "Executing: 100%|██████████| 4/4 [00:01<00:00,  2.48cell/s]\n",
                        "\r",
                        "Building task 'load':  50%|█████     | 1/2 [00:02<00:02,  2.00s/it]\r",
                        "Building task 'plot':  50%|█████     | 1/2 [00:02<00:02,  2.00s/it]\n",
                        "\r",
                        "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
                        "\r",
                        "Executing:  25%|██▌       | 1/4 [00:01<00:05,  1.80s/cell]\u001b[A\n",
                        "\r",
                        "Executing: 100%|██████████| 4/4 [00:02<00:00,  2.21cell/s]\u001b[A\r",
                        "Executing: 100%|██████████| 4/4 [00:02<00:00,  1.58cell/s]\n",
                        "\r",
                        "Building task 'plot': 100%|██████████| 2/2 [00:04<00:00,  2.52s/it]\r",
                        "Building task 'plot': 100%|██████████| 2/2 [00:04<00:00,  2.44s/it]\n",
                        "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:98: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
                        "  warnings.warn(res)\n"
                    ]
                }
            ],
            "source": [
                "%%sh\n",
                "# tell ploomber to use env.cron.yaml as config file\n",
                "export PLOOMBER_ENV_FILENAME=env.cron.yaml\n",
                "# build pipeline\n",
                "ploomber build\n",
                "# delete env variable\n",
                "unset PLOOMBER_ENV_FILENAME"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "02554a25",
            "metadata": {
                "papermill": {
                    "duration": 0.043817,
                    "end_time": "2022-03-12T19:09:23.577101",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:23.533284",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Let's see the contents of the products directory:"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "id": "e4da0559",
            "metadata": {
                "execution": {
                    "iopub.execute_input": "2022-03-12T19:09:23.668551Z",
                    "iopub.status.busy": "2022-03-12T19:09:23.667747Z",
                    "iopub.status.idle": "2022-03-12T19:09:23.730703Z",
                    "shell.execute_reply": "2022-03-12T19:09:23.731378Z"
                },
                "papermill": {
                    "duration": 0.111568,
                    "end_time": "2022-03-12T19:09:23.731579",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:23.620011",
                    "status": "completed"
                },
                "tags": []
            },
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "2022-03-12T13:09:16.783678\n",
                        "dev\n"
                    ]
                }
            ],
            "source": [
                "%%sh\n",
                "ls products"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "5e440e91",
            "metadata": {
                "papermill": {
                    "duration": 0.043804,
                    "end_time": "2022-03-12T19:09:23.822190",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:23.778386",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "You should see two folders, `dev/` and another one with the runtime timestamp."
            ]
        },
        {
            "cell_type": "markdown",
            "id": "53df9d09",
            "metadata": {
                "papermill": {
                    "duration": 0.0453,
                    "end_time": "2022-03-12T19:09:23.911172",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:23.865872",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "## Scheduling\n",
                "\n",
                "Now let's schedule in cron. First, to edit the cron configuration file:"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "f34bb154",
            "metadata": {
                "papermill": {
                    "duration": 0.043471,
                    "end_time": "2022-03-12T19:09:23.999080",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:23.955609",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```sh\n",
                "crontab -e\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "778e13fa",
            "metadata": {
                "papermill": {
                    "duration": 0.044044,
                    "end_time": "2022-03-12T19:09:24.091740",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:24.047696",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Note that this will open the configuration file in the default editor,\n",
                "if you don't know what that is, you can open it with `nano`:"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "774ff38f",
            "metadata": {
                "papermill": {
                    "duration": 0.045209,
                    "end_time": "2022-03-12T19:09:24.179515",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:24.134306",
                    "status": "completed"
                },
                "region_name": "md",
                "tags": []
            },
            "source": [
                "```sh\n",
                "EDITOR=nano crontab -e\n",
                "```"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "c7d8add9",
            "metadata": {
                "papermill": {
                    "duration": 0.04465,
                    "end_time": "2022-03-12T19:09:24.269856",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:24.225206",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Once the editor opens, add a line like this:\n",
                "\n",
                "```txt\n",
                "* * * * *  PROJ=/path/to/cron && cd $PROJ && bash run.sh >> cron.log 2>&1\n",
                "```\n",
                "\n",
                "**Note:** If using macOS Big Sur (11.6) or newer, you may need to follow a few [extra steps](https://osxdaily.com/2020/04/27/fix-cron-permissions-macos-full-disk-access/) to enable cron."
            ]
        },
        {
            "cell_type": "markdown",
            "id": "0b1878ae",
            "metadata": {
                "papermill": {
                    "duration": 0.043993,
                    "end_time": "2022-03-12T19:09:24.362439",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:24.318446",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "Replace the `/path/to/cron/` with the absolute path to the `cron/` directory that contains the sample code. (Tip: so get the absolute path, enter `pwd` in your terminal).\n",
                "\n",
                "If you opened the configuration file with `nano`, save your changes with `CTRL + O` and exit the editor with `CTRL + X`.\n",
                "\n",
                "After a minute, you'll start to see more directories in the products folder; this is what mine looks like:\n",
                "\n",
                "```\n",
                "2022-03-12T11:14:47.506532/ \n",
                "2022-03-12T11:25:12.707618/ \n",
                "dev/\n",
                "```\n",
                "\n",
                "If you see something like this, congratulations, you have a scheduled pipeline up and running!\n",
                "\n",
                "To learn how to modify the scheduling interval, see the Overview section in cron's [Wikipedia article.](https://en.wikipedia.org/wiki/Cron)"
            ]
        },
        {
            "cell_type": "markdown",
            "id": "d665ad1f",
            "metadata": {
                "papermill": {
                    "duration": 0.04505,
                    "end_time": "2022-03-12T19:09:24.450267",
                    "exception": false,
                    "start_time": "2022-03-12T19:09:24.405217",
                    "status": "completed"
                },
                "tags": []
            },
            "source": [
                "## Troubleshooting\n",
                "\n",
                "If you don't see the new directories, check out the `cron.log` file, which will contain any error messages, and ping us [on Slack](https://ploomber.io/community) so we can help you."
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3 (ipykernel)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.9.7"
        },
        "papermill": {
            "default_parameters": {},
            "duration": 23.534977,
            "end_time": "2022-03-12T19:09:24.709663",
            "environment_variables": {},
            "exception": null,
            "input_path": "guides/cron/_build/readme_preprocessed.ipynb",
            "output_path": "guides/cron/README.ipynb",
            "parameters": {},
            "start_time": "2022-03-12T19:09:01.174686",
            "version": "2.3.3"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 5
}