3_how_to_execute_the_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to execute the PyPSA-Africa workflow?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook aims to make you confident about understanding, executing and setting up PyPSA-Africa workflows.\n",
    " \n",
    "This tutorial will cover:\n",
    "- Snakemake\n",
    "- Snakefiles\n",
    "- How to create Directed Acyclic Graphs or `--dag`'s\n",
    "- Different ways to execute workflows\n",
    "- Wildcards\n",
    "\n",
    "Requirements:\n",
    "- Installed PyPSA-Africa (i.e. in `~/home/pypsa-africa`)\n",
    "- Installed anaconda environment `(pypsa-africa)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to install **ipython** while being in the `pypsa-africa` environment to choose the environment in VS code/Jupyter lab. Install the ipython kernel like this:\n",
    "\n",
    "```\n",
    "(pypsa-africa).../pypsa-africa % ipython kernel install --user --name=pypsa-africa\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we need to import a package to display images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Snakemake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Snakemake](https://snakemake.readthedocs.io/en/stable/) is our workflow management tool which allows you to define and excute workflow in a flexible manner. It safes you plenty of time! As a first inspiration:\n",
    "1. Open your VScode terminal in the `pypsa-africa` folder\n",
    "2. Copy and rename the `config.tutorial.yaml` > `config.yaml`\n",
    "3. Open the `config.yaml` and replace the `cluster: [6]` to `clusters: [5,10,20,30]`\n",
    "4. Activate your pypsa-africa environment: `conda activate pypsa-africa`\n",
    "5. Type the following in the terminal: \n",
    "\n",
    "```\n",
    "snakemake --dag solve_all_networks | dot -Tpng -o workflow.png\n",
    "```\n",
    "\n",
    "You should see now a image \"Directed Acyclic Graph\" or short \"dag\" in your folder which is saved as **workflow.png** and describes the workflow that you would execute. Just image you would need to execute script by script by hand. Not impossible for the workflow below, but just imagine you have 100 or 1000 optimisation. Snakemake saves your valuable time in such situations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display.Image(\"./images/hackathon/workflow_example.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Snakefile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the workflows are described in the `Snakefile` which you can find in the `pypsa-africa` repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display.Image(\"./images/hackathon/snakefile_location.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we open the Snakefile, we can see typically:\n",
    "- **rules**, gives a workflow component a name\n",
    "- **scripts**, describes what function is used from `pypsa-africa/scripts`\n",
    "- **inputs**, informs what data goes into the function\n",
    "- **outputs**, informs what data goes out of the function\n",
    "\n",
    "The other features like `threads` and `resources` can be used to allow parallel thread computing/ computing management. Further, the `log` is used to document the executed function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of the `rule build_shapes:`, the workflow inputs `eez` data which is [georeferenced marine data](https://www.marineregions.org/downloads.php) that we need in the `scripts/build_shapes.py` to allocate areas of the sea to specific countries. As output the rule creates:\n",
    "- country shapes\n",
    "- offshore shapes\n",
    "- general africa shape\n",
    "- regional to subregional zones or [gadm_shapes](https://gadm.org/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display.Image(\"./images/hackathon/build_shapes_rule.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ways to execute workflows\n",
    "There are three different ways to execute workflows. Compare the below examples with the above build_shapes rule:\n",
    "1. execute on the output `snakemake --cores 1 resources/country_shapes.geojson --dryrun`\n",
    "2. execute on the rule `snakemake -j 1 build_shapes -n` (Note, -j equals --cores, and -n equals --dryrun)\n",
    "3. execute on solve_all_networks (runs all rules -> defined in `Snakefile`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting an execution, it is always wise to start with a dry-run. This will not create any files or execute any functions, but it helps understanding what will happen if you do so. To execute a rule simply leave out the `-n` or `--dryrun`.\n",
    "\n",
    "Let's see what the `--dag` says before after the execution:\n",
    "\n",
    "```\n",
    "snakemake --dag build_shapes | dot -Tpng -o build_shapes_dag.png\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before Snakemake was executed\n",
    "display.Image(\"./images/hackathon/build_shapes_dag_prerun.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After Snakemake was executed\n",
    "display.Image(\"./images/hackathon/build_shapes_dag_postrun.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems alright. The build_shapes rule only requires inputs form the retrieve_data_bundle rule. Once the rule is executed it becomes dashes, revealing that there is already data available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are the {} in the outputs?\n",
    "\n",
    "Some rules such the `rule cluster_network` have this curly brackets in the inputs and outputs. These are called **wildcards**. One example:\n",
    "```\n",
    "network='networks/elec_s{simpl}_{clusters}.nc'\n",
    "```\n",
    "\n",
    "In PyPSA-Africa we implemented **wildcards** as they exist in PyPSA-Eur. Checking their [documentation](https://pypsa-eur.readthedocs.io/en/latest/wildcards.html#wildcards) out, one can see that we can remove `{simpl}` as preclustering is not necessary and replace `{cluster}` by any number of buses which the whole network should be aggregated towards.\n",
    "\n",
    "For instance:\n",
    "- `snakemake --cores 1 networks/elec_s_10.nc -n` will start a dry run for a network clustered to 10 nodes\n",
    "- `snakemake --cores 1 networks/elec_s_100.nc -n` will start a dry run for a network clustered to 100 nodes\n",
    "\n",
    "Note that these wildcards are what enables to run thousands of simulations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Try it yourself.\n",
    "\n",
    "**Find the below rule** in the `Snakefile` and create a dag:\n",
    "- `rule build_powerplants`\n",
    "\n",
    "**Find the below rules** and execute a dry-run from the 1) rule and 2) the output file:\n",
    "- `rule download_osm_data:`\n",
    "- `rule cluster_network:`\n",
    "\n",
    "**Find the rules with the below functions** and execute a dry-run from the 1) rule and 2) the output file:\n",
    "- `scripts/add_electricity.py`\n",
    "- `scripts/prepare_network.py`\n",
    "\n",
    "**Make a dry-run with the following output:**\n",
    "Use the [documentation](https://pypsa-eur.readthedocs.io/en/latest/wildcards.html#wildcards) to find values to replace the wildcards:\n",
    "```\n",
    "output: \"results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc\"\n",
    "```\n",
    "\n",
    "**Really cool. Modify the config.yaml and create a --dag:**\n",
    "\n",
    "Insert the following parameters in the `config.yaml`\n",
    "```\n",
    "scenario:\n",
    "  simpl: ['']\n",
    "  ll: ['copt']\n",
    "  clusters: [5,10,20,30]\n",
    "  opts: [Co2L0.0-1H, Co2L0.0-3H, Co2L0.0-6H, , Co2L0.0-12H]\n",
    "```\n",
    "\n",
    "Create a `--dag` with the solve_all_network rule. Can you explain from the `--dag` what will happen?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## snakemake --help\n",
    "There are some good execution tools which you might need. Read and test them if possible.\n",
    "- --forcerun\n",
    "- --forceall\n",
    "- --delete-all-output\n",
    "- --report\n",
    "- --debug\n",
    "\n",
    "To see the full list of snakemake options type `snakemake --help` in you terminal\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a new workflow rule\n",
    "\n",
    "Now that you know how to execute workflows, what about creating one?\n",
    "\n",
    "Just think first **what you want to achieve (=function)?**. Then simply add what you **expect as input and output**. This helps you designing multiple rules before even starting to code, supports organising your thougths and finally improves the rule integration into a larger workflow. Maybe you can even simplify or improve our existing snakemake rules? \n",
    "\n",
    "A good length of a digestible python script is around 300-500 lines.\n",
    "\n",
    "```\n",
    "rule YOUR_RULE:\n",
    "    input:\n",
    "        <Expected_inputs_required_by_the_function>=''\n",
    "    output:\n",
    "        <Expected_outputs_from_the_function>=''\n",
    "    log: \"logs/YOUR_RULE.log\"\n",
    "    threads: 1\n",
    "    resources: mem=1000\n",
    "    script: \"scripts/<YOUR_FUNCTION>.py\"\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Key points:\n",
    "- You understand the purpose of the Snakefile (rule, input, output, function)\n",
    "- You can create `--dag`'s (`snakemake --dag solve_all_networks | dot -Tpng -o workflow.png`)\n",
    "- You can execute rules in 3 different ways (output based, rule based or by running all workflows)\n",
    "- You understand how wildcards are used and executed\n",
    "- You know how to create a new workflow"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b34881217c78f48c1cb2cd3434cb375f471e98a15a99a90b485a8349351a20f8"
  },
  "kernelspec": {
   "display_name": "pypsa-africa",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}