Merge PR #959 from yerbol-akhmetov/pypsa-earth

* rename section and add subdirectories

* add customization_basics file

* add copernicus description file

* complete basic customization page

* add running the model page

* add running the model link

* complete run model page

* add extra emphasis on other configuration options

* fill up copernicus page further

* add general guidelines section

* add general guidelines page

* add model validation page

* address review comments

* address comments on pre-compiled cutouts

* Address additional comments

* address new comments on storage space

* expand explanation on rule execution

* remove unnecessary information

* Add minor fixes

* Update a top-level section

* Update a top-level section

* Fix typos

* remove oceania mention

* address comments for basic customization page

* add submodule names and reorder solver configuration

* reorder sections

* address general guidelines comments

* add validation part

* make validation separate page

* add short guidelines

* reorder pages and change numbering

* update URL for IRENA database

* Fix typos

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Authored-by: Yerbol Akhmetov <[email protected]>

doc/customization_basic1.rst

@@ -0,0 +1,109 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_basic1:
+
+#######################
+2. Basic customization
+#######################
+
+A good starting point to customize your model are settings of the default configuration file `config.default.yaml`. You may want to do a reserve copy of your current configuration file and then overwrite it by a default configuration:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % cp config.default.yaml config.yaml
+
+Specify the country/region of interest
+--------------------------------------
+
+The model can be adapted to include any country, multiple countries (e.g. `Nigeria` and `Benin`) or full continents (currently whole regions, such as `Africa`, `Asia`, `Europe`, `Oceania`, `NorthAmerica`, and `SouthAmerica`, are available for simulation) using ``countries`` argument:
+
+.. code:: yaml
+
+    countries: ["NG", "BJ"]
+
+Configure ``enable`` section to download/build data
+---------------------------------------------------------
+
+For a successful model run, ensure the download of essential open-source data, including databundle and cost data, is activated in the ``enable`` section:
+
+.. code:: yaml
+
+    enable:
+        retrieve_databundle: true  #  Recommended 'true', for the first run. Otherwise data might be missing.
+        retrieve_cost_data: true  # true: retrieves cost data from technology data and saves in resources/costs.csv, false: uses cost data in data/costs.csv
+        download_osm_data: true  # If 'true', OpenStreetMap data will be downloaded for the above given countries
+        build_natura_raster: false # If True, than an exclusion raster will be build
+        build_cutout: false
+        # If "build_cutout" : true, then environmental data is extracted according to `snapshots` date range and `countries`
+
+After the initial run, it is recommended to set the retrieval of databundle and cost data to ``false`` to prevent unnecessary redownloading of data.
+
+When ``build_natura_raster: false`` is utilized, the exclusion raster for protected areas is sourced from the pre-compiled ``data/natura.tiff`` file downloaded with the databundle. Conversely, if ``build_natura_raster`` is set to true, the exclusion raster, delineating areas where renewables cannot be installed, is computed using the ``build_natura_raster rule``.
+
+When using the weather year 2013, it is recommended to use default ``build_cutout: false`` because pre-compiled cutouts are automatically downloaded with ``retrieve_databundle: true``.
+On contrary, when simulating a weather year other than 2013, it is crucial to set ``build_cutout: true`` in order to generate custom cutouts. However, it is essential to first configure the `Copernicus Climate Data Store (CDS) API`. Detailed instructions for setting up the `Copernicus API` can be found in :ref:`customization_copernicus`.
+After initial run and successful generation of custom cutouts, ``build_cutout`` can be switched to false to avoid reconstructing the cutout.
+
+.. note::
+
+    No need to configure the `Copernicus API` if the weather year 2013 is used, as pre-compiled cutouts are automatically downloaded.
+
+Specify the weather year scope
+------------------------------
+
+Likewise, the example's temporal scope can be restricted (e.g. to 7 days):
+
+.. code:: yaml
+
+    snapshots:
+        start: "2013-03-01"
+        end: "2013-03-07"
+        inclusive: "left" # end is not inclusive
+
+.. note::
+
+    Ensure that the selected date range aligns with the dates available in the cutout dataset. If the weather data within the cutouts corresponds to the year 2013, then the range of snapshots should fall within that same year.
+
+Specify the demand year
+-----------------------
+
+Year-related parameters are also being used when specifying `load_options`:
+
+.. code:: yaml
+
+    load_options:
+      ssp: "ssp2-2.6"
+      weather_year: 2013
+      prediction_year: 2030
+      scale: 1
+
+The `weather_year` value corresponds to the weather data which was used to generate the electricity demand profiles for a selected area while `prediction_year` corresponds to the point of a `Shared Socioeconomic Pathways (SSP) <https://en.wikipedia.org/wiki/Shared_Socioeconomic_Pathways>`__ trajectory. PyPSA-Earth uses SSP2-2.6 scenario within the Shared Socioeconomic Pathways framework, which is characterized by medium challenges to mitigation and adaptation efforts resulting in a global warming of approximately 2.6°C by the end of the 21st century.
+The available values for `weather_year` and `prediction_year` can be checked by looking into `pypsa-earth/data/ssp2-2.6` folder. Currently, there are pre-calculated demand data for 2011, 2013, 2018 weather years and for 2030, 2040, 2050, and 2100 scenario prediction years.
+
+Configure `atlite` section
+--------------------------
+
+To accurately model the temporal and spatial availability of renewables such as wind and solar energy, we process historical weather data using `atlite <https://atlite.readthedocs.io/en/latest/>`__ package.
+Atlite configurations can be adjusted in ``config.yaml``:
+
+.. code:: yaml
+
+    atlite:
+        nprocesses: 4
+        cutouts:
+            cutout-2013-era5:
+                module: era5
+                dx: 0.3  # cutout resolution
+                dy: 0.3  # cutout resolution
+                # The cutout time is automatically set by the snapshot range.
+
+.. note::
+
+    No adjustments are required when utilizing pre-compiled cutouts. When using custom cutouts generated by ``build_cutout`` rule, replace all entries of ``cutout-2013-era5`` with the custom cutout name for a region of interest. For example, when simulating Kazakhstan with ``cutouts: asia-2013-era5``, every occurrence of ``cutout-2013-era5`` should be updated to ``asia-2013-era5`` which refers to ``asia-2013-era5.nc`` file generated in ``cutouts`` folder.
+
+Please note that a temporal dimension of the cutout should be consistent with the values set for `snapshots` parameter. A time range of the cutout is determined by the parameters set when building this cutout while the time resolution corresponds to those of the used climate archives. In case of ERA5 dataset used in PyPSA-Earth by default, hourly resolution is implied.
+
+To delve into the specifics of the provided configurations and explore additional settings, please refer to the :ref:`config` page.
+There are many more configuration options beyond what is adapted for the tutorial!

doc/customization_copernicus.rst

@@ -0,0 +1,31 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_copernicus:
+
+#######################
+5. Setup Copernicus API
+#######################
+
+
+To build custom cutouts, it is important to access to `Copernicus Climate Data Store <https://cds.climate.copernicus.eu>`__.
+
+.. note::
+
+    Skip this recommendation if you are fine with the 2013 weather year.
+
+Steps to get access to Copernicus database:
+
+1. Register to  the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu>`_;
+2. Setup your CDS API key as described `on their website <https://cds.climate.copernicus.eu/api-how-to>`_.
+
+These steps are required to use CDS API which allows an automatic file download while executing `build_cutouts` rule.
+
+The `build_cutout` flag should be set `true` to generate the cutout. After the cutout is ready, it's recommended to set `build_cutout` to `false` to avoid overwriting the existing cutout by accident. The `snapshots` values set when generating the cutout, will determine the temporal parameters of the cutout. Accessible years which can be used to build a cutout depend on ERA5 data availability. `ERA5 page <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_ explains that the data is available from 1950 and updated continuously with about 3 month delay while the data on 1950-1978 should be treated as preliminary as that is a rather recent development.
+
+.. note::
+
+    Building a cutout may require a significant amount of time and storage space. Continental cutouts, such as those for Asia, South America, and Africa, typically require around 20 GB of storage space, while cutouts for individual countries or small regions may occupy approximately 1-5 GB. The process of building a cutout can take between 2 to 3 hours.
+
+After the first run, if you don't change country and don't need to increase a considered time span wider than the one you created the cutout with, you may set both `retrieve_databundle` and `build_cutout` to false.

doc/customization_run.rst

@@ -0,0 +1,70 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_run:
+
+####################
+3. Running the model
+####################
+
+Run full model
+--------------
+
+PyPSA-Earth utilized ``snakemake`` workflow management tool inherited from PyPSA-Eur.
+Snakemake decomposes a large software process into a set of subtasks, or ’rules’, that are automatically chained to obtain the desired output.
+
+To execute a full model run, the following command needs to be applied:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake --cores 1 solve_all_networks
+
+Here, the flag ``--cores 1`` dictates the number of CPU cores allocated for the process. Notably, the ``solve_all_network`` rule within Snakemake orchestrates the process of solving the network.
+
+.. note::
+
+  ``Snakemake``, which is one of the major dependencies, will be automatically installed in ``pypsa-earth`` environment, thereby there is no need to install it manually.
+
+Execute different part of the workflow
+--------------------------------------
+
+The snakemake included in the ``pypsa-earth`` conda environment pypsa-earth can be used to execute any custom rule with the following command:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake < your custom rule >
+
+Starting with essential usability features, PyPSA-Earth `Snakemake procedure <https://github.com/pypsa-meets-earth/pypsa-earth/blob/main/Snakefile>`_ enables users to execute the entire workflow flexibly with diverse options, requiring no Python coding. For example, users can model the global energy system or specific subsets of countries using only necessary data. Wildcards, serving as special generic keys, adapt to multiple values based on configuration options, facilitating the execution of extensive workflows with parameter sweeps and diverse options.
+
+You can execute some parts of the workflow in case you are interested in some specific parts.
+E.g. power grid topology may be extracted and cleaned with the following command which refers to the script name:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake -j 1 clean_osm_data
+
+Solar profile for the requested area may be calculated using the output name:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake -j 1 resources/renewable_profiles/profile_solar.nc
+
+.. note::
+
+    Because of the wildcards in the ``build_renewable_profiles`` rule definition, the only available option to run the rule separately is by directly calling the output file in the snakemake command.
+
+Choose the solver
+-----------------
+
+To solve a full optimization problem, it is important to pick a solver in `config.yaml` file. For instance, this tutorial uses the open-source solver glpk and does not rely
+on the commercial solvers such as Gurobi or CPLEX (for which free academic licenses are available).
+
+.. code:: yaml
+
+    solver:
+        name: glpk
+
+.. note::
+
+    ``glpk`` can solve the network with low temporal and spatial resolution. To make a full model run, it is advised to use ``CPLEX``, ``Gurobi``, or open-source `HIGHs <https://highs.dev/>`__.

doc/customization_steps.rst

@@ -0,0 +1,21 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_steps:
+
+#######################################
+1. General Modeling Guidelines
+#######################################
+
+To simplify navigation through the :ref:`tutorial` section, we offer a summary as follows:
+
+1. **Basic customization:** Provides guidance on customizing the default configuration file to suit specific modeling needs, including specifying the country/region of interest and enabling essential data downloads.
+
+2. **Running the model:** Explains the process of running the model using the Snakemake workflow management tool, selecting a solver, and ensuring successful optimization.
+
+3. **Model Validation:** Outlines procedures for validating data obtained from PyPSA-Earth, including simulation procedures, reference data sources, and advanced validation examples.
+
+4. **Setup Copernicus API:** Describes the steps to access the Copernicus Climate Data Store for building custom cutouts, including registration, setting up the CDS API key, and considerations for cutout generation.
+
+Each section provides essential information for effective model usage and customization, ensuring accurate and reliable results.

doc/customization_validation.rst

@@ -0,0 +1,59 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_validation:
+
+###################
+4. Model Validation
+###################
+
+To validate the data obtained with PyPSA-Earth, we recommend to go through the procedure here detailed. An exampled of the validation procedure is available in the `Nigeria validation <https://github.com/pypsa-meets-earth/documentation/blob/main/notebooks/validation/validation_nigeria.ipynb>`_ notebook. Public information on the power system of Nigeria are compared to those obtained from the PyPSA-Earth model.
+
+Simulation procedure
+^^^^^^^^^^^^^^^^^^^^
+
+It may be recommended to check the following quantities in the validation:
+
+#. Inputs used by the model:
+
+    #. Network characteristics
+
+    #. Substations
+
+    #. Installed generation by type
+
+#. Outputs of the simulation:
+
+    #. Demand
+
+    #. Energy mix
+
+Where to look for reference data
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Data availability for many parts of the world is still quite limited. Usually the best sources to compare with are regional data hubs. There is also a collection of harmonized datasets curated by the international organisations. A non-exhaustive list of helpful sources:
+
+* `World Bank <https://energydata.info/>`_;
+
+* International Renewable Energy Agency `IRENA <https://pxweb.irena.org/pxweb/en/IRENASTAT/>`_;
+
+* International Energy Agency `IEA <https://www.iea.org/data-and-statistics>`_;
+
+* `BP <https://www.bp.com/en/global/corporate/energy-economics/statistical-review-of-world-energy.html>`_ Statistical Review of World Energy;
+
+* `Ember <https://ember-climate.org/data/data-explorer/>`_ Data Explorer.
+
+
+Advanced validation examples
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following validation notebooks are worth a look when validating your energy model:
+
+1. A detailed `network validation <https://github.com/pypsa-meets-earth/documentation/blob/main/notebooks/validation/network_validation.ipynb>`_.
+
+2. Analysis of `the installed capacity <https://github.com/pypsa-meets-earth/documentation/blob/main/notebooks/validation/capacity_validation.ipynb>`_ for the considered area.
+
+3. Validation of `the power demand <https://github.com/pypsa-meets-earth/documentation/blob/main/notebooks/validation/demand_validation.ipynb>`_ values and profile.
+
+4. Validation of `hydro <https://github.com/pypsa-meets-earth/documentation/blob/main/notebooks/validation/hydro_generation_validation.ipynb>`_, `solar and wind <https://github.com/pypsa-meets-earth/documentation/blob/main/notebooks/validation/renewable_potential_validation.ipynb>`_ potentials.