Improve documentation 2

doc/customization_basic1.rst

@@ -0,0 +1,89 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_basic1:
+
+#######################
+1. Basic customization
+#######################
+
+A good starting point to customize your model are settings of the default configuration file `config.default`. You may want to do a reserve copy of your current configuration file and then overwrite it by a default configuration:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % cp config.default.yaml config.yaml
+
+The model can be adapted to include any country, multiple countries (e.g. `Nigeria` and `Benin`) or full continents (currently whole regions, such as `Africa`, `Asia`, `Europe`, `Oceania`, `NorthAmerica`, and `SouthAmerica`, are available for simulation) using ``countries`` argument:
+
+.. code:: yaml
+
+    countries: ["NG", "BJ"]
+
+.. note::
+
+    To build the model for countries of Oceania region, it is important to setup ``Copernicus Climate Data Store (CDS) API`` to build custom cutouts.
+    The same is true if the weather year other than 2013 is considered for any country or continents. The detailed instructions for setting up the Copernicus API can be found in :ref:`customization_copernicus`.
+    However, if you are using default weather year of 2013, there is no need to configure ``Copernicus API`` as pre-compiled cutouts are downloaded automatically. Currently, ongoing work is being conducted to produce pre-compiled cutouts for the Oceania region.
+
+Likewise, the example's temporal scope can be restricted (e.g. to 7 days):
+
+.. code:: yaml
+
+    snapshots:
+        start: "2013-03-01"
+        end: "2013-03-07"
+        inclusive: "left" # end is not inclusive
+
+.. note::
+
+    Ensure that the selected data range aligns with the dates available in the cutout dataset. If the weather data within the cutouts corresponds to the year 2013, then the range of snapshots should fall within that same year.
+
+Year-related parameters are also being used when specifying `load_options`:
+
+.. code:: yaml
+
+    load_options:
+      ssp: "ssp2-2.6"
+      weather_year: 2013
+      prediction_year: 2030
+      scale: 1
+
+The `weather_year` value corresponds to the weather data which was used to generate the electricity demand profiles for a selected area while `prediction_year` correspond to the point of a `Shared Socioeconomic Pathways (SSP) <https://en.wikipedia.org/wiki/Shared_Socioeconomic_Pathways>`__ trajectory. PyPSA-Earth uses SSP2-2.6 scenario within the Shared Socioeconomic Pathways framework, which is characterized by medium challenges to mitigation and adaptation efforts resulting in a global warming of approximately 2.6°C by the end of the 21st century.
+The available values for `weather_year` and `prediction_year` can be checked by looking into `pypsa-earth/data/ssp2-2.6` folder. Currently, there are pre-calculated demand data for 2011, 2013, 2018 weather years and for 2030, 2040, 2050, and 2100 scenario prediction years.
+
+To accurately model the temporal and spatial availability of renewables such as wind and solar energy, we process historical weather data using `atlite <https://atlite.readthedocs.io/en/latest/>`__ package.
+Atlite configurations can be adjusted in ``config.yaml``:
+
+.. code:: yaml
+
+    atlite:
+        nprocesses: 4
+        cutouts:
+            cutout-2013-era5:
+                module: era5
+                dx: 0.3  # cutout resolution
+                dy: 0.3  # cutout resolution
+                # The cutout time is automatically set by the snapshot range.
+
+.. note::
+
+    No adjustments are required when utilizing pre-compiled cutouts. When using custom cutouts generated by ``build_cutout`` rule, replace all entries of ``cutout-2013-era5`` with the custom cutout name for a region of interest. For example, when simulating Kazakhstan with ``cutouts: asia-2013-era5``, every occurrence of ``cutout-2013-era5`` should be updated to ``asia-2013-era5`` which refers to ``asia-2013-era5.nc`` file generated in ``cutouts`` folder.
+
+Please note that a temporal dimension of the cutout should be consistent with the values set for `snapshots` parameter. A time range of the cutout is determined by the parameters set when building this cutout while the time resolution corresponds to those of the used climate archives. In case of ERA5 dataset used in PyPSA-Earth by default, hourly resolution is implied.
+Visit :ref:`config` page to get familiar with configuration details.
+
+Finally, for Oceania countries or when simulating different weather year than 2013, it is imperative to initially set ``build_cutout: true`` for the first run. This facilitates the construction of cutouts from weather data. Subsequently, it can be switched to false to avoid reconstructing the cutout. The same applies for other parameters in ``enable`` section.
+
+.. code:: yaml
+
+    enable:
+        retrieve_databundle: true  #  Recommended 'true', for the first run. Otherwise data might be missing.
+        retrieve_cost_data: true  # true: retrieves cost data from technology data and saves in resources/costs.csv, false: uses cost data in data/costs.csv
+        download_osm_data: true  # If 'true', OpenStreetMap data will be downloaded for the above given countries
+        build_natura_raster: true # If True, than an exclusion raster will be build
+        build_cutout: false
+        # If "build_cutout" : true, then environmental data is extracted according to `snapshots` date range and `countries`
+
+To delve into the specifics of the provided configurations and explore additional settings, please refer to the :ref:`config` page.
+There are many more configuration options beyond what is adapted for the tutorial!

doc/customization_copernicus.rst

@@ -0,0 +1,31 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_copernicus:
+
+####################
+Setup Copernicus API
+####################
+
+
+To build custom cutouts, it is important to access to `Copernicus Climate Data Store <https://cds.climate.copernicus.eu>`__.
+
+.. note::
+
+    Skip this recommendation if the region of your interest is anywhere except Oceania and you are fine with the 2013 weather year.
+
+Steps to get access to Copernicus database:
+
+1. Register to  the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu>`_;
+2. Setup your CDS API key as described `on their website <https://cds.climate.copernicus.eu/api-how-to>`_.
+
+These steps are required to use CDS API which allows an automatic file download while executing `build_cutouts` rule.
+
+The `build_cutout` flag should be set `true` to generate the cutout. After the cutout is ready, it's recommended to set `build_cutout` to `false` to avoid overwriting the existing cutout by accident. The `snapshots` values set when generating the cutout, will determine the temporal parameters of the cutout. Accessible years which can be used to build a cutout depend on ERA5 data availability. `ERA5 page <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_ explains that the data is available from 1950 and updated continuously with about 3 month delay while the data on 1950-1978 should be treated as preliminary as that is a rather recent development.
+
+.. note::
+
+    Building a cutout may require a significant amount of time and storage space. Continental cutouts, such as those for Asia, South America, and Africa, typically require around 20 GB of storage space, while cutouts for individual countries or small regions may occupy approximately 1-5 GB. The process of building a cutout can take between 2 to 3 hours.
+
+After the first run, if you don't change country and don't need to increase a considered time span wider than the one you created the cutout with, you may set both `retrieve_databundle` and `build_cutout` to false.

doc/customization_run.rst

@@ -0,0 +1,60 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_run:
+
+####################
+2. Running the model
+####################
+
+To solve full optimization problem, it is important to pick a solver in `config.yaml` file. For instance, this tutorial uses the open-source solver glpk and does not rely
+on the commercial solvers such as Gurobi or CPLEX (for which free academic licenses are available).
+
+.. code:: yaml
+
+    solver:
+        name: glpk
+
+.. note::
+
+    ``glpk`` can solve the network with low temporal and spatial resolution. To make a full model run, it is advised to use ``CPLEX``, ``Gurobi``, or open-source `HIGHs <https://highs.dev/>`__.
+
+To execute a full model run, the following command needs to be applied:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake --cores 1 solve_all_networks
+
+Here, ``snakemake`` is a workflow management tool inherited by PyPSA-Earth from PyPSA-Eur.
+Snakemake decomposes a large software process into a set of subtasks, or ’rules’, that are automatically chained to obtain the desired output.
+The flag ``--cores 1`` dictates the number of CPU cores allocated for the process. Notably, the ``solve_all_network`` rule within Snakemake orchestrates the process of solving the network.
+
+.. note::
+
+  ``Snakemake``, which is one of the major dependencies, will be automatically installed in ``pypsa-earth`` environment, thereby there is no need to install it manually.
+
+The snakemake included in the ``pypsa-earth`` conda environment pypsa-earth can be used to execute any custom rule with the following command:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake < your custom rule >
+
+Starting with essential usability features, PyPSA-Earth `Snakemake procedure <https://github.com/pypsa-meets-earth/pypsa-earth/blob/main/Snakefile>`_ enables users to execute the entire workflow flexibly with diverse options, requiring no Python coding. For example, users can model the global energy system or specific subsets of countries using only necessary data. Wildcards, serving as special generic keys, adapt to multiple values based on configuration options, facilitating the execution of extensive workflows with parameter sweeps and diverse options.
+
+You can execute some parts of the workflow in case you are interested in some specific parts.
+E.g. power grid topology may be extracted and cleaned with the following command which refers to the script name:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake -j 1 clean_osm_data
+
+Solar profile for the requested area may be calculated using the output name:
+
+.. code:: bash
+
+    .../pypsa-earth (pypsa-earth) % snakemake -j 1 resources/renewable_profiles/profile_solar.nc
+
+.. note::
+
+    Because of the wildcards in the ``build_renewable_profiles`` rule definition, the only available option to run the rule separately is by directly calling the output file in the snakemake command.

doc/customization_steps.rst

@@ -0,0 +1,55 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_steps:
+
+#######################################
+3. General Guidelines for Modeling
+#######################################
+
+
+1. Configure Model Parameters
+-----------------------------
+
+Begin by adjusting the model inputs in the ``config.yaml`` configuration file. This file serves as the cornerstone for tailoring PyPSA-Earth to specific regional requirements.
+
+Pay particular attention to regional-dependent parameters such as:
+
+* ``countries``: This parameter allows you to specify the countries to be included in your model. It's crucial for accurately representing the geographical scope of your analysis.
+
+* ``cutouts``: These parameters refer to the climate data archive name used for calculating renewable potential. By defining appropriate cutouts, you ensure that your model is equipped with the necessary weather data for accurate simulations. By default, pre-compiled cutouts have standard name of ``cutout-2013-era5``. Adjust it accordingly, if custom cutouts are utilized.
+
+2. Generate Custom Cutout
+-------------------------
+
+The concept of a ``cutout`` is central to climate data management in the PyPSA ecosystem, facilitated through the ``build_cutout`` rule.
+A cutout essentially represents a spatio-temporal subset of topology and weather datasets, allowing you to focus on specific regions and timeframes relevant to your analysis.
+While pre-built cutouts are readily available for the 2013 year for Africa, Europe, Asia and America, users interested in other regions or updated data can generate their own cutouts by enabling the ``build_cutout`` rule. This process involves accessing the Copernicus Climate Data Store and installing the necessary packages for data retrieval as described in :ref:`customization_copernicus`.
+
+1. Assess `natura.tiff` Raster
+--------------------------------
+
+The ``natura.tiff`` raster file delineates areas designated as protected and reserved nature zones. These zones impose land use restrictions, influencing the calculation of renewable potential within a given geographical region.
+By activating the ``natura`` option and incorporating the ``natura.tiff`` file, users can account for protected and reserved nature areas, where renewable assets cannot be installed:
+
+.. code:: yaml
+
+    renewable:
+        onwind:
+            natura: true
+
+.. note::
+
+    By default, the option ``natura: true`` is set for all renewable sources.
+
+It's worth noting that a pre-built ``natura.tiff`` file is included in the dataset and downloaded through ``retrieve_databundle_light`` rule, offering a global coverage of protected areas. This default file serves as a foundational resource for model execution.
+
+While the default file suffices for most analyses, users seeking the latest updates and revisions can opt to enable the ``build_natura_raster`` rule:
+
+.. code:: yaml
+
+    enable:
+        build_natura_raster: true
+
+However, it's worth noting that generating a customized ``natura.tiff`` file with the latest data may require a significant amount of time due to processing complexities.

doc/customization_validation.rst

@@ -0,0 +1,9 @@
+.. SPDX-FileCopyrightText:  PyPSA-Earth and PyPSA-Eur Authors
+..
+.. SPDX-License-Identifier: CC-BY-4.0
+
+.. _customization_validation:
+
+################
+Model Validation
+################