Hypat workflow adaptations

.gitignore

@@ -46,6 +46,13 @@ doc/_build
 /scripts/data
 /sample_pypsa_eur
 
+/data/energy_totals_BS_2030.csv
+/data/energy_totals_BS_2050.csv
+/data/energy_totals_AP_2030.csv
+/data/energy_totals_AP_2050.csv
+/data/energy_totals_NZ_2030.csv
+/data/energy_totals_NZ_2050.csv
+
 # File extensions
 *.org
 *.nc

.pre-commit-config.yaml

@@ -36,7 +36,7 @@ repos:
 
 # Formatting with "black" coding style
 - repo: https://github.com/psf/black
-  rev: 23.7.0
+  rev: 23.1.0
   hooks:
   # Format Python files
   - id: black
@@ -45,21 +45,21 @@ repos:
 
 # Do YAML formatting (before the linter checks it for misses)
 - repo: https://github.com/macisamuele/language-formatters-pre-commit-hooks
-  rev: v2.10.0
+  rev: v2.7.0
   hooks:
   - id: pretty-format-yaml
     args: [--autofix, --indent, '2', --preserve-quotes]
 
 # Use yamllint to check for valid YAML files and list syntax errors
 - repo: https://github.com/adrienverge/yamllint.git
-  rev: v1.32.0
+  rev: v1.29.0
   hooks:
   - id: yamllint
     args: [--format, parsable, -c=.yamllint]
 
 # Format Snakemake rule / workflow files
 - repo: https://github.com/snakemake/snakefmt
-  rev: v0.8.4
+  rev: v0.8.2
   hooks:
   - id: snakefmt
 

README.md

@@ -17,7 +17,7 @@
 
 **Disclaimer: PyPSA-Earth-Sec is still under development.**
 
-The workflow is adaped to work smoothly for the following countries: Morocco, Namibia, Nigeria and Benin. The spatial and temporal resolution of the model are flexible. It's advisable to use more than 3 nodes per country and a timestep not smaller than 3-hours.
+The workflow is adaped to work smoothly for the following countries: Morocco, Namibia, Nigeria and Benin. The spatial and temporal resolution of the model are felxible. It's advisable to use more than 3 nodes per country and a timestep not smaller than 3-hours. 
 
 
 Currently, no real sectoral demand data is used for the country inspected, instead, we use dummy data. The collection, compilation and processing of real data is underway.
@@ -27,64 +27,60 @@ The model now includes the following energy carriers: **electricity**, **hydroge
 
 The demand sectors covered are: **residential**, **industry**, **land transport**, **aviation**, **shipping**, **services** and agriculture.
 
-The diagram below depicts one representative clustered node showing the combination of carriers and sectors covered in the model as well as the generation and conversion technologies included.
+The diagram below depicts one representative clustered node showing the combination of carriers and sectors covered in the model as well as the generation and conversion technologies included. 
 
 ![alt text](https://github.com/pypsa-meets-earth/pypsa-earth-sec/blob/main/docs/pes_v0.0.2.png?raw=true)
 
 
 
 ## Installation
 
-1. Open your terminal at a location where you want to install pypsa-earth-sec. Type the following in your terminal to download the package and the dependency (pypsa-earth) from GitHub.
-   Note that the tag `--recursive-submodules` is needed to automatically clone also the pypsa-earth dependency.
-
-   ```bash
-       .../some/path/without/spaces % git clone --recurse-submodules https://github.com/pypsa-meets-earth/pypsa-earth-sec.git
-   ```
-   To make sure you run the latest version of the submodule (if desired), run the following command to update the git submodules:
-   ```bash
-       .../some/path/without/spaces % git submodule update
-   ```
-
-2. Move the current directory to the head of the repository.
-   ```bash
-       .../some/path/without/spaces % cd pypsa-earth-sec
-   ```
-
-3. (optional) For reproducibility and compatibility purposes, it is possible to specify a specific version of the pypsa-earth submodule.
-   To do so, feel to reproduce the following lines, yet this is not mandatory.
-   If you desire to run the latest pypsa-earth model, please skip this point 3.
-
-   ```bash
-       .../pypsa-earth-sec % cd pypsa-earth
-       .../pypsa-earth % git reset --hard \{commit id\}
-       .../pypsa-earth % cd ..
-   ```
-
-4. The python package requirements are curated in the `envs/environment.yaml` file of the pypsa-earth repository.
+1. Open your terminal at a location where you want to install pypsa-earth-sec. Type the following in your terminal to download the package and the dependency (pypsa-earth) from GitHub:
+
+```bash
+    .../some/path/without/spaces % git clone https://github.com/pypsa-meets-earth/pypsa-earth.git
+    .../some/path/without/spaces % git clone https://github.com/pypsa-meets-earth/pypsa-earth-sec.git
+```
+
+2. The python package requirements are curated in the `envs/environment.yaml` file.
    The environment can be installed using:
 
-   ```bash
-       .../pypsa-earth-sec % conda env create -f pypsa-earth/envs/environment.yaml
-   ```
+```bash
+    .../some/path/without/spaces % cd pypsa-earth
+    .../pypsa-earth % conda env create -f envs/environment.yaml
+```
+
+3. For running the optimization one has to install the solver. We can recommend the open source HiGHs solver which installation manual is given [here](https://github.com/PyPSA/PyPSA/blob/633669d3f940ea256fb0a2313c7a499cbe0122a5/pypsa/linopt.py#L608-L632).
 
-5. For running the optimization one has to install the solver. We can recommend the open source HiGHs solver which installation manual is given [here](https://github.com/PyPSA/PyPSA/blob/633669d3f940ea256fb0a2313c7a499cbe0122a5/pypsa/linopt.py#L608-L632).
+4. To use jupyter lab (new jupyter notebooks) **continue** with the [ipython kernel installation](http://echrislynch.com/2019/02/01/adding-an-environment-to-jupyter-notebooks/) and test if your jupyter lab works:
 
-6. To use jupyter lab (new jupyter notebooks) **continue** with the [ipython kernel installation](http://echrislynch.com/2019/02/01/adding-an-environment-to-jupyter-notebooks/) and test if your jupyter lab works:
+```bash
+    .../pypsa-earth % ipython kernel install --user --name=pypsa-earth
 
-   ```bash
-       .../pypsa-earth % ipython kernel install --user --name=pypsa-earth
-       .../pypsa-earth % jupyter lab
-   ```
+    .../pypsa-earth % jupyter lab
+```
 
 ## Test run
 
-- In the folder *pypsa-earth-sec* open a terminal/command window to be located at this path `./pypsa-earth-sec/`
+- In the folder *pypsa-earth* open a terminal window to be located at this path `~/pypsa-earth/`
+- Rename config.default.yaml to config.yaml. For instance in Linux:
+`mv config.default.yaml config.yaml`
+- Open the file `config.yaml` and 
+  - choose the country you want to model. For example
+    `countries: ["MA"]`
+  - Make sure `build_cutout` is set to `true`
+- In the terminal run the following commands:
+  - Activate the environment: `conda activate pypsa-earth`
+  - Create the base network for the country by running : `snakemake -j 1 base_network`
+
+
+- In the folder *pypsa-earth-sec* open a terminal/command window to be located at this path `~/pypsa-earth-sec/`
 - Rename config.default.yaml to config.yaml. For instance in Linux:
 `mv config.default.yaml config.yaml`
 - Open the file `config.yaml` and follow the steps done before in pypsa-earth
   - choose the country you want to model. For example
     `countries: ["MA"]`
+ - Open the file `config.pypsa-earth.yaml` and make sure `build_cutout` is set to `true`
 
 - Run a dryrun of the Snakemake workflow by typing simply in the terminal:
   ```bash

Snakefile

@@ -12,9 +12,6 @@ if not exists("config.yaml"):
 configfile: "config.yaml"
 
 
-PYPSAEARTH_FOLDER = "./pypsa-earth"
-
-
 SDIR = config["summary_dir"] + config["run"]
 RDIR = config["results_dir"] + config["run"]
 CDIR = config["costs_dir"]
@@ -39,9 +36,9 @@ wildcard_constraints:
 
 subworkflow pypsaearth:
     workdir:
-        PYPSAEARTH_FOLDER
+        "../pypsa-earth"
     snakefile:
-        PYPSAEARTH_FOLDER + "/Snakefile"
+        "../pypsa-earth/Snakefile"
     configfile:
         "./config.pypsa-earth.yaml"
 
@@ -78,14 +75,6 @@ rule solve_all_networks:
         ),
 
 
-rule prepare_ports:
-    output:
-        ports="data/ports.csv",
-        # TODO move from data to resources
-    script:
-        "scripts/prepare_ports.py"
-
-
 rule prepare_sector_network:
     input:
         network=RDIR
@@ -113,6 +102,7 @@ rule prepare_sector_network:
         shapes_path=pypsaearth(
             "resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
         ),
+        pipelines="resources/custom_data/pipelines.csv",
     output:
         RDIR
         + "/prenetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}.nc",
@@ -132,7 +122,6 @@ rule add_export:
     input:
         overrides="data/override_component_attrs",
         export_ports="data/export_ports.csv",
-        costs=CDIR + "costs_{planning_horizons}.csv",
         network=RDIR
         + "/prenetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}.nc",
         shapes_path=pypsaearth(
@@ -496,9 +485,8 @@ rule run_test:
     run:
         import yaml
 
-        with open(PYPSAEARTH_FOLDER + "/config.tutorial.yaml") as file:
+        with open("../pypsa-earth/config.tutorial.yaml") as file:
             config_pypsaearth = yaml.full_load(file)
-            config_pypsaearth["retrieve_databundle"] = {"show_progress": False}
             config_pypsaearth["electricity"]["extendable_carriers"]["Store"] = []
             config_pypsaearth["electricity"]["extendable_carriers"]["Link"] = []
             config_pypsaearth["electricity"]["co2limit"] = 7.75e7
@@ -512,8 +500,8 @@ rule run_test:
 
 rule clean:
     run:
-        shell("rm -r " + PYPSAEARTH_FOLDER + "/resources")
-        shell("rm -r " + PYPSAEARTH_FOLDER + "/networks")
+        shell("rm -r ../pypsa-earth/resources")
+        shell("rm -r ../pypsa-earth/networks")
 
 
 if config["custom_data"].get("industry_demand", False) == True:
@@ -525,8 +513,10 @@ if config["custom_data"].get("industry_demand", False) == True:
             ),
             clustered_pop_layout="resources/population_shares/pop_layout_elec_s{simpl}_{clusters}.csv",
             industrial_database="resources/custom_data/industrial_database.csv",
-            #shapes_path=pypsaearth("resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson")
-            shapes_path=PYPSAEARTH_FOLDER + "/resources/shapes/MAR2.geojson",
+            shapes_path=pypsaearth(
+                "resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
+            ),
+            #shapes_path="../pypsa-earth/resources/shapes/MAR2.geojson",
         output:
             industrial_distribution_key="resources/demand/industrial_distribution_key_elec_s{simpl}_{clusters}.csv",
         threads: 1

config.default.yaml

@@ -5,7 +5,7 @@ results_dir: results/
 summary_dir: results/
 costs_dir: data/ #TODO change to the equivalent of technology data
 
-run: test_h2_cons_yearly
+run: runname
 
 foresight: overnight
 
@@ -26,13 +26,10 @@ scenario:
   - "DF"
 
 policy_config:
-  policy: "H2_export_yearly_constraint" #either "H2_export_yearly_constraint", "H2_export_monthly_constraint", "no_res_matching"
-  monthly: # Specify attributes for the monthly constraint
-    allowed_excess: 1.0
-    reference_case: false # RE electricity is equal to the amount required for additional hydrogen export compared to the 0 export case ("reference_case")
-  yearly: # Specify attributes for the yearly constraint
-    re_country_load: false # Set to "True" to force the RE electricity to be equal to the electricity required for hydrogen export and the country electricity load. "False" excludes the country electricity load from the constraint.
-
+  policy: "H2_export_monthly_constraint" #either "H2_export_yearly_constraint", "H2_export_monthly_constraint", "no_res_matching" # TODO check wether "export" is required here or misleading
+  allowed_excess: 3 # currently only implemented for the monthly constraint
+  reference_case: true
+  allowed_excess: 1.0
 
 clustering_options:
   alternative_clustering: true
@@ -49,9 +46,7 @@ hydrogen_underground_storage: false
 
 
 export:
-  h2export: [120] # Yearly export demand in TWh
-  store: true # [True, False] # specifies wether an export store to balance demand is implemented
-  store_capital_costs: "no_costs" # ["standard_costs", "no_costs"] # specifies the costs of the export store. "standard_costs" takes CAPEX of "hydrogen storage tank incl. compressor"
+  h2export: [1, 5] # Yearly export demand in TWh
 
 custom_data:
   renewables: [] # ['csp', 'rooftop-solar', 'solar']
@@ -264,7 +259,7 @@ snapshots:
   # arguments to pd.date_range
   start: "2013-01-01"
   end: "2014-01-01"
-  inclusive: "left" # end is not inclusive
+  closed: "left" # end is not inclusive
 
 # atlite:
 #   cutout: ./cutouts/africa-2013-era5.nc