Establish snakemake workflow by importing pypsa-earth rules as module

[yerbol-akhmetov]

Good day. This draft PR is focused on importing the pypsa-earth rules into local snakemake. It is still a draft PR, it is still not finished.

The current implementation enables to execute snakemake --cores 1 retrieve_databundle_light --configfile configs/calibration/config.usa_PE.yaml in efuels-supply-potentials directory. However, it requires few changes in pypsa-earth/scripts/_helpers.py. In particular, the path for config.default.yaml in check_config_version function needs to be absolute. This was the change: https://github.com/yerbol-akhmetov/pypsa-earth/blob/1ccce2e128dc72b0ebb6344b7758cce8dbee369d/scripts/_helpers.py#L42-L48
I have pushed this change as snakemake_import branch: https://github.com/yerbol-akhmetov/pypsa-earth/tree/snakemake_import

Currently, if we execute retrieve databundle command, the command will execute, but files will be stored in main drive folder (C://). It is because the path for output files in pypsa-earth Snakefile refer to relative path. This is given as example. The outputs are generally: "results/", "resources/.." and etc.

I have tried a step further by adding absolute path to outputs and inputs in Snakefile of pypsa-earth and it seemed to work.

@FabianHofmann, @davide-f, @martacki, I would be very glad to get a hint if such approach is right. Currently, the problem is when outer Snakemake rules are executed, they treat inner pypsa-earth rules being executed as in outside (in efuels-supply-potential directory). Appending sys.path.append('submodules/pypsa-earth') did not helped though. The only option I currently see is to make pypsa-earth rules use absolute paths, something like this:

[yerbol-akhmetov]

Inclusion of submodules/pypsa-earth as a prefix for module import helped to resolve the relative path issue for imported rules' input/outputs. However, there is an issue left related to not finding log destination for simplify_network and cluster_network rules. Other rules works fine from efuels-supply-potential repository.

I have added test_modify_network rule for testing the ability of the custom rules to trigger imported rules. This will be helpful when starting to add scripts to the workflow (we can delete it too).

[yerbol-akhmetov]

Hi, @GbotemiB. Could you please test the PR. Try solving the network from efuels-supply-potential repository. You can run snakemake -call add_electricity --configfile configs/calibration/config.base.yaml (or solve network). As pypsa-earth version you can use my PRs (pypsa-meets-earth/pypsa-earth#1134) and (pypsa-meets-earth/pypsa-earth#1137) in upstream. There are two files need to be changed: 1) add_extra_components.py and 2) _helpers.py. I suggest commenting build_osm_network and build_renewable_profiles in pypsa-earth/Snakefile to avoid running those long rules.

[GbotemiB]

Hi @yerbol-akhmetov, I tested it on a fresh repo. I got this error

Config file submodules/pypsa-earth/config.default.yaml is extended by additional config specified via the command line.
Config file submodules/pypsa-earth/configs/bundle_config.yaml is extended by additional config specified via the command line.
Config file configs/config.main.yaml is extended by additional config specified via the command line.
FileNotFoundError in file /Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/submodules/pypsa-earth/Snakefile, line 36:
[Errno 2] No such file or directory: 'config.default.yaml'
  File "/Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/Snakefile", line 47, in <module>
  File "/Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/submodules/pypsa-earth/Snakefile", line 36, in <module>
  File "/Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/submodules/pypsa-earth/scripts/_helpers.py", line 59, in check_config_version

Looks like it was not able to find the config.default.yaml file. By the way, I ran the code in the main working directory.

[yerbol-akhmetov]

Hi @yerbol-akhmetov, I tested it on a fresh repo. I got this error

Config file submodules/pypsa-earth/config.default.yaml is extended by additional config specified via the command line.
Config file submodules/pypsa-earth/configs/bundle_config.yaml is extended by additional config specified via the command line.
Config file configs/config.main.yaml is extended by additional config specified via the command line.
FileNotFoundError in file /Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/submodules/pypsa-earth/Snakefile, line 36:
[Errno 2] No such file or directory: 'config.default.yaml'
  File "/Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/Snakefile", line 47, in <module>
  File "/Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/submodules/pypsa-earth/Snakefile", line 36, in <module>
  File "/Volumes/MBP Ext Disk/Yerbol/efuels-supply-potentials/submodules/pypsa-earth/scripts/_helpers.py", line 59, in check_config_version

Looks like it was not able to find the config.default.yaml file. By the way, I ran the code in the main working directory.

@GbotemiB. Yes it will not work with main. Please make the following changes to the files. I have created a PR to upstream already. But for now try locally:

1. pypsa-earth/scripts/_helpers.py
   
2. pypsa-earth/scripts/add_extra_components.py
   

[GbotemiB]

@yerbol-akhmetov, I have tested the code and it works fine. Although I had to comment so other rules since it was taking time.

[yerbol-akhmetov]

@yerbol-akhmetov, I have tested the code and it works fine. Although I had to comment so other rules since it was taking time.

Which rules were tested? In my case, I run everything except build_osm_network, build_renewable_profiles.

[GbotemiB]

@yerbol-akhmetov, I have tested the code and it works fine. Although I had to comment so other rules since it was taking time.

Which rules were tested? In my case, I run everything except build_osm_network, build_renewable_profiles.

I only test the add electricity rule. Other files were present which was why i skipped it, but if you want testing with just removing build_osm_network, build_renewable_profiles, I can run the model again.

[yerbol-akhmetov]

@yerbol-akhmetov, I have tested the code and it works fine. Although I had to comment so other rules since it was taking time.

Which rules were tested? In my case, I run everything except build_osm_network, build_renewable_profiles.

I only test the add electricity rule. Other files were present which was why i skipped it, but if you want testing with just removing build_osm_network, build_renewable_profiles, I can run the model again.

Can you please fully solve it? Skip build_osm_network, build_renewable_profiles rules. I used to have some issue if simplify_network rule, but which seemed resolved. I want to make sure.

[yerbol-akhmetov]

@GbotemiB, if everything works, then we can merge it I think. For now, we do not have custom scripts that will use into pypsa-earth workflow except validate_all rule. Note, it will trigger solving if no solved network is found, what I think is cool.

[GbotemiB]

@yerbol-akhmetov I ran the model again, and got this error.

/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning:

The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 

Can't find the repo path.
ERROR:_helpers:An error happened in module '/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py', function '_open': [Errno 2] No such file or directory: '/submodules/pypsa-earth/logs/US_2021/simplify_network/elec_s.log'
Traceback (most recent call last):
  File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/.snakemake/scripts/tmps2t3jmow.simplify_network.py", line 972, in <module>
    configure_logging(snakemake)
  File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/submodules/pypsa-earth/scripts/_helpers.py", line 222, in configure_logging
    logging.FileHandler(logfile),
  File "/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py", line 1169, in __init__
    StreamHandler.__init__(self, self._open())
  File "/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py", line 1201, in _open
    return open_func(self.baseFilename, self.mode,
FileNotFoundError: [Errno 2] No such file or directory: '/submodules/pypsa-earth/logs/US_2021/simplify_network/elec_s.log'

Is this similar to the error you encountered?

[yerbol-akhmetov]

@yerbol-akhmetov I ran the model again, and got this error.

/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning:

The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 

Can't find the repo path.
ERROR:_helpers:An error happened in module '/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py', function '_open': [Errno 2] No such file or directory: '/submodules/pypsa-earth/logs/US_2021/simplify_network/elec_s.log'
Traceback (most recent call last):
  File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/.snakemake/scripts/tmps2t3jmow.simplify_network.py", line 972, in <module>
    configure_logging(snakemake)
  File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/submodules/pypsa-earth/scripts/_helpers.py", line 222, in configure_logging
    logging.FileHandler(logfile),
  File "/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py", line 1169, in __init__
    StreamHandler.__init__(self, self._open())
  File "/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py", line 1201, in _open
    return open_func(self.baseFilename, self.mode,
FileNotFoundError: [Errno 2] No such file or directory: '/submodules/pypsa-earth/logs/US_2021/simplify_network/elec_s.log'

Is this similar to the error you encountered?

Yes, I used to encounter such error. But now I don't, I do not quite know the reason. It needs to be investigated. I run base thought pypsa-earth repo. Then, I run through outside snakemake. Maybe path the log storing location was missing. Can you also try to find the reason? I do not encounter such error now, but I will try to replicate somehow.

[GbotemiB]

@yerbol-akhmetov I ran the model again, and got this error.

/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning:

The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 

Can't find the repo path.
ERROR:_helpers:An error happened in module '/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py', function '_open': [Errno 2] No such file or directory: '/submodules/pypsa-earth/logs/US_2021/simplify_network/elec_s.log'
Traceback (most recent call last):
  File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/.snakemake/scripts/tmps2t3jmow.simplify_network.py", line 972, in <module>
    configure_logging(snakemake)
  File "/Users/gbotemi/Documents/code/PYPSA/US/efuels-supply-potentials/submodules/pypsa-earth/scripts/_helpers.py", line 222, in configure_logging
    logging.FileHandler(logfile),
  File "/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py", line 1169, in __init__
    StreamHandler.__init__(self, self._open())
  File "/Users/gbotemi/miniconda3/envs/pypsa-earth/lib/python3.10/logging/__init__.py", line 1201, in _open
    return open_func(self.baseFilename, self.mode,
FileNotFoundError: [Errno 2] No such file or directory: '/submodules/pypsa-earth/logs/US_2021/simplify_network/elec_s.log'

Is this similar to the error you encountered?

Yes, I used to encounter such error. But now I don't, I do not quite know the reason. It needs to be investigated. I run base thought pypsa-earth repo. Then, I run through outside snakemake. Maybe path the log storing location was missing. Can you also try to find the reason? I do not encounter such error now, but I will try to replicate somehow.

Alright, I will try to investigate the issue.