Move PyPSA dependency back to the main repository #538

ekatef · 2022-12-15T16:48:43Z

Changes proposed in this Pull Request

After merging PyPSA/PyPSA#517 we may git rid on the fork dependency

Checklist

I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

davide-f

Awesome Katia! :D
Rather than installing with pip, can we do it with conda?
Could you change that for pypsa and powerplantmatching as well?
Thanks :)

ekatef · 2022-12-18T16:21:13Z

Agree that it would be better to use conda everywhere. Have tested environment versions where both or one of pypsa and powerplantmatching are being installed with conda instead of pip.

Such change for powerplantmatching leads to an error TypeError: duke() got an unexpected keyword argument 'config_update' (the full listing attached bellow). That is the case even despite powerplantmatching itself is 0.5.4 whenever installed with pip or with conda. The only difference in the packages ppm depends on is entsoe-py which is 0.5.9 if installed with pip and 0.5.8 if installed with conda.

For pypsa using conda installation instead of pip leads to a version downgrade from 0.21.3 to 0.20.1 which means in particular that we loss Network.statistics method.

@davide-f would be very grateful for your suggestions on how to handle this environment zoo properly :)

An environment-related powerplantmatching error looks as follows:

INFO:snakemake.logging:[Sun Dec 18 17:49:17 2022]
rule build_powerplants:
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log
    jobid: 13
    reason: Missing output files: resources/powerplants.csv; Input files updated by another job: resources/shapes/gadm_shapes.geojson, resources/osm/clean/africa_all_generators.csv, networks/base.nc
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=500, mem_mib=477
INFO:snakemake.logging:rule build_powerplants:
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log
    jobid: 13
    reason: Missing output files: resources/powerplants.csv; Input files updated by another job: resources/shapes/gadm_shapes.geojson, resources/osm/clean/africa_all_generators.csv, networks/base.nc
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=500, mem_mib=477

INFO:snakemake.logging:
This is the repository path:  ~/pypsa-earth
Had to go 0 folder(s) up.
INFO:pypsa.io:Imported network base.nc has buses, lines, transformers
INFO:powerplantmatching.collection:Create combined dataset for BEYONDCOAL, ENTSOE, GEO, GPD, JRC, OPSD, WIKIPEDIA
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'ENTSOE'.
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GEO'.
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'.
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'JRC'.
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.
  where = df.astype(str).apply(func).any(1)
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'OPSD'.
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'WIKIPEDIA'.
INFO:powerplantmatching.matching:Comparing data sources `BEYONDCOAL` and `ENTSOE`
Traceback (most recent call last):
  File "~/pypsa-earth/.snakemake/scripts/tmpkaxp6jw7.build_powerplants.py", line 280, in <module>
    pm.powerplants(from_url=False, update=True, config_update=config)
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/collection.py", line 218, in matched_data
    matched = collect(matching_sources, config=config, **collection_kwargs)
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/collection.py", line 97, in collect
    matched = combine_multiple_datasets(dfs, datasets, config=config, **dukeargs)
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/matching.py", line 255, in combine_multiple_datasets
    crossmatches = link_multiple_datasets(datasets, labels, config=config, **dukeargs)
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/matching.py", line 203, in link_multiple_datasets
    all_matches = parmap(comp_dfs, mapargs)
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/utils.py", line 378, in parmap
    return list(map(f, arg_list))
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/matching.py", line 200, in comp_dfs
    return compare_two_datasets(dfs_lbs[:2], dfs_lbs[2:], config=config, **dukeargs)
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/matching.py", line 105, in compare_two_datasets
    links = pd.concat([country_link(dfs, c) for c in countries])
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/matching.py", line 105, in <listcomp>
    links = pd.concat([country_link(dfs, c) for c in countries])
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/site-packages/powerplantmatching/matching.py", line 97, in country_link
    return duke(
TypeError: duke() got an unexpected keyword argument 'config_update'
[Sun Dec 18 17:51:34 2022]
INFO:snakemake.logging:[Sun Dec 18 17:51:34 2022]
Error in rule build_powerplants:
    jobid: 13
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_powerplants:
    jobid: 13
    input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error details)

RuleException:
CalledProcessError in file ~/pypsa-earth/Snakefile, line 348:
Command 'set -euo pipefail;  /Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/bin/python3.10 ~/pypsa-earth/.snakemake/scripts/tmpkaxp6jw7.build_powerplants.py' returned non-zero exit status 1.
  File "~/pypsa-earth/Snakefile", line 348, in __rule_build_powerplants
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file ~/pypsa-earth/Snakefile, line 348:
Command 'set -euo pipefail;  /Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/bin/python3.10 ~/pypsa-earth/.snakemake/scripts/tmpkaxp6jw7.build_powerplants.py' returned non-zero exit status 1.
  File "~/pypsa-earth/Snakefile", line 348, in __rule_build_powerplants
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth_test3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
WARNING:snakemake.logging:Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-12-18T174640.981617.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2022-12-18T174640.981617.snakemake.log

davide-f · 2022-12-18T17:37:22Z

I think we need to add version constraints for pypsa and ppl to the latest versions.
Like ppl>=0.5.9 and similarly pypsa.

Have you tried that?

ekatef · 2022-12-18T19:19:43Z

Thank you! Have played a bit with versions constraints.

Adding a version constrain on PyPSA works perfectly.

As for ppm, specifying >=0.5.4 doesn't help to avoid the duke-error if using conda-forge. I suspect the reason is that the latest version of entsoe-py (0.5.9) is available via pip but isn't via conda-forge.

davide-f · 2022-12-20T21:36:39Z

The problem is that the latest updates of ppl that were in our fork have been merged into main but no ppl release includes them.
I've opened an issue PyPSA/powerplantmatching#92 in ppl to notify that.
@FabianHofmann do you think it may be possible to soon release a new minor release of ppl for conda?

FabianHofmann · 2022-12-21T10:41:33Z

hey @davide-f, I am on it. Expect it to be on pip today, conda latest tomorrow.

davide-f · 2022-12-21T11:03:26Z

hey @davide-f, I am on it. Expect it to be on pip today, conda latest tomorrow.

Thank you very much! Fantastic :D

ekatef · 2022-12-25T14:01:26Z

I had an error in build_renewable_profiels when testing with powerplantmatching 0.5.5. and additional restriction numpy<1.24 as in #542. It was the same ValueError in shape = int((top - bottom) / res), int((right - left) / res) as reported by @pz-max for Africa

pz-max · 2022-12-25T14:12:53Z

I had an error in build_renewable_profiels when testing with powerplantmatching 0.5.5. and additional restriction numpy<1.24 as in #542. It was the same ValueError in shape = int((top - bottom) / res), int((right - left) / res) as reported by @pz-max for Africa

Good to hear that I am not the only one experiencing this issue.
The error message @ekatef and I receive:
ValueError: cannot convert float NaN to integer

It's actually the same as described in this merged Atlite PR. I am not sure yet what is happening

Remove ppl by pip

davide-f · 2022-12-25T15:00:50Z

TheCI for Linux is now passing.
For sure it is an environment issue, as the complete CI Succeceds, I'd merge and we can create the environment from scratch

ekatef · 2022-12-25T15:45:47Z

Building environment from scratch using the updated specification has fixed the issue with renewable_profile_generation when tested locally.

I had a hope it could also resolve #537 but that is not the case yet :)

ekatef added 2 commits December 15, 2022 00:24

Update a name of a DC linetype

c1259bf

Replace local fork with the main repo for PyPSA dependency

c75b19f

ekatef mentioned this pull request Dec 15, 2022

Fix geometry issues #532

Closed

12 tasks

ekatef added 3 commits December 16, 2022 10:49

Restrict shapely version

bd2b44c

Fix shapely version

8ff88bd

Merge branch 'main' into change_pypsa_dependency

ad5b438

ekatef marked this pull request as ready for review December 17, 2022 18:49

Add a release note

56f38d4

davide-f self-requested a review December 17, 2022 23:28

davide-f approved these changes Dec 17, 2022

View reviewed changes

davide-f requested changes Dec 17, 2022

View reviewed changes

ekatef added 2 commits December 18, 2022 22:14

Change an installation channel for PyPSA

3463bf9

Add a comment to a powerplantmatching version

880443d

davide-f added 2 commits December 25, 2022 14:02

Update environment.yaml

c1afa58

Merge branch 'main' into change_pypsa_dependency

168a7b2

ekatef and others added 2 commits December 25, 2022 17:16

Merge remote-tracking branch 'origin/main' into change_pypsa_dependency

df42c9c

Update environment.yaml

32c46c8

Remove ppl by pip

davide-f approved these changes Dec 25, 2022

View reviewed changes

davide-f merged commit 738676b into pypsa-meets-earth:main Dec 25, 2022

pz-max mentioned this pull request Dec 27, 2022

Error in building the base_network, tutorial, release 0.61 PyPSA/pypsa-eur#455

Closed

ekatef deleted the change_pypsa_dependency branch November 14, 2023 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move PyPSA dependency back to the main repository #538

Move PyPSA dependency back to the main repository #538

ekatef commented Dec 15, 2022 •

edited

Loading

davide-f left a comment •

edited

Loading

ekatef commented Dec 18, 2022 •

edited

Loading

davide-f commented Dec 18, 2022

ekatef commented Dec 18, 2022

davide-f commented Dec 20, 2022

FabianHofmann commented Dec 21, 2022

davide-f commented Dec 21, 2022

ekatef commented Dec 25, 2022

pz-max commented Dec 25, 2022

davide-f commented Dec 25, 2022

ekatef commented Dec 25, 2022

Move PyPSA dependency back to the main repository #538

Move PyPSA dependency back to the main repository #538

Conversation

ekatef commented Dec 15, 2022 • edited Loading

Changes proposed in this Pull Request

Checklist

davide-f left a comment • edited Loading

Choose a reason for hiding this comment

ekatef commented Dec 18, 2022 • edited Loading

davide-f commented Dec 18, 2022

ekatef commented Dec 18, 2022

davide-f commented Dec 20, 2022

FabianHofmann commented Dec 21, 2022

davide-f commented Dec 21, 2022

ekatef commented Dec 25, 2022

pz-max commented Dec 25, 2022

davide-f commented Dec 25, 2022

ekatef commented Dec 25, 2022

ekatef commented Dec 15, 2022 •

edited

Loading

davide-f left a comment •

edited

Loading

ekatef commented Dec 18, 2022 •

edited

Loading