Powerplantmatching error for some countries when alternative clustering is on #537
Closed
2 tasks done
Labels
bug
Something isn't working
Milestone
Comments
ekatef
changed the title
|
@davide-f, yes it appears under the latest environment even if it was not updated but built from scratch |
|
@ekatef may you try to check if this problem is still present with the new environment with shapely fixed? |
|
@davide-f The problem still persists under the fixed environment and appears for some cases only. For example, the error is there for Namibia but not for Nigeria |
7 tasks
|
Heave re-tested of "CD" and "NA", and it appears that the error has been fixed. Great work on improving |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
mainbranch or the latest release. Please indicate.pypsa-earthenvironment. Update viaconda env update -f envs/environment.yaml.Describe the Bug
For some countries
build_powerplantsthrows an error when looking for the powerplants locations in the GADM geo-dataframe ifalternative_clustering: true. In particular, the error appears for "IN", "CD" and "NA" but doesn't for "KZ"Error Message
rule build_powerplants: input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson output: resources/powerplants.csv, resources/powerplants_osm2pm.csv log: logs/build_powerplants.log jobid: 13 reason: Missing output files: resources/powerplants.csv; Input files updated by another job: resources/shapes/gadm_shapes.geojson, networks/base.nc, resources/osm/clean/africa_all_generators.csv resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=500 INFO:snakemake.logging:rule build_powerplants: input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson output: resources/powerplants.csv, resources/powerplants_osm2pm.csv log: logs/build_powerplants.log jobid: 13 reason: Missing output files: resources/powerplants.csv; Input files updated by another job: resources/shapes/gadm_shapes.geojson, networks/base.nc, resources/osm/clean/africa_all_generators.csv resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=500 INFO:snakemake.logging: This is the repository path: ~/pypsa-earth Had to go 0 folder(s) up. INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers INFO:powerplantmatching.collection:Create combined dataset for GEO, GPD /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GEO'. /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'. INFO:powerplantmatching.matching:Comparing data sources `GEO` and `GPD` /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/cleaning.py:170: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only. where = df.astype(str).apply(func).any(1) INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GPD'. /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/powerplantmatching/heuristics.py:255: FutureWarning: In a future version, `df.iloc[:, i] = newvals` will attempt to set the values inplace instead of always setting a new array. To retain the old behavior, use either `df[df.columns[i]] = newvals` or, if columns are non-unique, `df.isetitem(i, newvals)` df.loc[:, "DateIn"] = df.DateIn.astype(int) Traceback (most recent call last): File "~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py", line 325, in <module> ppl["region_id"] = ppl[["lon", "lat", "Country"]].apply( File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/frame.py", line 9565, in apply return op.apply().__finalize__(self, method="apply") File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 746, in apply return self.apply_standard() File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 873, in apply_standard results, res_index = self.apply_series_generator() File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/apply.py", line 889, in apply_series_generator results[i] = self.f(v) File "~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py", line 326, in <lambda> lambda pp: locate_bus(pp[["lon", "lat"]], pp["Country"]), axis=1 File "~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py", line 323, in locate_bus return gdf_co[gdf_co.contains(point)]["GADM_ID"].item() File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/base.py", line 347, in item raise ValueError("can only convert an array of size 1 to a Python scalar") ValueError: can only convert an array of size 1 to a Python scalar [Thu Dec 15 01:26:00 2022] INFO:snakemake.logging:[Thu Dec 15 01:26:00 2022] Error in rule build_powerplants: jobid: 13 input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson output: resources/powerplants.csv, resources/powerplants_osm2pm.csv log: logs/build_powerplants.log (check log file(s) for error message) ERROR:snakemake.logging:Error in rule build_powerplants: jobid: 13 input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/africa_all_generators.csv, resources/shapes/gadm_shapes.geojson output: resources/powerplants.csv, resources/powerplants_osm2pm.csv log: logs/build_powerplants.log (check log file(s) for error message) RuleException: CalledProcessError in line 348 of ~/pypsa-earth/Snakefile: Command 'set -euo pipefail; /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/bin/python3.10 ~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py' returned non-zero exit status 1. File "~/pypsa-earth/Snakefile", line 348, in __rule_build_powerplants File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run ERROR:snakemake.logging:RuleException: CalledProcessError in line 348 of ~/pypsa-earth/Snakefile: Command 'set -euo pipefail; /Users/ekaterina/opt/miniconda3/envs/pypsa-earth/bin/python3.10 ~/pypsa-earth/.snakemake/scripts/tmp77e59b1q.build_powerplants.py' returned non-zero exit status 1. File "~/pypsa-earth/Snakefile", line 348, in __rule_build_powerplants File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run Removing output files of failed job build_powerplants since they might be corrupted: resources/powerplants_osm2pm.csv WARNING:snakemake.logging:Removing output files of failed job build_powerplants since they might be corrupted: resources/powerplants_osm2pm.csv Shutting down, this might take some time. WARNING:snakemake.logging:Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2022-12-15T012441.713972.snakemake.log WARNING:snakemake.logging:Complete log: .snakemake/log/2022-12-15T012441.713972.snakemake.logThe text was updated successfully, but these errors were encountered: