Fix geometry issues #532

ekatef · 2022-12-03T22:09:29Z

It has been found during workflow testing on data around the world that underlaying geographical shapes sometimes can be sometimes problematic, namely:

it looks like the GID_0 feature in the gadm file may sometimes have values which do not exactly correspond to the country name. As a result, theGADM_ID in the gadm_shapes.geojson in may have some unexpected values like "Z02.28_1" or "Z03.28_1" (taken from data for China), and leads in country_shapes.geojson file to zeros for the population value and "name": null. This problem may be reproduced by running build_shapes for China or for India and leads to numerous troubles along the workflow.
sometimes the country area is a multiply connected region (an area with holes) which is treated by shapely as invalid geometry. In particular, that is the case for Dahagram–Angarpota Bangladeshi enclave in India.
it may happen that Voronoi partition leads to empty polygons which are not filtered by dropna(), are being written as none geometry into "regions_onshore.geojson" and cause some troubles propagating along the workflow. In particular, that is the case for Kazakhstan shapes (OSM data on 4.12.2022, 10 clusters).

Changes proposed in this Pull Request

This pull request is intended to fix such regional-specific issues.

Checklist

for more information, see https://pre-commit.ci

ekatef · 2022-12-04T21:12:35Z

The GADM issue means that some GADM_ID may have non-standard GADM_ID, e.g. for India (part of gadm_shapes):

"GADM_ID": "Z07.3_1", "country": "not found", "pop": 0.0, 
"GADM_ID": "Z04.13_1", "country": "not found", "pop": 0.0,
"GADM_ID": "Z09.13_1", "country": "not found", "pop": 0.0,
"GADM_ID": "Z01.14_1", "country": "not found", "pop": 0.0,
"GADM_ID": "Z05.35_1", "country": "not found", "pop": 0.0,
"GADM_ID": "Z09.35_1", "country": "not found", "pop": 0.0,

A visual check gives the following picture (which actually doen't clarify the issue):

Regions with non-standard GADM_IDs are highlighted by violet.

ekatef · 2022-12-05T10:24:38Z

It appears that non-standard GADM_IDs cause troubles when being interpreted as country names (e.g. 'not found') and propagating into the cleaned osm datasets. E.g. that is currently the case for India while for China non-standard GADM_IDs seem to remain in the country_shape and gadm_shape only do not causing any harm downstream. Although, such a difference may be a random effect of a particular workflow realisation.

for more information, see https://pre-commit.ci

ekatef · 2022-12-05T10:30:40Z

A picture for non-standard GADM_IDs for China looks as follows

"GADM_ID": "Z02.28_1", "country": null, "pop": 0.0, "gdp": 13450274.0
"GADM_ID": "Z03.28_1", "country": null, "pop": 0.0, "gdp": 45227044.0
"GADM_ID": "Z03.29_1", "country": null, "pop": 0.0, "gdp": 5931376.0 
"GADM_ID": "Z08.29_1", "country": null, "pop": 0.0, "gdp": 3446499.0

…_geom_fixes

for more information, see https://pre-commit.ci

ekatef · 2022-12-05T20:10:22Z

It looks like a fix is needed in build_shapes to get rid of GADM_IDs non-valid as countries names. A current work-around with len(no_data_countries) > 0 works for build_osm_network but it's likely that non-relevant values of the country name will lead to the problems downstream. E.g. if non-standard names are kept into shapes files, cluster_networks gives AssertionError: The following countries have no load: ['not found'] for distribute_cluster: ['load'].

The most strait-forward solution would be to filter by GID_0 (aka country name) values with something like df_countries = df_countries.query("name in @countries"). However, it looks like these areas represent meaningful geographical regions and a more data-saving approach may be preferable.

…_geom_fixes

for more information, see https://pre-commit.ci

ekatef · 2022-12-08T17:38:37Z

In a current implementation the onshore shape consists of a some shapes each corresponding to a special administrative area (those ones notated with the non-standard GADM_IDs)

"geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 69.
"geometry": { "type": "Polygon", "coordinates": [ [ [ 75.0716094
"geometry": { "type": "Polygon", "coordinates": [ [ [ 78.6513452
"geometry": { "type": "Polygon", "coordinates": [ [ [ 80.0879363
"geometry": { "type": "Polygon", "coordinates": [ [ [ 94.1912460
"geometry": { "type": "Polygon", "coordinates": [ [ [ 78.9089126

That leads to some troubles along the workflow. In particular, build_bus_regions terminates with an error:

rule build_bus_regions:
    input: resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, networks/base.nc, resources/shapes/gadm_shapes.geojson
    output: resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    log: logs/build_bus_regions.log
    jobid: 0
    reason: Input files updated by another job: networks/base.nc
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=1000
INFO:snakemake.logging:rule build_bus_regions:
    input: resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, networks/base.nc, resources/shapes/gadm_shapes.geojson
    output: resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    log: logs/build_bus_regions.log
    jobid: 0
    reason: Input files updated by another job: networks/base.nc
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=1000

INFO:snakemake.logging:
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
Traceback (most recent call last):
  File "~pypsa-earth/.snakemake/scripts/tmp0qppxgve.build_bus_regions.py", line 221, in <module>
    onshore_geometry = custom_voronoi_partition_pts(
  File "~pypsa-earth/.snakemake/scripts/tmp0qppxgve.build_bus_regions.py", line 105, in custom_voronoi_partition_pts
    xmin = min(xmin, minx_o)
TypeError: '>' not supported between instances of 'numpy.ndarray' and 'str'
[Thu Dec  8 20:21:24 2022]
INFO:snakemake.logging:[Thu Dec  8 20:21:24 2022]
Error in rule build_bus_regions:
    jobid: 0
    output: resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    log: logs/build_bus_regions.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule build_bus_regions:
    jobid: 0
    output: resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    log: logs/build_bus_regions.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 279 of ~pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekaterina/opt/miniconda3/envs/pypsa-earth-pdfix/bin/python3.10 ~pypsa-earth/.snakemake/scripts/tmp0qppxgve.build_bus_regions.py' returned non-zero exit status 1.
  File "~pypsa-earth/Snakefile", line 279, in __rule_build_bus_regions
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth-pdfix/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 279 of ~pypsa-earth/Snakefile:
Command 'set -euo pipefail;  /Users/ekaterina/opt/miniconda3/envs/pypsa-earth-pdfix/bin/python3.10 ~pypsa-earth/.snakemake/scripts/tmp0qppxgve.build_bus_regions.py' returned non-zero exit status 1.
  File "~pypsa-earth/Snakefile", line 279, in __rule_build_bus_regions
  File "/Users/ekaterina/opt/miniconda3/envs/pypsa-earth-pdfix/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.

That issue is caused by a fact that bounds method returns a dataframe instead of a single boundary vector. That problem may be easily resolved by unifying outline shapes before applying Voronoi partition.

ekatef · 2022-12-08T17:58:05Z

It could be a good idea to unify all the polygons when generating onshore_shape. The problem is to merge all the geometries simultaneously keeping other features (like GID_0).

It's worth to investigate if the geometry consisting of different polygons would lead to any problems along the workflow apart of the mentioned one in build_bus_regions.

Apart of that, it may be worth to test the workflow on the countries which consist of different areas (e.g. Malaysia)

…_geom_fixes

ekatef · 2022-12-10T22:37:50Z

It looks like onshore_shape composed of different geometries doesn't lead to other problems along the workflow apart the known one in build_bus_regions which can be fixed by a direct application of unary_union() just before bound calculation.

However, it appears that for some reasons geometries were duplicated in offshore_shape. Currently offshore_shape consists of five identical geometries which needs to be investigated.

ekatef · 2022-12-14T22:30:24Z

The issue with duplicated offshore geometries was fixed for India.

However, during tests with alternative_clustering: true another error appeared (an error listing is bellow). However, this error persisted after changing the region, e.g. to ["CD"] or ["NA"] but doesn't appears for ["KZ"]. So, it looks like that is connected rather to some specific topology of the power grid (HVDC lines??) than to the geometry issues addressed by this PR. The issue #537 has been created to track this error

ekatef · 2022-12-15T16:56:18Z

There are some weird issues during renewable profiles generation:

one error appears during generation on a hydro profile: AttributeError: 'numpy.int64' object has no attribute 'intersects' when calculating indicator_matrix in inflow = correction_factor * func(capacity_factor=True, **resource) (it can be reproduced, e.g. for ["IN"] using add_to_snakefile: true for augmented_line_connection)
another error appears during generation of a solar profile TypeError: ufunc 'linestrings' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' when forming a linestring line = LineString([p, regions.loc[bus, ["x", "y"]]] (that is the case for "KZ" or "NG" even for runs from the main branch which previously worked perfectly)

That is quite weird and most likely linked to some environment issues. #538 attempts to fix it

ekatef · 2022-12-15T17:11:43Z

Update: in the CI run there is currently one of the errors listed above, namely AttributeError: 'numpy.int64' object has no attribute 'intersects'. The full listing:

INFO:snakemake.logging:
[Thu Dec 15 17:07:34 2022]
INFO:snakemake.logging:[Thu Dec 15 17:07:34 2022]
rule build_renewable_profiles:
    input: networks/base.nc, resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/africa-2013-era5-tutorial.nc
    output: resources/renewable_profiles/profile_hydro.nc
    log: logs/build_renewable_profile_hydro.log
    jobid: 20
    benchmark: benchmarks/build_renewable_profiles_hydro
    reason: Missing output files: resources/renewable_profiles/profile_hydro.nc; Input files updated by another job: networks/base.nc, resources/shapes/offshore_shapes.geojson, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, resources/powerplants.csv, data/hydro_capacities.csv, cutouts/africa-2013-era5-tutorial.nc, resources/shapes/country_shapes.geojson, data/gebco/GEBCO_2021_TID.nc, resources/bus_regions/regions_onshore.geojson, resources/natura.tiff
    wildcards: technology=hydro
    threads: 2
    resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
INFO:snakemake.logging:rule build_renewable_profiles:
    input: networks/base.nc, resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/africa-2013-era5-tutorial.nc
    output: resources/renewable_profiles/profile_hydro.nc
    log: logs/build_renewable_profile_hydro.log
    jobid: 20
    benchmark: benchmarks/build_renewable_profiles_hydro
    reason: Missing output files: resources/renewable_profiles/profile_hydro.nc; Input files updated by another job: networks/base.nc, resources/shapes/offshore_shapes.geojson, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, resources/powerplants.csv, data/hydro_capacities.csv, cutouts/africa-2013-era5-tutorial.nc, resources/shapes/country_shapes.geojson, data/gebco/GEBCO_2021_TID.nc, resources/bus_regions/regions_onshore.geojson, resources/natura.tiff
    wildcards: technology=hydro
    threads: 2
    resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074

INFO:snakemake.logging:
INFO:__main__:Hydro normalization mode hydro_capacities

Determine upstream basins per plant: 0it [00:00, ?it/s]
Determine upstream basins per plant: 3it [00:00, 201.72it/s]
Traceback (most recent call last):
  File "/home/runner/work/pypsa-earth/pypsa-earth/.snakemake/scripts/tmpndxdiqdd.build_renewable_profiles.py", line 455, in <module>
    inflow = correction_factor * func(capacity_factor=True, **resource)
  File "/usr/share/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/convert.py", line 805, in hydro
    matrix = cutout.indicatormatrix(basins.shapes)
  File "/usr/share/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/cutout.py", line 536, in indicatormatrix
    return compute_indicatormatrix(self.grid, shapes, self.crs, shapes_crs)
  File "/usr/share/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/atlite/gis.py", line 148, in compute_indicatormatrix
    if o.intersects(d):
AttributeError: 'numpy.int64' object has no attribute 'intersects'
[Thu Dec 15 17:07:40 2022]
INFO:snakemake.logging:[Thu Dec 15 17:07:40 2022]
Error in rule build_renewable_profiles:
    jobid: 20
    input: networks/base.nc, resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/africa-2013-era5-tutorial.nc
    output: resources/renewable_profiles/profile_hydro.nc
    log: logs/build_renewable_profile_hydro.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_renewable_profiles:
    jobid: 20
    input: networks/base.nc, resources/natura.tiff, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/gebco/GEBCO_2021_TID.nc, resources/shapes/country_shapes.geojson, resources/shapes/offshore_shapes.geojson, data/hydro_capacities.csv, data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/bus_regions/regions_onshore.geojson, cutouts/africa-2013-era5-tutorial.nc
    output: resources/renewable_profiles/profile_hydro.nc
    log: logs/build_renewable_profile_hydro.log (check log file(s) for error details)

RuleException:
CalledProcessError in file /home/runner/work/pypsa-earth/pypsa-earth/Snakefile, line 325:
Command 'set -euo pipefail;  /usr/share/miniconda3/envs/pypsa-earth/bin/python3.10 /home/runner/work/pypsa-earth/pypsa-earth/.snakemake/scripts/tmpndxdiqdd.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/home/runner/work/pypsa-earth/pypsa-earth/Snakefile", line 325, in __rule_build_renewable_profiles
  File "/usr/share/miniconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file /home/runner/work/pypsa-earth/pypsa-earth/Snakefile, line 325:
Command 'set -euo pipefail;  /usr/share/miniconda3/envs/pypsa-earth/bin/python3.10 /home/runner/work/pypsa-earth/pypsa-earth/.snakemake/scripts/tmpndxdiqdd.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/home/runner/work/pypsa-earth/pypsa-earth/Snakefile", line 325, in __rule_build_renewable_profiles
  File "/usr/share/miniconda3/envs/pypsa-earth/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-12-15T165417.221370.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2022-12-15T165417.221370.snakemake.log
Error: Process completed with exit code 1.

…_geom_fixes

for more information, see https://pre-commit.ci

…_geom_fixes

for more information, see https://pre-commit.ci

ekatef · 2023-01-09T22:06:41Z

@davide-f, thank you so much for the review and the discussion! My feeling is that we are close to finalise the first bug-fixing keeping advanced features for the second step.

The only point left is replacing a if else block with pandas.where(). Have to look into it in some more details: not sure if the replacing function has a vectorised version in Python.

If you don't have major objections regarding the introduced changes, I'll move the changes into a clean PR

…_geom_fixes

davide-f · 2023-01-22T17:59:39Z

@ekatef May we close this PR since we worked in the others?

ekatef · 2023-01-22T21:07:50Z

Closed to be finalised in a cleaned version

ekatef and others added 4 commits December 4, 2022 00:59

Add an emptiness check for geometry

d489e77

Add a comment to a check added

59a0e83

[pre-commit.ci] auto fixes from pre-commit.com hooks

9ddb8bf

for more information, see https://pre-commit.ci

Merge branch 'main' into country_geom_fixes

b15717a

ekatef and others added 2 commits December 5, 2022 13:26

Fix processing country names for a case of not relevant entries

8dd7ed6

[pre-commit.ci] auto fixes from pre-commit.com hooks

5438808

for more information, see https://pre-commit.ci

ekatef and others added 4 commits December 5, 2022 22:45

Fix geometry with holes

a5e5529

Improve comment

94c5df8

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

4b35011

…_geom_fixes

[pre-commit.ci] auto fixes from pre-commit.com hooks

21d167b

for more information, see https://pre-commit.ci

ekatef and others added 6 commits December 6, 2022 19:28

Add a function to restore a country code from the country name

5c94a02

Add an import

6b8f98f

Apply the code restoring

1bbbf08

Drop non-found country names

8841088

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

d42c6eb

…_geom_fixes

[pre-commit.ci] auto fixes from pre-commit.com hooks

05e954f

for more information, see https://pre-commit.ci

ekatef added 2 commits December 11, 2022 01:04

Unify outline before to calculate Voronoi patrition

23f5e7c

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

2f41dcf

…_geom_fixes

ekatef added 2 commits December 11, 2022 16:39

Fix offshore_shapes duplication

1357965

Merge branch 'pypsa-meets-earth:main' into country_geom_fixes

40c4a3a

ekatef and others added 19 commits January 9, 2023 00:09

Merge branch 'pypsa-meets-earth:main' into country_geom_fixes

1bf0048

docs(contributor): contrib-readme-action has updated readme

05345ef

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

076198d

…_geom_fixes

Improve structure

b7026e1

Wrap a function around building a gadm-layer dataframe

bc2436d

Remove duplication

04a4fb1

Move geometry validation inside a function

3606638

Add temporary comments

aced6fa

Fix variable name inside a function

1c3b962

Fix definition of variables

d6a49d4

Fix indent

d9f4e33

Keep all the columns

7d457e9

[pre-commit.ci] auto fixes from pre-commit.com hooks

4603e37

for more information, see https://pre-commit.ci

Merge remote-tracking branch 'upstream/main' into country_geom_fixes

daec9e8

Merge branch 'pypsa-meets-earth:main' into country_geom_fixes

d69e64a

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

b414a98

…_geom_fixes

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

21aba6b

…_geom_fixes

Remove duplication

a4da003

[pre-commit.ci] auto fixes from pre-commit.com hooks

3b25277

for more information, see https://pre-commit.ci

ekatef marked this pull request as ready for review January 9, 2023 21:55

ekatef added 3 commits January 11, 2023 01:04

Merge branch 'main' of https://github.com/ekatef/pypsa-earth

f314100

Merge branch 'main' into country_geom_fixes

495477d

Merge remote-tracking branch 'origin/country_geom_fixes' into country…

ee6f07a

…_geom_fixes

ekatef mentioned this pull request Jan 10, 2023

Fix geometry issues - clean version #563

Closed

7 tasks

davide-f closed this Jan 22, 2023

ekatef mentioned this pull request Jan 31, 2023

TopologyException in build_bus_regions for India #583

Closed

2 tasks

ekatef deleted the country_geom_fixes branch November 14, 2023 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix geometry issues #532

Fix geometry issues #532

ekatef commented Dec 3, 2022 •

edited

Loading

ekatef commented Dec 4, 2022 •

edited

Loading

ekatef commented Dec 5, 2022

ekatef commented Dec 5, 2022 •

edited

Loading

ekatef commented Dec 5, 2022

ekatef commented Dec 8, 2022

ekatef commented Dec 8, 2022

ekatef commented Dec 10, 2022

ekatef commented Dec 14, 2022

ekatef commented Dec 15, 2022

ekatef commented Dec 15, 2022

ekatef commented Jan 9, 2023

davide-f commented Jan 22, 2023

ekatef commented Jan 22, 2023

Fix geometry issues #532

Fix geometry issues #532

Conversation

ekatef commented Dec 3, 2022 • edited Loading

Changes proposed in this Pull Request

Checklist

ekatef commented Dec 4, 2022 • edited Loading

ekatef commented Dec 5, 2022

ekatef commented Dec 5, 2022 • edited Loading

ekatef commented Dec 5, 2022

ekatef commented Dec 8, 2022

ekatef commented Dec 8, 2022

ekatef commented Dec 10, 2022

ekatef commented Dec 14, 2022

ekatef commented Dec 15, 2022

ekatef commented Dec 15, 2022

ekatef commented Jan 9, 2023

davide-f commented Jan 22, 2023

ekatef commented Jan 22, 2023

ekatef commented Dec 3, 2022 •

edited

Loading

ekatef commented Dec 4, 2022 •

edited

Loading

ekatef commented Dec 5, 2022 •

edited

Loading