Merge pull request #318 from ImperialCollegeLondon/extended-demo

Extended demo

.github/workflows/ci_template.yml

@@ -30,7 +30,7 @@ jobs:
         python-version: ${{ matrix.python-version }}
 
     - name: Install dependencies
-      run: pip install .[dev]
+      run: pip install -e .[dev]
 
     - name: Run tests
       run: pytest
@@ -41,4 +41,4 @@ jobs:
       with:
         fail_ci_if_error: true
         token: ${{ secrets.codecov_token }}
-        verbose: true
+        verbose: true

docs/config_guide.md

@@ -52,7 +52,7 @@ parameter_overrides:
 Note that we must provide the parameter category for the parameter that we are
 changing (`subcatchment_derivation` above).
 
-As our SWMManywhere paper [link preprint] demonstrates, you can capture an enormously wide range of UDM behaviours through changing parameters. However, if your system is particularly unusual, or you are testing out new behaviours then you may need to adopt a more elaborate approach.
+As our SWMManywhere paper [link preprint](https://doi.org/10.31223/X5GT5X) demonstrates, you can capture an enormously wide range of UDM behaviours through changing parameters. However, if your system is particularly unusual, or you are testing out new behaviours then you may need to adopt a more elaborate approach.
 
 ### Customise `graphfcns`
 

docs/custom.css

@@ -0,0 +1,15 @@
+:root {
+    --md-primary-fg-color: #00BCD4;
+    --md-accent-fg-color: #00BCD4;
+  }
+
+.jupyter-wrapper .jp-OutputArea {
+    max-height: 700px;
+    overflow-y: auto;
+    border: 1px solid #ddd;
+    padding: 5px;
+}
+
+.jupyter-wrapper .jp-OutputArea-child {
+    display: block; /* Make sure child elements are block-level to avoid flex wrapping issues */
+}

docs/index.md

@@ -9,6 +9,7 @@ derive a synthetic urban drainage network anywhere in the world.
 - [Quickstart](quickstart.md)
 - Guides:
     - [Configuration file](config_guide.md)
+    - [Extended demo](./notebooks/extended_demo.py)
     - [Graph functions](graphfcns_guide.md)
     - [Metrics guide](metrics_guide.md)
 - [Contributing](CONTRIBUTING.md)

docs/notebooks/extended_demo.py

@@ -1 +1,190 @@
-"""Force a strange error."""
+# %% [markdown]
+# # Extended Demo
+# Note - this script can also be opened in interactive Python if you wanted to
+# play around. On the GitHub it is in [docs/notebooks]. To run this on your
+# local machine, you will need to install the optional dependencies for `doc`:
+#
+# `pip install swmmanywhere[doc]`
+#
+# %% [markdown]
+# ## Introduction
+# This script demonstrates a simple use case of `swmmanywhere`, building on the
+# [quickstart](https://imperialcollegelondon.github.io/SWMManywhere/quickstart/)
+# example, but including plotting and alterations.
+#
+# Since this is a notebook, we will define [`config`](https://imperialcollegelondon.github.io/SWMManywhere/config_guide/)
+# as a dictionary rather than a `yaml` file, but the same principles apply.
+#
+# ## Initial run
+#
+# Here we will run the [quickstart](https://imperialcollegelondon.github.io/SWMManywhere/config_guide/)
+# configuration, keeping everything in a temporary directory.
+# %%
+# Imports
+from __future__ import annotations
+
+import tempfile
+from pathlib import Path
+
+import folium
+import geopandas as gpd
+import pandas as pd
+
+from swmmanywhere.logging import set_verbose
+from swmmanywhere.swmmanywhere import swmmanywhere
+from swmmanywhere.utilities import plot_map
+
+# Create temporary directory
+temp_dir = tempfile.TemporaryDirectory()
+base_dir = Path(temp_dir.name)
+
+# Define minimum viable config (with shorter duration so better inspect results)
+config = {
+    "base_dir": base_dir,
+    "project": "my_first_swmm",
+    "bbox": [1.52740, 42.50524, 1.54273, 42.51259],
+    "run_settings": {"duration": 3600},
+}
+
+# Run SWMManywhere
+outputs = swmmanywhere(config)
+
+# Verify the output
+model_file = outputs[0]
+if not model_file.exists():
+    raise FileNotFoundError(f"Model file not created: {model_file}")
+
+
+# %% [markdown]
+# ## Plotting output
+#
+# If you do not have a real UDM, the majority of your interpretation will be
+# around the synthesised `nodes` and `edges`. These are
+# created in the same directory as the `model_file`. Let's have a look at them.
+# Note that the `outfall` that each node drains to is specified in the `outfall`
+# attribute, we will plot these in red and other nodes in black with the built-
+# in `swmmanywhere.utilities.plot_map` function.
+# %%
+# Create a folium map and add the nodes and edges
+plot_map(model_file.parent)
+
+# %% [markdown]
+# OK, it's done something! Though perhaps we're not super satisfied with the output.
+#
+# ## Customising outputs
+#
+# Some things stick out on first glance,
+#
+# - Probably we do not need pipes in the hills to the South, these seem to be along
+# pedestrian routes, which can be adjusted with the `allowable_networks` parameter.
+# - We will also remove any types under the `omit_edges` entry, here you can specify
+# to not allow pipes to cross bridges, tunnels, motorways, etc., however, this is
+# such a small area we probably don't want to restrict things so much.
+# - We have far too few outfalls, it seems implausible that so many riverside streets
+# would not have outfalls. Furthermore, there are points that are quite far from the
+# river that have been assigned as outfalls. We can reduce the `river_buffer_distance`
+# to make nodes nearer the river more likely to be outfalls, but also reduce the
+# `outfall_length` distance parameter to enable `swmmanywhere` to more freely select
+# outfalls that are adjacent to the river.
+#
+# Let's just demonstrate that using the
+# [`parameter_overrides` functionality](https://imperialcollegelondon.github.io/SWMManywhere/config_guide/#changing-parameters).
+#
+# %%
+config["parameter_overrides"] = {
+    "topology_derivation": {
+        "allowable_networks": ["drive"],
+        "omit_edges": ["bridge"],
+    },
+    "outfall_derivation": {
+        "outfall_length": 5,
+        "river_buffer_distance": 30,
+    },
+}
+outputs = swmmanywhere(config)
+plot_map(outputs[0].parent)
+
+# %% [markdown]
+# OK that clearly helped, although we have appear to have stranded pipes (e.g., along
+# *Carrer dels Canals* in North West), presumably due to some mistake in the
+# OSM specifying that it
+# is connected via a pedestrian route. We won't remedy this in the tutorial, but you can
+# manually provide your
+# [`starting_graph`](https://imperialcollegelondon.github.io/SWMManywhere/config_guide/#change-starting_graph)
+# via the configuration file to address such mistakes.
+#
+# More importantly we can see some distinctive unconnected network in the South West.
+# What is going on there? To explain this we will have to turn on verbosity to print the
+# intermediate files used in model derivation.
+#
+# To do this with a command line call we simply add the flag `--verbose=True`.
+# Though in code we will have to use `set_verbose` from the `logging` module.
+
+# %%
+# Make verbose
+set_verbose(True)  # Set verbosity
+
+# Run again
+outputs = swmmanywhere(config)
+model_dir = outputs[0].parent
+m = plot_map(model_dir)
+
+# %% [markdown]
+# That's a lot of information! However, the reason we are currently interested
+# in this is because the files associated with
+# each workflow step are saved when `verbose=True`.
+#
+# We will load a file called `subbasins` and add it to the map.
+
+# %%
+subbasins = gpd.read_file(model_dir / "subbasins.geoparquet")
+folium.GeoJson(subbasins, fill_opacity=0, color="blue", weight=2).add_to(m)
+m
+
+# %% [markdown]
+# Although this can be customised, the default behaviour of `swmmanywhere` is to not
+# allow edges to cross hydrological subbasins. It is now super clear why these
+# unconnected  networks have appeared, and are ultimately due to the underlying DEM.
+# If you did desperately care about these streets, then you should probably
+# widen your bounding box.
+#
+# ## Plotting results
+#
+# Because we have run the model with `verbose=True` we will also see that a new
+# `results` file has appeared, which contains all of the simulation results from SWMM.
+
+# %%
+df = pd.read_parquet(model_dir / "results.parquet")
+df.head()
+
+# %% [markdown]
+# `results` contains all simulation results in long format, with `flooding` at
+# nodes and `flow` at edges. We will plot a random `flow`.
+
+# %%
+flows = df.loc[df.variable == "flow"]
+flows.loc[flows.id == flows.iloc[0].id].set_index("date").value.plot(
+    ylabel="flow (l/s)"
+)
+
+
+# %% [markdown]
+# If `results` are present in the `model_dir`, `plot_map` will make clickable
+# elements to view plots,
+# now you can inspect your results in a much more elegant way than the SWMM GUI.
+# Just click a node or link to view the flooding or flow timeseries!
+
+
+# %%
+plot_map(model_dir)
+
+# %% [markdown]
+# If we explore around, clicking on edges, we can see that flows are often
+# looking sensible, though we can definitely some areas that have been hampered
+# by our starting street graph (e.g., in the Western portion of *Carrer del Sant Andreu*
+# in North West we can see negative flows meaning the direction is different from
+# what the topology derivation assumed flow would be going in!).
+# The first suggestion here would be to examine the starting graph,
+# however, if you want to
+# make more sophisticated customisations then your probably want to learn about
+# [graph functions](https://imperialcollegelondon.github.io/SWMManywhere/graphfcns_guide/).

mkdocs.yml

@@ -2,20 +2,17 @@ site_name: SWMManywhere docs
 
 theme:
   name: "material"
-  palette:
-    primary: 'cyan'
+
   icon:
     repo: fontawesome/brands/github
   features:
     - content.code.copy
-
-extra_css:
-  - stylesheets/extra.css
+extra_css: [custom.css]
 
 plugins:
   - mkdocstrings
   - mkdocs-jupyter: 
-      execute: false
+      execute: true
   - search
   - include-markdown
 
@@ -44,6 +41,7 @@ nav:
   - Quickstart: quickstart.md
   - Guides:
     - Configuration guide: config_guide.md
+    - Extended demo: ./notebooks/extended_demo.py
     - Graph functions: graphfcns_guide.md
     - Metrics guide: metrics_guide.md
   - Contributing: CONTRIBUTING.md

pyproject.toml

@@ -36,11 +36,13 @@ dynamic = [
 dependencies = [
   "cdsapi",
   "cytoolz",
+  "folium",
   "geopandas>=1",
   "geopy",
   "joblib",
   "jsonschema",
   "loguru",
+  "matplotlib",
   "netcdf4",
   "networkx>=3",
   "numpy>=2",
@@ -99,6 +101,7 @@ version-file = "_version.py"
 
 [tool.ruff]
 lint.select = [ "D", "E", "F", "I" ]                                   # pydocstyle, pycodestyle, Pyflakes, isort
+lint.per-file-ignores."docs/notebooks/*" = [ "D100" ]
 lint.per-file-ignores."src/netcomp/*" = [ "D", "F" ]                   # Ignore all checks for netcomp
 lint.per-file-ignores."tests/*" = [ "D100", "D104" ]
 lint.isort.known-first-party = [ "swmmanywhere", "netcomp" ]
@@ -107,7 +110,7 @@ lint.pydocstyle.convention = "google"
 
 [tool.codespell]
 skip = "src/swmmanywhere/defs/iso_converter.yml,*.inp"
-ignore-words-list = "gage,gages"
+ignore-words-list = "gage,gages,Carrer"
 
 [tool.pytest.ini_options]
 addopts = "-v --cov=src/swmmanywhere --cov-report=xml --doctest-modules --ignore=src/swmmanywhere/logging.py"

src/swmmanywhere/graphfcns/network_cleaning_graphfcns.py

@@ -77,7 +77,7 @@ def __call__(self, G: nx.Graph, **kwargs) -> nx.Graph:
         # Set the attribute (weight) used to determine which parallel edge to
         # retain. Could make this a parameter in parameters.py if needed.
         weight = "length"
-        graph = ox.get_digraph(G)
+        graph = ox.convert.to_digraph(G)
         _, _, attr_list = next(iter(graph.edges(data=True)))  # type: ignore
         attr_list = cast("dict[str, Any]", attr_list)
         if weight not in attr_list:

src/swmmanywhere/post_processing.py

@@ -34,8 +34,6 @@ def synthetic_write(addresses: FilePaths):
     Args:
         addresses (FilePaths): A dictionary of file paths.
     """
-    # TODO these node/edge names are probably not good or extendible defulats
-    # revisit once overall software architecture is more clear.
     nodes = gpd.read_file(addresses.model_paths.nodes)
     edges = gpd.read_file(addresses.model_paths.edges)
 

src/swmmanywhere/swmmanywhere.py

@@ -105,6 +105,13 @@ def swmmanywhere(config: dict) -> tuple[Path, dict | None]:
             logger.info(f"Setting {category} {key} to {val}")
             setattr(params[category], key, val)
 
+    # If `allowable_networks` has been changed, force a redownload of street graph.
+    if "allowable_networks" in config.get("parameter_overrides", {}).get(
+        "topology_derivation", {}
+    ):
+        logger.info("Allowable networks have been changed, removing old street graph.")
+        addresses.bbox_paths.street.unlink(missing_ok=True)
+
     # Run downloads
     logger.info("Running downloads.")
     preprocessing.run_downloads(