updating tipping points wrt main (#479)

* take most of the env creation script from frontend and put it in the … (#467)

* take most of the env creation script from frontend and put it in the backend

* move env_docs.yml to the docs dir

* update readme

* added temporary event name for plotting rainfall and river (#470)

* Move env creation to backend (#468)

* take most of the env creation script from frontend and put it in the backend

* move env_docs.yml to the docs dir

* update readme

* fix bug in env creation

* black

* pre-commit version and fiat adapter test not skipped anymore (#463)

* updated pre-commit version

* updated svn repo so this test can be run now

* return periods test is not skipped anymore with fiat_toolbox updates

* changed cht_observation to git repo install

* chore: close all loggers & handlers in database fixture setup and teardown

* fix: Output more info if incompatible river coordinates in SFINCS adapter and sitetoml

* add reset() to dbs_controller.py

* reset database in conftest.py

* black

* fix versions of formatting tools

---------

Co-authored-by: LuukBlom <[email protected]>
Co-authored-by: LuukBlom <[email protected]>

* Add self check to hazard run check (#471)

* Add self check to hazard run check

* Fix black

* Updates in site.toml to clear not-used attributes and update some vairables (#462)

* removed unused attributes from site toml

* small update in lint

* made observation points non mandatory

* changed site attribute obs_station to tide_gauge and removed unused attributes

* changed the flooding_threshold attribute to belong to a flood_frequency variable

* added default value for flood frequency flooding threshold

* when no river is present template discharge is used for now

* added source option to tide_gauge site attribute

* small correction in docstring

* changed slr.scenarios in site.toml and made them not mandatory

* offshore model and cyclone tracks not mandatory anymore

* small updates

* naming change

* make sure that the workflow does not break when there is nothing to plot

* todo on api for aggregation

* revert rename (#474)

* Fix logging and tests (#475)

* fix api output fixture and tests

* return -> yield in all fixtures

* return -> yield in all fixtures

* fix scenario.run() tests

* add logging class and implement in codebase

* bugfix logger contextmanager

* improve test logging

* fix bug in create_roads

* Add attribute check to adapter logger

* Config changes (#476)

* database path updates

* Update config getters to return Paths where needed

* Update lint.yml

* Update lint.yml

---------

Co-authored-by: GundulaW <[email protected]>

* Split mandatory and optional metrics (#477)

---------

Co-authored-by: LuukBlom <[email protected]>
Co-authored-by: Panos Athanasiou <[email protected]>
Co-authored-by: LuukBlom <[email protected]>
Co-authored-by: dladrichem <[email protected]>
Co-authored-by: Daley Adrichem <[email protected]>
Co-authored-by: GundulaW <[email protected]>

.github/workflows/docs.yml

@@ -18,8 +18,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: ["ubuntu-latest" ] 
-        python-version: ["3.11"] 
+        os: ["ubuntu-latest" ]
+        python-version: ["3.11"]
         include:
           - os: ubuntu-latest
             label: linux-64
@@ -43,7 +43,7 @@ jobs:
       - name: Setup env
         run: |
           pip install tomli
-          mamba env create --file=environment_docs.yml
+          mamba env create --file=docs/environment_docs.yml
 
       - name: Setup quarto
         run: |
@@ -54,7 +54,7 @@ jobs:
           chmod +x tmp/quarto.deb
           sudo gdebi -n tmp/quarto.deb
 
-#      - name: Setup api -double check 
+#      - name: Setup api -double check
 #        run: |
 #          export PATH=/usr/share/miniconda3/bin:$PATH
 #          mamba run -n floodadapt_docs python -m pip install tomli-w

.github/workflows/lint.yml

@@ -11,10 +11,10 @@ jobs:
           python-version: "3.10"
       - name: Install dependencies
         run: |
-          pip install ruff
+          pip install ruff==0.4.8 # Make sure these are the same as the versions in pyproject.toml
       # Include `--format=github` to enable automatic inline annotations.
       - name: Run Ruff
-        run: ruff check .
+        run: ruff check . --fix
   black:
     runs-on: ubuntu-latest
     steps:
@@ -25,7 +25,7 @@ jobs:
           python-version: "3.10"
       - name: Install dependencies
         run: |
-          pip install black
+          pip install black==24.1.1 # Make sure these are the same as the versions in pyproject.toml
       - name: Check black version
         run: black --version
       - name: Run black

.pre-commit-config.yaml

@@ -9,6 +9,7 @@ repos:
     -   id: check-toml
     -   id: check-merge-conflict
     -   id: check-added-large-files
+        exclude: '^environment/'
     -   id: check-merge-conflict
     -   id: trailing-whitespace
 -   repo: https://github.com/psf/black
@@ -21,6 +22,6 @@ repos:
     -   id: ruff
         args: [--fix, --exit-non-zero-on-fix]
 -   repo: https://github.com/crate-ci/typos
-    rev: v1.22.0
+    rev: v1.22.7
     hooks:
     -   id: typos

README.md

@@ -1,95 +1,127 @@
 # FloodAdapt
 FloodAdapt is a decision-support tool that seeks to advance and accelerate flooding-related adaptation planning. It brings rapid, physics-based compound flood and detailed impact modelling into an easy-to-use system, allowing non-expert end-users to evaluate a wide variety of compound events, future conditions, and adaptation options in minutes.  FloodAdapt serves as a connector between scientific advances and practitioner needs, improving and increasing the uptake and impact of adaptation research and development.
 
-To make decisions on flood adaptation, communities need to understand how climate and socio-economic changes will affect flood risk and the risk-reduction potential of various adaptation options. This type of information is usually costly to acquire, and models are often too slow and labor-intensive to evaluate all the scenarios required to understand the impacts and effectiveness of potential adaptation decisions. FloodAdapt addresses this by making rapid, physics-based compound flood modeling and detailed impact modeling accessible to non-expert end-users, allowing them to evaluate a wide variety of compound events, future conditions, and adaptation options in minutes.  
+To make decisions on flood adaptation, communities need to understand how climate and socio-economic changes will affect flood risk and the risk-reduction potential of various adaptation options. This type of information is usually costly to acquire, and models are often too slow and labor-intensive to evaluate all the scenarios required to understand the impacts and effectiveness of potential adaptation decisions. FloodAdapt addresses this by making rapid, physics-based compound flood modeling and detailed impact modeling accessible to non-expert end-users, allowing them to evaluate a wide variety of compound events, future conditions, and adaptation options in minutes.
 
 FloodAdapt was developed as a rapid planning tool with a straightforward graphical user interface for scenario generation, simulation, and visualization of spatial flooding and flooding impacts. Decision-making needs at the community level were central to the design of FloodAdapt. Users can answer planning questions like: “How will potential adaptation options reduce flood impacts?”, “How will those options perform for different types of events, like hurricanes, king tides, or heavy rainfall?”, “Which neighborhoods will benefit most?”, “How will those options hold up in the future?”
 
-Users specify what-if scenarios composed of historic or synthetic weather events, climate or socio-economic future projections, and adaptation measures. 
+Users specify what-if scenarios composed of historic or synthetic weather events, climate or socio-economic future projections, and adaptation measures.
 The backend of FloodAdapt leverages the open-source, state-of-the-art process-based compound flood model SFINCS (https://github.com/Deltares/SFINCS) that can accurately predict compound flooding due to surge, rainfall, and river discharge, at a fraction of the computation time typically required by physics-based models. The damage model included in FloodAdapt is the Deltares-developed flood impact assessment tool Delft-FIAT (https://github.com/Deltares/Delft-FIAT). It calculates the flood damages to individual buildings and roads, and – when social vulnerability data is available – aggregates these damages over vulnerability classes.
 
-FloodAdapt can greatly support adaptation planning by allowing users to explore many scenarios. It can be used to evaluate flooding and impacts due to compound weather events, like hurricanes, king tides, and rainfall events. Users can evaluate flooding, impacts, and risk considering user-specified projections of sea level rise, precipitation increase, storm frequency increase, population growth, and economic growth. Users can also test out adaptation options, like sea walls, levees, pumps, home elevations, buyouts and floodproofing. 
+FloodAdapt can greatly support adaptation planning by allowing users to explore many scenarios. It can be used to evaluate flooding and impacts due to compound weather events, like hurricanes, king tides, and rainfall events. Users can evaluate flooding, impacts, and risk considering user-specified projections of sea level rise, precipitation increase, storm frequency increase, population growth, and economic growth. Users can also test out adaptation options, like sea walls, levees, pumps, home elevations, buyouts and floodproofing.
 
 Recent developments of the decision-support system include (1) simplifying and partially automating the setup of the SFINCS and Delft-FIAT models, (2) improving the user experience, (3) better supporting adaptation planning with improvements like metrics tables, infographics, better visualizations in the user interface, adding in additional adaptation options to evaluate, and calculating benefits of adaptation options, and (4) incorporating social vulnerability and equity into the evaluation of adaptation options to support equitable adaptation planning.
 
-FloodAdapt is currently in an intensive development stage. Independent usage of the repository will be challenging prior to end-of-year 2024. FloodAdapt documentation will be expanded on throughout 2024.  
-
-# Joining the team
-
-Whenever someone joins the team, a few steps needs to be taken before you can access the full list of repositories that we use. First of all, you need to be part of the Deltares and Deltares-research organisations. This access needs to be granted by one of the enterprise owners. When this access is granted, you can request to join the FloodAdaptAdmins team if you are a part of the core team, the FloodAdaptDevelopers if you'll sporadically be working on Flood Adapt and the FloodAdaptUsers team if you are testing the application.
-
-# Contributing
-
+FloodAdapt is currently in an intensive development stage. Independent usage of the repository will be challenging prior to end-of-year 2024. FloodAdapt documentation will be expanded on throughout 2024.
 
 ## Setting up conda
 
-In order to develop on `flood_adapt` locally, please follow the following steps:
+In order to develop the FloodAdapt-GUI locally, please follow the following steps:
 
 - Download and install [mambaforge](https://mamba.readthedocs.io/en/latest/installation.html#fresh-install).
 
-- Initialize `conda` by running the following in the `Miniconda prompt`:
-
-```
-conda init
-```
-
 - Depending on your company settings, you might also have to run the following in a Powershell terminal as administrator:
 
-```
+```bash
 Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
 ```
 
-## Creating (or updating) the environment
+## Installation
 
-- Create (or update) the environment by executing the following in your terminal:
+Before starting the installation process, make sure you have access to all required private repositories by ensuring you are in the Teams `FloodAdaptUsers` in the [Deltares](https://github.com/orgs/Deltares/teams/floodadaptusers) and [Deltares-research](https://github.com/orgs/Deltares-research/teams/floodadaptusers) organizations.
 
-```
-mamba env create --file=environment.yml --force
-```
+To run FloodAdapt, GeoPandas needs to be installed. GeoPandas depends for its spatial functionality on a large geospatial, open source stack of libraries (GEOS, GDAL, PROJ). These base C libraries can sometimes be a challenge to install.
 
-## Installing FloodAdapt
+Pre built binaries are provided in `environment/geospatial-wheels`, and are installed using `environment/make_environment.py`. They can be manually installed into your environment using: `pip install <wheel_path>`. All other dependencies can be either found on pip (https://pypi.org/) or have pyproj.toml files to use during installation.
 
-- Activate the environment
+## Creating the environment
 
-```
-conda activate flood_adapt
-```
+The environment for running and developing FloodAdapt can be made with the script: `make_environment.py`.
 
-In order to develop on `flood_adapt` locally, execute the following line inside your virtual environment
+Create a virtual environment named `ENV_NAME` with the core dependencies of FloodAdapt by running the following in a terminal:
 
 ```bash
-pip install -e .
+python make_environment.py -n ENV_NAME
 ```
 
-## Installing pre-commit (For developers)
-Hook scripts are useful for identifying simple issues before submission to code review.
-Precommit runs automatically on every commit point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks. Install the git hook scripts with:
+Additional installation options that can be added to the end of the command:
+
+`-n ENV_NAME` or `--name ENV_NAME`
+- The name for the environment to be created. If it already exists, it will be removed and recreated from scratch.
+
+`-p PREFIX` or `--prefix PREFIX`
+- Creates the environment at `prefix/name` instead of the default conda location.
+
+`-d GROUP` or `--dev GROUP`
+- Install optional dependencies of FloodAdapt-GUI and FloodAdapt in addition the core ones. GROUP: dev | build | all
 
+`-e` or `--editable`
+- Do an editable install of the FloodAdapt-GUI and FloodAdapt packages.
+
+## Updating the environment
+To update your current environment, run the following commands:
 ```bash
-pre-commit install
+conda activate ENV_NAME
+
+cd FloodAdapt
+
+# Update all packages to the latest available versions allowed by requirements defined in pyproject.toml files.
+# This also performs the update for the backend dependencies, but deactivates the editable installation and installs everything as a package.
+python -m pip install --upgrade --upgrade-strategy eager -e .
 ```
 
-Running this commands for the first time may take a minute or two, since the hook scripts need to be fetched and installed.
-This scripts are cached, and now pre-commit will run automatically on every ```git commit```.
-## Running the tests
+## Configure database
 
+FloodAdapt uses a database to store, handle and organize input files, output files and static data. This database needs to be configured the first time you want to use FloodAdapt. Which is done via `flood_adapt/config.py` which contains functions to set environment variables, specific to your system.
 
-If no already done, activate the environment
+To validate and set these variables, write a .toml file containing the following lines:
+```bash
+# The path to the FloodAdapt Database containing databases with possibly multiple different sites.
+DATABASE_ROOT="<path/to/Database>"
+
+# The site name to use at startup.
+DATABASE_NAME="<site_name>"
 
+# The path to the `system` folder containing the binaries for SFINCS and fiat.
+SYSTEM_FOLDER="<path/to/system>"
 ```
-conda activate flood_adapt
+
+Then, before doing any calculations, add the following lines to the top of your script / initialize function to validate and set the environment variables:
+```python
+from flood_adapt.config import parse_config
+config_path = "path/to/your.toml"
+parse_config(config_path)
 ```
 
+## Developing FloodAdapt
+
+Clone the repository
 
-To run the tests execute:
+```bash
+git clone https://github.com/Deltares/FloodAdapt
+```
+
+Create a developer environment `example_name`, with an editable installation FloodAdapt core and the `dev` optional dependency group .
 
 ```bash
-pytest tests
+python make_environment.py -n example_name --editable --dev dev
 ```
 
-## Setup Visual Studio Code (optional)
+### Optional Dependencies
+Different groups of packages are required for various tasks that are not required to run the application. The optional dependency groups for FloodAdapt are:
+- `dev` - linting, pre-commit hooks & testing
+- `build` - distribution related packages & publishing to pip
+- `docs` - generating documentation & example notebooks
+- `all` - all of the above
+
+An optional dependency group can be installed in addition to installing the core using `pip install .[group_name]`, where '.' can be replaced by a directory path that contains a pyproj.toml file.
 
-1. Install the [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python), [ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) and [autoDocstring](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring) extensions.
+### Setup Visual Studio Code (optional)
+
+1. Initialize pre-commit to run locally before you commit by running the following command:
+```
+pre-commit install
+```
 
 2. Add the following to your `.vscode/settings.json` file in your workspace