Data Model Abstraction Refactor (#272)

This PR contains the result of a refactor of the default user-facing data model in PowerModelsDistribution to a new `ENGINEERING` data model

A merge of this PR will result in the v0.9.0 release of PowerModelsDistribution

# What is the New Data Model

We have designed a data model that more closely matches the engineering realities of a distribution network, that allows for easy transformation and visualization of the case and includes more component types than are represented in the original (and still existing under the hood) `MATHEMATICAL` data model with which previous users will be familiar.

The new data model is documented in detail in the documentation, including all of the currently supported component types are their individual fields, the types of values, and any defaults we might assume.

# Why Do We Need a New Data Model?

In distribution networks the variety of components in a standard network is much more diverse than in a traditional transmission network, and their mathematical representations are comparatively complex. For example, in order to represent n-winding multiphase transformers with loss models we have to represent a single transformer with a large number of lines, buses, and lossless 2-winding transformers, which previously users have found confusing when attempting to compare their parsed network to the originating file. To simplify the user experience, and to introduce the ability to easily make transformations of the networks, we formulated the `ENGINEERING` data model.

# How Does It Work?

For the most part, the commands that you were using before will not change appreciably. `parse_file` is still the standard way to import data, `run_mc_` commands will still run problem specifications, and PowerModel types should not be different at all. The workflow for most users should be identical; although there are now some intermediate steps to building the JuMP model from the default data dictionary, these steps should be largely invisible to the user unless they want to expose them. Users should experience the following workflow by default:

`parse_file` returns `ENGINEERING` data model -> `run_mc_{}` runs optimization problem and returns result in `ENGINEERING` model format

What is happening behind the scenes is the following:

`parse_file` returns `ENGINEERING` data model -> `run_mc_{}` converts to `MATHEMATICAL` data model in per-unit, builds JuMP model, runs optimization, converts results in `MATHEMATICAL` model format to SI units and converts back to `ENGINEERING` format.

We have exposed to the user the functions that do these conversions if they wish to use them, and have provided examples in Jupyter Notebooks in the `/examples` folder that demonstrates both of these types of workflows.

# If the user experience is the same, what changed besides the default model?

There were a multitude of changes that happened in order to support the functionality for this new data model, as well as some other changes that address concerns and comments of users

- We have eliminated some problem types. This is perhaps one of the biggest changes users will notice; variants of problems like `run_mc_opf_iv` and `run_mc_opf_bf` are eliminated in favor of having users use `run_mc_opf` only, and redirecting to the appropriate problem definitions using multiple dispatch. The only variants that should continue to exist are truly unique problem specifications, _e.g._ `run_mc_opf_oltc` or `run_mc_mld`, etc.
- We have changed the names of variable and constraint functions to support PowerModels v0.17
- We have updated the solution building to use the capabilities in InfrastructureModels v0.5
- We have made updates to the mathematical models for transformers to make them more accurate
- We have added new capabilities to the DSS parser, parsing additional components such as loadshapes, xycurves, xfmrcodes
- We have squashed a lot of outstanding bugs in the DSS parser
- We have ensured that the component naming conventions, particularly for virtual components in the `MATHEMATICAL` model, are consistent across the package
- We have added helper functions to create models from the REPL or scripts, _e.g._ `Model`, `add_bus!`, etc., rather than rely solely on dss inputs (see notebook in `/examples`)
- `parse_dss` originally followed a similar parsing pattern as `parse_pti` in PowerModels, parsing component dictionaries into Vectors. Instead, we now parse into a structure that is much closer to the final data structures that users utilize, in order to make debugging and future parsing easier
- fixed broken SDP/SOC relaxations (see #262)

# Feature X that I want isn't in the new data model

Definitely make a new Github Issue. We view any additions to the data model to be additive and therefore not a breaking change, which means we can make rapid releases containing those additions without any problem.

# Related Issues

Closes #221
Closes #256
Closes #260
Closes #242
Closes #245
Closes #33
Closes #249
Closes #253
Closes #193
Closes #257
Closes #246
Closes #248
Closes #247
Closes #252
Closes #175
Closes #251
Closes #234
Closes #238
Closes #240
Closes #236
Closes #237
Closes #230
Closes #239
Closes #243
Closes #227
Closes #264
Closes #235
Closes #56
Closes #274

Co-authored-by: Sander Claeys <[email protected]>

.gitignore

@@ -6,4 +6,6 @@
 deps/deps.jl
 docs/build
 
-Manifest.toml
+Manifest.toml
+
+.ipynb_checkpoints/

.travis.yml

@@ -5,15 +5,15 @@ os:
   - windows
 julia:
   - 1.0
-  - 1.3
+  - 1.4
   - nightly
 codecov: true
 jobs:
   allow_failures:
     - julia: nightly
   include:
     - stage: "Documentation"
-      julia: 1.3
+      julia: 1.4
       os: linux
       script:
         - julia --project=docs/ -e 'using Pkg; Pkg.instantiate(); Pkg.develop(PackageSpec(path=pwd()))'

CHANGELOG.md

@@ -2,14 +2,40 @@
 
 ## staged
 
+- none
+
+## v0.9.0
+
+- Add `instantiate_mc_model` to aid in building JuMP model from ENGINEERING data model
+- SDP and SOC relaxations were broken but are fixed again (unit tests added)
+- Remove `run_mc_opf_iv`, `run_mc_opf_bf`, `run_mc_opf_bf_lm`, `run_mc_pf_bf`, `run_mc_pf_iv`, these can be accessed by using the correct formulation with `run_mc_opf` and `run_mc_pf`
+- Add support for Memento 1.1
+- Add support for PowerModels v0.17 (breaking)
+- Add support for InfrastructureModels v0.5
+- Updates JSON parser to handle enum (`"data_model"` values)
+- Adds some commonly used InfrastructureModels and PowerModels functions as exports
+- Adds model building functions `add_{component}!` to aid in building simple models for testing (experimental)
+- Add `run_mc_model` (adds `ref_add_arcs_transformer!` to ref_extensions, and sets `multiconductor=true` by default) (breaking)
+- Rename `ref_add_arcs_trans` -> `ref_add_arcs_transformer` (breaking)
+- Update `count_nodes`, now counts source nodes as well, excludes \_virtual objects
+- Change \_PMs and \_IMs to \_PM, \_IM, respectively
+- Add example for PowerModelsDistribution usage (see Jupyter notebooks in `/examples`)
+- Update transformer mathematical model
+- Introduce new data models: ENGINEERING, MATHEMATICAL (see data model documentation) (breaking)
+- Update DSS parser to be more robust, and parse into new format (breaking)
+- Updates DSS paser to parse more options/commands, moves these into `"options"` dict (breaking)
+- Updates how dss `like` is applied to better match opendss (almost all properties are copied with like) (breaking)
+- Add support for new OpenDSS components (loadshape, xfmrcode, xycurve)
+- Add support for JuMP v0.22 (exports `optimizer_with_attributtes` by default)
+- Add support for PowerModels v0.16 (breaking)
 - Add support for Memento v0.13, v1.0
 
 ## v0.8.1
 
 - Update to support JuMP v0.21
 - Makes bounds optional, turned on by default (#250)
 - Updated transformer data model in the mathematical model (#250)
-- Add automatic parsing of lon,lat from buscoords file into PMD data structure (#245, #249)
+- Add automatic parsing of lon,lat from buscoords file into PowerModelsDistribution data structure (#245, #249)
 - Updates virtual_sourcebus, which is intended to represent a voltage source, to have a fixed voltage magnitude (#246,#248)
 - Add parsing of series data files into array fields in OpenDSS parser
 - Add LoadShape parsing to OpenDSS parser (#247)
@@ -22,9 +48,9 @@
 
 ## v0.8.0
 
-- Update solution building infrastructure (PMs #77) (breaking). The reported solution is now consistent with the variable space of the formulation.
+- Update solution building infrastructure (PowerModels #77) (breaking). The reported solution is now consistent with the variable space of the formulation.
 - Moved multi-conductor support from PowerModels into PowerModelsDistribution. (breaking)
-- PMs.var no longer takes conductor as an argument
+- PowerModels.var no longer takes conductor as an argument
 - Constraints have been (partially) re-written to use vectorized JuMP syntax where possible.
 - Bugfixes: generator on-off and storage on-off constraints were incorrect
 - Removal of SOCWRPowerModel

Project.toml

@@ -2,7 +2,7 @@ name = "PowerModelsDistribution"
 uuid = "d7431456-977f-11e9-2de3-97ff7677985e"
 authors = ["David M Fobes <[email protected]>", "Carleton Coffrin"]
 repo = "https://github.com/lanl-ansi/PowerModelsDistribution.jl.git"
-version = "0.8.1"
+version = "0.9.0"
 
 [deps]
 InfrastructureModels = "2030c09a-7f63-5d83-885d-db604e0e9cc0"
@@ -16,14 +16,14 @@ Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 
 [compat]
 Cbc = ">= 0.4"
-InfrastructureModels = "~0.4"
+InfrastructureModels = "~0.5"
 Ipopt = ">= 0.4"
 JSON = "~0.18, ~0.19, ~0.20, ~0.21"
 JuMP = "~0.19.2, ~0.20, ~0.21"
 Juniper = ">= 0.4"
 MathOptInterface = "~0.8, ~0.9"
-Memento = "~0.10, ~0.11, ~0.12, ~0.13, ~1.0"
-PowerModels = "~0.15"
+Memento = "~0.10, ~0.11, ~0.12, ~0.13, ~1.0, ~1.1"
+PowerModels = "~0.17"
 SCS = ">= 0.4"
 julia = "^1"
 

README.md

@@ -16,22 +16,37 @@ This enables the definition of a wide variety of power network formulations and
 ## Core Problem Specifications
 
 - Power Flow (pf)
-- Optimal Power Flow (opf), for the Bus Injection Model (BIM) as well as the Branch Flow Model (BFM)
-- Continuous load shed, minimum load delta (mld), for the Branch Flow Model (LPLinUBFPowerModel), AC Polar (ACPPowerModel), and Network Flow Approximation (NFAPowerModel)
+  - ACP, ACR, IVR, LinDist3Flow, NFA, DCP
+- Optimal Power Flow (opf)
+  - ACP, ACR, IVR, LinDist3Flow, NFA, DCP
+- Continuous load shed, minimum load delta (mld)
+  - ACP, LinDist3Flow, NFA
+- Optimal Power Flow with on-load tap-changer (opf_oltc)
+  - ACP
+
+**Note: The documentation is somewhat lagging behind development and the parings of network features with problem specifications with formulations has not been enumerated. We are working to correct this for you.**
 
 ## Core Network Formulations
 
-- AC (polar and rectangular coordinates)
-- SDP BFM relaxation
-- SOC BFM and BIM relaxation (W-space)
-- Linear approximation (LinDist3Flow and simplified unbalanced DistFlow)
+- Nonlinear
+  - ACP
+  - ACR
+  - IVR
+- Relaxations
+  - SDP BFM
+  - SOC BFM and BIM relaxation (W-space)
+- Linear Approximations
+  - LinDist3Flow
+  - NFA
+  - DCP
 
 ## Network Data Formats
 
-- Matlab ".m" files (extended for three-phase)
 - OpenDSS ".dss" files
 
-**Warning:** This package is under active development and may change drastically without warning.
+## Examples
+
+Examples of how to use PowerModelsDistribution can be found in the main documentation and in Jupyter Notebooks inside the `/examples` directory
 
 ## Development
 
@@ -44,6 +59,21 @@ This code has been developed as part of the Advanced Network Science Initiative
 - Sander Claeys (@sanderclaeys) KU Leuven, transformer models and ACR formulation
 - Frederik Geth (@frederikgeth) CSIRO, Distribution modeling advise
 
+## Citing PowerModelsDistribution
+
+If you find PowerModelsDistribution useful for your work, we kindly request that you cite the following [https://arxiv.org/abs/2004.10081](publication):
+
+```bibtex
+@misc{fobes2020powermodelsdistributionjl,
+    title={PowerModelsDistribution.jl: An Open-Source Framework for Exploring Distribution Power Flow Formulations},
+    author={David M Fobes and Sander Claeys and Frederik Geth and Carleton Coffrin},
+    year={2020},
+    eprint={2004.10081},
+    archivePrefix={arXiv},
+    primaryClass={cs.CE}
+}
+```
+
 ## License
 
 This code is provided under a BSD license as part of the Multi-Infrastructure Control and Optimization Toolkit (MICOT) project, LA-CC-13-108.

docs/Project.toml

@@ -4,5 +4,5 @@ InfrastructureModels = "2030c09a-7f63-5d83-885d-db604e0e9cc0"
 PowerModels = "c36e90e8-916a-50a6-bd94-075b64ef4655"
 
 [compat]
-InfrastructureModels = "~0.4"
-PowerModels = "~0.15"
+InfrastructureModels = "~0.5"
+PowerModels = "~0.17"

docs/make.jl

@@ -9,8 +9,13 @@ makedocs(
         "Home" => "index.md",
         "Manual" => [
             "Getting Started" => "quickguide.md",
+            "Connecting Components" => "connections.md",
             "Mathematical Model" => "math-model.md",
-            "Data Formats" => "data-formats.md",
+            "Engineering Data Model" => "eng-data-model.md",
+            "Enums in Engineering Model" => "enums.md",
+            "Conversion to Mathematical Model" => "eng2math.md",
+            "External Data Formats" => "external-data-formats.md",
+            "Examples" => "engineering_model.md",
         ],
         "Library" => [
             "Network Formulations" => "formulations.md",