RFC: subtype System to enable modelling alternative market clearing formulations #34

BSnelling · 2022-10-28T12:47:39Z

Alternative to #33 for enabling modelling different markets. The required fields needed in the System struct are fairly different between the two markets. For example the new market has load services, which don't exist in the first market. Also energy bids (both offers and demand bids) need to contain extra time series information (hence the new type EnergyBids).

Open question, both markets have the concept of GeneratorTimeSeries but the requirements might be different enough that this should be subtyped too.

codecov · 2022-10-28T12:51:14Z

Codecov Report

Merging #34 (05e4d61) into main (2733354) will decrease coverage by 0.72%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main      #34      +/-   ##
==========================================
- Coverage   85.89%   85.16%   -0.73%     
==========================================
  Files           5        5              
  Lines         234      236       +2     
==========================================
  Hits          201      201              
- Misses         33       35       +2

Impacted Files	Coverage Δ
src/system.jl	`47.91% <0.00%> (-2.09%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

BSnelling · 2022-10-28T12:49:22Z

src/system.jl

    "Generator maximum output in the ancillary services market (pu)"
-    regulation_max::KeyedArray{Float64, 2}
+    regulation_max::Union{Missing, KeyedArray{Float64, 2}}


regulation_min and regulation_max do not exist as parameters for modelling the new market so these fields can be missing.

If we intend to reuse code, is it easier to set the values to something like Inf rather than checking for missings?

Interesting question, this is how regulation_max is used in FNModels. I think if we set it to Inf and tried to define that constraint, that wouldn't give us the intended behaviour?

I'm not sure but it looks like it would work out to a constraint that is always true? We might have to reformat the expression or add an extra constraint but it might be worth it to avoid having to special-case things to account for possible missings.

BSnelling · 2022-10-28T12:56:01Z

src/system.jl

@@ -249,11 +262,11 @@ Base.@kwdef struct GeneratorTimeSeries
    "Generation of the generator at the start of the time period (pu)"


This type may also need additional fields for multi_hour time series data. The offer curves don't appear to have multi hour flags but the ancillary services do. If the distribution of multi hours flags for a given day and resource can be different for the different services, we would need four multi hour time series, one for each of the ancillary services.

BSnelling · 2022-10-28T12:58:02Z

src/system.jl

+    Bool indicating whether the bid is a fixed type which can only clear at the given price
+    and volume.
+    """
+    is_fixed_type::KeyedArray{Bool, 2}


Is a Boolean going to be sufficient here? There are three types of bid: curve, variable and fixed. In terms of how the formulation is implemented I think variable is just a curve with only one block. In which case we just have to distinguish which ones are fixed.

BSnelling · 2022-10-28T13:04:29Z

src/system.jl

+    off_supplemental_offers::KeyedArray{Union{Missing, PriceSensitiveBid}, 2}
+end
+
+struct EnergyBids


Is this a good name? When we refer to "bids" it can mean bids in either direction. But in the market data "bid" means demand and "offer" means supply.

BSnelling · 2022-10-28T13:06:10Z

src/system.jl

+    "Energy offers time series data. `KeyedArray` where the axis keys are `bid ids x datetimes`"
+    energy_offers::EnergyBids
+    "Energy demands time series data. `KeyedArray` where the axis keys are `bid ids x datetimes`"
+    energy_demands::EnergyBids


Do these field names make sense? Previously we have referred to increment and decrement bids but these EnergyBids are not necessarily virtual, so do "energy_offers" and "energy_demands" make sense?

I think the names make sense but since we're already explaining things in terms of "supply" and "demand" to clarify, perhaps those are the more straightforward terms to use? Or will all markets use these terms consistently?

raphaelsaavedra · 2022-10-28T13:33:31Z

I like the idea of subtyping system because ISO-specific systems seem to be significantly different. The main problem is that this doesn't just apply to systems, as evidenced by

regulation_min and regulation_max do not exist as parameters for modelling the new market so these fields can be missing.

So my feeling is that we need to gauge if subtyping just system is good enough, or if we need to parametrise everything by ISO – e.g., we could have this package depend on ElectricityMarkets (which would make this not open source anymore...), and then have types for each ISO following a structured type hierarchy so that we can easily dispatch things on System down the line.

Essentially, it seems to be that the main challenges stem from us trying to reconcile ISOs being very different with the fact that we want to have them all structured under a single, general framework. It might be the case that the best thing is for each one to have its own structures.

BSnelling · 2022-10-28T16:20:18Z

we need to gauge if subtyping just system is good enough, or if we need to parametrise everything by ISO

From what I can see at the moment the time series requirements between the two ISOs are pretty different. To accommodate both will require subtyping System and GeneratorTimeSeries.

So far we only know about 1.5 ISOs, but it seems like the static components are consistent (Generator, Branch, Bus) and I assume the calculation of PTDF and LODFs will be similar too? If that's the case those types and that functionality are worth keeping here as part of a shared framework.

raphaelsaavedra · 2022-10-28T16:38:05Z

seems like the static components are consistent (Generator, Branch, Bus) and I assume the calculation of PTDF and LODFs will be similar too? If that's the case those types and that functionality are worth keeping here as part of a shared framework.

I have a feeling that will be the case (system and time series are different, static components are the same / very similar)

morris25 · 2022-10-31T13:43:52Z

src/system.jl

+Fields:
+$TYPEDFIELDS
+"""
+Base.@kwdef mutable struct FinancialSystemDA <: System


I'm not sure about this name since it clashes with another widely-used susbystem in our codebase that we are probably going to have to deal with eventually. Is there another appropriate name?

morris25 · 2022-10-31T18:09:48Z

src/system.jl

    "Generator maximum output in the ancillary services market (pu)"
-    regulation_max::KeyedArray{Float64, 2}
+    regulation_max::Union{Missing, KeyedArray{Float64, 2}}


If we intend to reuse code, is it easier to set the values to something like Inf rather than checking for missings?

morris25 · 2022-10-31T18:31:25Z

src/system.jl

+    "`Dictionary` where the keys are bus names and the values are generator ids at that bus"
+    gens_per_bus::Dictionary{BusName, Vector{String}}
+    "`Dictionary` where the keys are bus names and the values are increment bid ids at that bus"
+    energy_offers_per_bus::Dictionary{BusName, Vector{BidName}}


Are these similar to incs_per_bus in the other systems? Should we try to keep the names consistent?

Yes, I think they are identical in terms of type. It's a question of different concepts. incs/increments refer specifically to virtual bids, whereas here the energy_offers can be physical or virtual and we don't have a way to distinguish between the two.

Perhaps we should go with the most generic option then use a type to differentiate if we need to? If we use them similarly and they behave in a similar way that is.
It seems a shame to have duplicate code when we could use the same accessor across systems,

morris25 · 2022-10-31T18:44:09Z

src/system.jl

+    load_services::LoadServices
+
+    "Energy offers time series data. `KeyedArray` where the axis keys are `bid ids x datetimes`"
+    energy_offers::EnergyBids


Should we switch the other systems to use EnergyBids as well? It looks like we should be able to infer the extra information required

I think that could work. I suppose there would be some (minimal?) overhead to storing the extra information, but the advantages of reusable code.

Since it's repeated information and is generally read-only we could probably use an efficient data structure to avoid too much extra storage

morris25 · 2022-10-31T18:46:56Z

src/system.jl

@@ -1,6 +1,7 @@
 const MARKET_WIDE_ZONE = -9999
 const BidName = InlineString31
 const ZoneNum = Int64
+const PriceSensitiveBid = Vector{Tuple{Float64, Float64}}


We should probably document what the tuple members represent

implement alternative to zones

05e4d61

BSnelling commented Oct 28, 2022

View reviewed changes

morris25 reviewed Oct 31, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: subtype System to enable modelling alternative market clearing formulations #34

RFC: subtype System to enable modelling alternative market clearing formulations #34

BSnelling commented Oct 28, 2022

codecov bot commented Oct 28, 2022

BSnelling Oct 28, 2022

morris25 Oct 31, 2022

BSnelling Nov 1, 2022

morris25 Nov 1, 2022

BSnelling Oct 28, 2022

BSnelling Oct 28, 2022

BSnelling Oct 28, 2022

BSnelling Oct 28, 2022

morris25 Nov 1, 2022

raphaelsaavedra commented Oct 28, 2022

BSnelling commented Oct 28, 2022

raphaelsaavedra commented Oct 28, 2022

morris25 Oct 31, 2022

morris25 Oct 31, 2022

morris25 Oct 31, 2022

BSnelling Nov 1, 2022

morris25 Nov 1, 2022

morris25 Oct 31, 2022

BSnelling Nov 1, 2022

morris25 Nov 1, 2022

morris25 Oct 31, 2022

		@@ -249,11 +262,11 @@ Base.@kwdef struct GeneratorTimeSeries
		"Generation of the generator at the start of the time period (pu)"

RFC: subtype System to enable modelling alternative market clearing formulations #34

Are you sure you want to change the base?

RFC: subtype System to enable modelling alternative market clearing formulations #34

Conversation

BSnelling commented Oct 28, 2022

codecov bot commented Oct 28, 2022

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raphaelsaavedra commented Oct 28, 2022

BSnelling commented Oct 28, 2022

raphaelsaavedra commented Oct 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment