Topology and network analytics

[bjpalmer]

Opening this issue to track development of network topology and analytics methodology.

[bjpalmer]

I've added a base analytics interface to the bus and branch base classes. This supports numGenerators(), numLoads() and numLines() functions and has some basic implementations based on the contents of the DataCollection object associated with each network component. These functions can be overwritten to fine-tune behavior for individual applications.

I added a stub for a function numStorage() on the buses but I didn't implement it since I have no idea how to count these elements.

There is also a function setData(DataCollection *data) that can be used to make the data collection object for each component visible to the default implementations. I think this can be used in the constructor for whatever we end up using as the upper level topology query module.

[wperkins]

I've added a base analytics interface to the bus and branch base classes. This supports numGenerators(), numLoads() and numLines() functions and has some basic implementations based on the contents of the DataCollection object associated with each network component.
[...]

I've implemented these up through the HADREC application and its Python interface. I can't test yet because of #198.

[bjpalmer]

I'm modifying the base implementations so that only elements that have status 'ON' are counted. If no status variable is found, the element is assumed to be 'ON'.

[wperkins]

@bjpalmer, you are probably working on this, but somewhere in the component tree, BaseBusAnalyticsInterface::setData() needs to be called. I would suggest overriding BaseComponent::load() in BaseBranchComponent and BaseBusComponent.

[bjpalmer]

Won't it get overridden again, when the load function is implemented by the application? Another possibiity would be to include it in the BaseFactory implementation of the BaseFactory::load() function. This does not usually get overwritten. Yet another possibility would be to include it at the end of the BaseNetwork::partition() function. Not sure how these play out as far as maintaining clean hierarchies.

[wperkins]

Won't it get overridden again, when the load function is implemented by the application? Another possibiity would be to include it in the BaseFactory implementation of the BaseFactory::load() function. This does not usually get overwritten. Yet another possibility would be to include it at the end of the BaseNetwork::partition() function. Not sure how these play out as far as maintaining clean hierarchies.

You know best. I'm probably making bad assumptions about how the component class tree inheritence is coded.

[bjpalmer]

The analytics functionality seems to be working. I put some calls in the powerflow application that report on the total number of generators, loads and lines at the start of the calculation. I'm still a little queasy about putting this in the network class since this deals exclusively with managing the topology and is agnostic to any functionality representing the actual application. It might be better to put it an a separate template class the is initialized with the network.

[bjpalmer]

I created a separate analytics module and put the functionality for getting the total number of generators, etc. in that. I also added it to the powerflow application. I personally prefer having this functionality in a separate class but I'll go with the majority. From an implementation view, it's pretty easy to do it either way.

[wperkins]

@jainmilan, @bjpalmer, and @shri: The current (on application-topology branch) Python topology and analytics interface, WRT @jainmilan document, is as follows:

• hadapp.totalBuses(): number of buses (grid.n_bus, grid.bus.shape[0])
• hadapp.totalBranches(): number of branches
• hadapp.numGenerators(): number of generators (grid.n_gen, grid.gen.shape[0])
• hadapp.numLoads(): number of loads (grid.n_load, grid.load.shape[0])
• hadapp.numLines(): number of lines + transformers (grid.n_lines, grid.line.shape[1] + grid.trafo.shape[1])
• hadapp.numStorage(): number of storage devices
• hadapp.getBranchEndpoints(): from and to bus (grid.line.from_bus, grid.line.to_bus)
• hadapp.getConnectedBranches(): branches connected to bus

We don't have anything transformer specific.

If we need to assemble tables like those, then we need to expose devices within buses and branches. First, we need counts:

• NetworkAnalytics::numLoads(const int& bus_index): number of loads on a specific bus
• NetworkAnalytics::numGenerators(const int& bus_index): number of generators on a specific bus
• NetworkAnalytics::numLines(const int& branch_index): number of lines composing a specific branch

with corresponding Python methods. These are easy to add.

We also need to expose individual device information. I suggest for device properties (not device state) we have routines like this:

• NetworkAnalytics::loadInfo(const int& bus_index, const int& load_index)

or

• NetworkAnalytics::loadInfo(const int& bus_index, const std::string& name)

Returning whatever "Info" is, a single value, array, or structure. This will require some discussion.

Retrieving device state also needs discussion.

Just my $0.02.

[wperkins]

If we need to assemble tables like those, then we need to expose devices within buses and branches. First, we need counts:

* `NetworkAnalytics::numLoads(const int& bus_index)`: number of loads on a specific bus

* `NetworkAnalytics::numGenerators(const int& bus_index)`: number of generators on a specific bus

* `NetworkAnalytics::numLines(const int& branch_index)`: number of lines composing a specific branch

with corresponding Python methods. These are easy to add.

I added a complete set of bus/branch specific methods like these, with Python API in HADREC.

[wperkins]

I've rebased the application-topology branch to include the dynamic simulation Python API. I've extended that API so that gridpack.dynamic_simulation.DSFullApp now has the same NetworkAnalytics API, as described above, as the gridpack.hadrec.Module. So, either can be used for dynamic simulation in Python.

[wperkins]

As was concluded in the meeting the other day, we only need to query the DataCollection of buses and branches. To provide access to detailed device information, I suggest something like

template <typename T> 
bool NetworkAnalytics::loadInfo(const int& bus_idx, const int& load_idx, const std::string& field, T& value)

or, more likely, non-template versions

bool NetworkAnalytics::loadInfo(const int& bus_idx, const int& load_idx, const std::string& field, int& value);
bool NetworkAnalytics::loadInfo(const int& bus_idx, const int& load_idx, const std::string& field, double& value);
bool NetworkAnalytics::loadInfo(const int& bus_idx, const int& load_idx, const std::string& field, std::string& value);

within GridPACK C++. These would return true and fill value if the field exists and false otherwise. Similar methods would be added to DS and HADREC.

In Python, the interfaces to these would look like

result = dsapp.loadInfo(bus_idx, load_idx, field)

which would return the value (whatever type) if successful, None otherwise.

To use this interface, we would need to relate the necessary grid2op fields to the GridPACK data collection dictionary and know the field type.

I have a few wild guesses in this table:
Grid2Opt-GridPACK-dictionary.ods

[wperkins]

As was concluded in the meeting the other day, we only need to query the DataCollection of buses and branches. To provide access to detailed device information, I suggest something like
[...]

I had to change course a little. GridPACK data collection objects are contained within a bus or branch. All device (load, line, generator, etc.) fields are interspersed amongst the bus and branch fields. Concequently, there is no real distinction between querying a generator or load field and this has to be done at the bus level. Also, querying a field requires knowing the type of the result which further complicates the interface.

Here is a working Python script that dumps bus information from a dynamic simulation:

def network_analytics_dump(ds_app):
    nbus = ds_app.totalBuses()
    for bus in range(0, nbus):
        bnum = ds_app.getBusInfoInt(bus, "BUS_NUMBER", -1)
        btype = ds_app.getBusInfoInt(bus, "BUS_TYPE", -1)
        bname = ds_app.getBusInfoString(bus, "BUS_NAME", -1)
        bvmag = ds_app.getBusInfoReal(bus, "BUS_VOLTAGE_MAG", -1)
        print(bus, bnum, bname, btype, ds_app.numGenerators(bus),
               ds_app.numLoads(bus), bvmag)
        for g in range(ds_app.numGenerators(bus)):
            print(" gen: ", g,
                  ds_app.getBusInfoInt(bus, "GENERATOR_NUMBER", g),
                  ds_app.getBusInfoString(bus, "GENERATOR_ID", g),
                  ds_app.getBusInfoReal(bus, "GENERATOR_PG", g),
                  ds_app.getBusInfoReal(bus, "GENERATOR_QG", g))
        for l in range(ds_app.numLoads(bus)):
            print("load: ", l,
                  ds_app.getBusInfoInt(bus, "LOAD_NUMBER", l),
                  ds_app.getBusInfoString(bus, "LOAD_ID", l),
                  ds_app.getBusInfoReal(bus, "LOAD_PL", l),
                  ds_app.getBusInfoReal(bus, "LOAD_QL", l))

Using the 9-bus simulation, here is what this produces:

0 1 'bus-1       ' 3 1 0 1.04
 gen:  0 None 1  0.669999999997691 0.7010759010981538
1 2 'bus-2       ' 2 1 0 1.02533
 gen:  0 None 1  1.63 0.6028676952146661
2 3 'bus-3       ' 2 1 0 1.02536
 gen:  0 None 1  0.85 0.4202344495139094
3 4 'bus-4       ' 1 0 0 1.0018586298562873
4 5 'bus-5       ' 1 0 1 0.9816273801688302
load:  0 None 1  90.0 30.0
5 6 'bus-6       ' 1 0 0 1.002520962766657
6 7 'bus-7       ' 1 0 1 0.9812684602469763
load:  0 None 1  100.0 35.0
7 8 'bus-8       ' 1 0 0 0.9935621039389169
8 9 'bus-9       ' 1 0 1 0.9661929333427239
load:  0 None 1  125.0 50.0

[wperkins]

Updated with branch info ...

def network_analytics_dump(ds_app):
    nbus = ds_app.totalBuses()
    for bus in range(nbus):
        print(bus,
              ds_app.getBusInfoInt(bus, "BUS_NUMBER", -1),
              ds_app.getBusInfoString(bus, "BUS_NAME", -1),
              ds_app.getBusInfoInt(bus, "BUS_TYPE", -1),
              ds_app.numGenerators(bus),
              ds_app.numLoads(bus),
              ds_app.getBusInfoReal(bus, "BUS_VOLTAGE_MAG", -1))
        for g in range(ds_app.numGenerators(bus)):
            print(" gen: ", g,
                  ds_app.getBusInfoInt(bus, "GENERATOR_NUMBER", g),
                  ds_app.getBusInfoString(bus, "GENERATOR_ID", g),
                  ds_app.getBusInfoReal(bus, "GENERATOR_PG", g),
                  ds_app.getBusInfoReal(bus, "GENERATOR_QG", g))
        for l in range(ds_app.numLoads(bus)):
            print("load: ", l,
                  ds_app.getBusInfoInt(bus, "LOAD_NUMBER", l),
                  ds_app.getBusInfoString(bus, "LOAD_ID", l),
                  ds_app.getBusInfoReal(bus, "LOAD_PL", l),
                  ds_app.getBusInfoReal(bus, "LOAD_QL", l))
    nbranch = ds_app.totalBranches()
    for branch in range(0, nbranch):
        (f, t) = ds_app.getBranchEndpoints(branch)
        print(branch, f, t, 
              ds_app.getBranchInfoInt(branch, "BRANCH_ELEMENTS", -1),
              ds_app.getBranchInfoInt(branch, "BRANCH_INDEX", -1),
              ds_app.getBranchInfoString(branch, "BRANCH_NAME", -1),
              ds_app.getBranchInfoReal(branch, "BRANCH_LENGTH", -1))

produces

0 1 'bus-1       ' 3 1 0 1.04
 gen:  0 None 1  0.669999999997691 0.7010759010981538
1 2 'bus-2       ' 2 1 0 1.02533
 gen:  0 None 1  1.63 0.6028676952146661
2 3 'bus-3       ' 2 1 0 1.02536
 gen:  0 None 1  0.85 0.4202344495139094
3 4 'bus-4       ' 1 0 0 1.0018586298562873
4 5 'bus-5       ' 1 0 1 0.9816273801688302
load:  0 None 1  90.0 30.0
5 6 'bus-6       ' 1 0 0 1.002520962766657
6 7 'bus-7       ' 1 0 1 0.9812684602469763
load:  0 None 1  100.0 35.0
7 8 'bus-8       ' 1 0 0 0.9935621039389169
8 9 'bus-9       ' 1 0 1 0.9661929333427239
load:  0 None 1  125.0 50.0
0 1 4 0 None None None
1 2 8 1 None None None
2 3 6 2 None None None
3 4 5 3 None None None
4 4 9 4 None None None
5 5 6 5 None None None
6 6 7 6 None None None
7 7 8 7 None None None
8 8 9 8 None None None

for the 9-bus case.

The changes are checked into the application-topology branch.

[wperkins]

Reposted with @jainmilan's descriptions:
Grid2Opt-GridPACK-dictionary.ods

[wperkins]

The -1 device index is no longer required for bus/branch level information. So,

ds_app.getBusInfoInt(bus, "BUS_NUMBER", -1)

is just

ds_app.getBusInfoInt(bus, "BUS_NUMBER")

A device index is only required for device (generator, load, line, etc.) information.