Duplicate ENTSO-e prices sensor with offset

[rhpijnacker]

I'm using the entsoe plugin to load the data into flexmeasures.
However, for optimal schedule generation, I require a 2nd prices sensor, with the additional markup that my energy company adds when I buy energy (compared to when I sell it back to them).

Do you have some suggestions on what would be the easiest way to create such a sensor, and keep it up-to-date any time new data gets posted from ENTSO-e?

[nhoening]

That would be a reporter! We do that in the V2GLiberty project ourselves. I would ask if @Flix6x has more help, but my rough idea is

• Add the sensor
• Add a datasource that stores the reporter config (which is a simple computation to add the offset). Adding a source can be done by flexmeasures add source
• Add a CLI command which creates a report (computes and saves the data) by flexmeasures add report

The documentation to build and compute reporters is not as good as it could be, as we lack the time. Some links:

• https://flexmeasures.io/016-profitloss-reporter/
• https://flexmeasures.readthedocs.io/en/latest/tut/toy-example-reporter.html (the toy-tutorial code adding the structure is here)

[nhoening]

@Flix6x Maybe we can share an example of how we do this?

[rhpijnacker]

@Flix6x, if you have some time, I would love to see that example.

[Flix6x]

Here's a script we run daily, named get-prices.sh. The start-offset is there for filling small gaps (previous 2 days). The end-offset is there for applying the transformations on price data (forecasts) further ahead too, in case you have those.

#!/bin/bash
flexmeasures entsoe import-day-ahead-prices --country NL --timezone Europe/Amsterdam;
flexmeasures add report --config reporters/supplier-config.json --parameters reporters/tibber-consumption.json --end-offset 3D,DB --start-offset -2D,DB
flexmeasures add report --config reporters/supplier-config.json --parameters reporters/tibber-production.json --end-offset 3D,DB --start-offset -2D,DB;

It uses one file to configure reporters for supplier prices, and another file to specify where to get the parameters from, for a specific supplier.

reporters/supplier-config.json:

{
    "required_input": [
        {
            "name" : "vat_df"
        },
        {
            "name": "energy_tax_df"
        },
        {
            "name": "supplier_surcharge_df"
        },
        {
            "name": "day_ahead_df"
        }
    ],
    "required_output": [
        {
            "name": "day_ahead_df"
        }
    ],
    "transformations": [
        {
            "df_input": "vat_df",
            "method": "droplevel",
            "args": [
                [
                    1,
                    2,
                    3
                ]
            ]
        },
        {
            "method": "divide",
            "args": [
                100
            ]
        },
        {
            "method": "add",
            "args": [
                1
            ]
        },
        {
            "df_input": "energy_tax_df",
            "method": "droplevel",
            "args": [
                [
                    1,
                    2,
                    3
                ]
            ]
        },
        {
            "df_input": "supplier_surcharge_df",
            "method": "droplevel",
            "args": [
                [
                    1,
                    2,
                    3
                ]
            ]
        },
        {
            "df_input": "day_ahead_df",
            "method": "droplevel",
            "args": [
                [
                    1,
                    2,
                    3
                ]
            ]
        },
        {
            "method": "add",
            "args": [
                "@supplier_surcharge_df"
            ]
        },
        {
            "method": "add",
            "args": [
                "@energy_tax_df"
            ]
        },
        {
            "method": "multiply",
            "args": [
                "@vat_df"
            ]
        }
    ]
}

reporters/tibber-consumption.json:

{
    "input": [
        {
            "sensor": 55,
            "name" : "vat_df"
        },
        {
            "sensor": 56,
            "name": "energy_tax_df"
        },
        {
            "sensor": 57,
            "name": "supplier_surcharge_df"
        },
        {
            "sensor": 14,
            "name": "day_ahead_df"
        }
    ],
    "output": [
        {
            "sensor": 58,
            "name": "day_ahead_df"
        }
    ]
}

Sensor IDs will be different for your local database.

[rhpijnacker]

Hi @Flix6x,
Thank you for the example. Could you also share how you set up the input sensors for vat etc.?
Do I also need to create the sensor for the output sensor beforehand, or is this automagically created by running the add report command?

A question about the supplier-config: what does the droplevel statement do?

[Flix6x]

Could you also share how you set up the input sensors for vat etc.?

flexmeasures add sensor --name VAT --unit '%' there is more, but I need to come back to you on this. I want to check our sensor setup for you, but I am currently indisposed.

Do I also need to create the sensor for the output sensor beforehand, or is this automagically created by running the add report command?

The output sensor should be created beforehand as well.

A question about the supplier-config: what does the droplevel statement do?

droplevel in the config drops all index levels of the pandas DataFrame except the one describing event starts, leaving a simpler representation of the time series than the one in timely-beliefs.

[Flix6x]

I checked. Nothing special. Just the event resolution was missing:

flexmeasures add sensor --name VAT --unit '%' --resolution P366D

This assumes the VAT is set for a period of 366 days. Unfortunately we haven't found a way yet to let SQLAlchemy's ORM load nominal durations defined in Postgres, such as P1Y, to be converted to nominal Pandas Offsets in Python. It currently loads it as datetime.timedelta Python objects, which are strictly absolute durations. That is, P1Y would become timedelta(days=365), which would result in a missing day of data in leap years.

Likewise, we use P31D for sensors that record values that can change every month, because SQLAlchemy loads P1M as timedelta(days=30) and we would miss a day or more in some months.

We record their values on midnight of January 1st (for yearly sensors) and on the 1st of each month (for monthly sensors). We found that the resulting overlap in non-leap years and months with fewer than 31 days is not problematic, because our search method always returns the most recent value by default.

You can use flexmeasures add beliefs to add data to each sensor. There are some examples in our docs (1st tutorial?) that use this command to load (price) data from CSV.

(We investigated the issue of loading nominal durations before, in SeitaBV/timely-beliefs#141)

[rhpijnacker]

Thanks for the additional information.
It looks like I need to associate these sensors with an asset. It seems odd to add them to the EVSE asset. Instead, I could add them to the asset created for the Entso-E data. However, I think it makes even more sense to create a separate asset for this, since this data is tightly coupled to my specific energy contract. Can I ask how Seita set this up for the V2G-Liberty accounts?

A related question: In the CLI interface, I find it hard to detect the asset details for the generated Entso-E data. I need to know up-front what the asset-id is, but there seems to be no interface to show all assets. Am I missing something?

[Flix6x]

Thanks for the additional information.
It looks like I need to associate these sensors with an asset. It seems odd to add them to the EVSE asset. Instead, I could add them to the asset created for the Entso-E data. However, I think it makes even more sense to create a separate asset for this, since this data is tightly coupled to my specific energy contract. Can I ask how Seita set this up for the V2G-Liberty accounts?

We set up some hierarchical structure of assets to conceptually model the Dutch electricity system, or at least the parts that are relevant to us. For instance, one Asset for the transmission system and another Asset for the electricity market. Our supplier price sensors live under the latter. But the structure really doesn't matter, and I consider it a personal choice that our company will also likely revisit (restructure) in the future. If you don't need to model many suppliers, I suggest you keep the nesting of Assets to a minimum, so you don't need to click around so much in the UI.

A related question: In the CLI interface, I find it hard to detect the asset details for the generated Entso-E data. I need to know up-front what the asset-id is, but there seems to be no interface to show all assets. Am I missing something?

If we do this through the CLI, we'd use flexmeasures show accounts to find out the account ID, and flexmeasures show account <ID> to then list its assets and users. But since this is a public asset, there is no account associated with it, so you've stumbled upon a missing feature, namely: FlexMeasures/flexmeasures#1214 (created just now).

[rhpijnacker]

@Flix6x Sorry to keep bothering you...
The Dutch energy tax is 13.165 eurocent per kWh incl VAT. It looks like you are adding energy tax and surcharge before applying VAT. Does that mean I should enter the tax without VAT? (Or is this applied twice in NL?)

Just to make sure: I need to add this in EUR/MWh in FM, so that would be 131.65, right?

I see that you are using the same supplier_config.json for calculating the production prices. I would expect that the production prices do not include the surcharge. How does that work?

[rhpijnacker]

I'm getting this warning:

/usr/local/lib/python3.10/dist-packages/timely_beliefs/beliefs/utils.py:1183: FutureWarning: BeliefsDataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.

Should I be concerned about that?

[Flix6x]

The Dutch energy tax is 13.165 eurocent per kWh incl VAT. It looks like you are adding energy tax and surcharge before applying VAT. Does that mean I should enter the tax without VAT? (Or is this applied twice in NL?)

Indeed, without VAT. The VAT is applied last as a multiplication factor to the total of wholesale energy price, energy tax and surcharge (i.e. the three components all before VAT).

Just to make sure: I need to add this in EUR/MWh in FM, so that would be 131.65, right?

I believe the Reporters do not automatically convert units yet, so yes.

I see that you are using the same supplier_config.json for calculating the production prices. I would expect that the production prices do not include the surcharge. How does that work?

In case the surcharge does not apply to the production process, then you could create a separate config for production that doesn't add the surcharge. Using the same config corresponds to the principle of saldering.

[Flix6x]

I'm getting this warning:

/usr/local/lib/python3.10/dist-packages/timely_beliefs/beliefs/utils.py:1183: FutureWarning: BeliefsDataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.

Should I be concerned about that?

Looks like an issue of compatibility of timely-beliefs and pandas, which should be dealt with in timely-beliefs. Looks like a simple enough replacement, if you want to make a PR out if it.

[rhpijnacker]

I see that you are using the same supplier_config.json for calculating the production prices. I would expect that the production prices do not include the surcharge. How does that work?

In case the surcharge does not apply to the production process, then you could create a separate config for production that doesn't add the surcharge. Using the same config corresponds to the principle of saldering.

Oh, I think I may have misunderstood how "saldering" works with a dynamic energy contract. I thought that this only applied to the VAT and energy tax, but the surcharge is still added for consumption prices, but not to production prices.