Feature Request - Forecast Horizon - extend beyond 48 values

[kcoffau]

Describe the improvement
I have been happily using 48 forecast values in 30 min blocks for a long time now. I am now able to forecast 6 days in 30 min increments, 180 variables.

I updated my curl to push 96 values (as a starting point) and get error:
"IndexError: index 95 is out of bounds for axis 0 with size 48"

So....

I dont have time this week, but I was going to have a look next week and see what I can do, but before I do, I want to check, is my understanding of 48 values in the forecast the current limitation? Is that for performance or other?

Cheers,
Kel.

[davidusb-geek]

Yes this is a hard coded parameter for performance concerns. If extended we could quickly fall into memory issues and that will be bad for your HA instance. However we could be flexible and add an additional optional parameter to extend this limitation if needed by the user.

Why is your goal for extending this limitation and optimizing over 24 h span?

[kcoffau]

I am trying to allow longer range forecasting to allow better charge/discharge of battery.

Example deeper discharge if it can see a good solar day in 2 days, or higher state of charge if solar forecast is low. I think 2-3 days would allow battery cycling to be more accurate instead of chasing end charge state 24 hours away, if it was 3 days away the end state influence would be diluted.

Agree it could be a variable to manage resource issues.

[MikaelHoogen]

+1 on this. The battery forecasting would be better. I live in Sweden with nordpool, so when we get the prices for the upcoming day there is very much use to go beyond 48 values.

For my part an optional parameter would be great. The performance shouldn't be a problem here, my server needs something to do :)

[kcoffau]

Just on performance, I am running it in a VM, so I can just allocate more resources - if needed.

I appreciate those running on devices with restricted resources will need a way of limiting the forecast either by days or intervals.

[davidusb-geek]

When these problems fall into the curse of dimensionality, the memory need grows extremely fast exponentially. Even in a machine over teraflops you are done

[davidusb-geek]

But still this feature will be added, just be warned

[davidusb-geek]

This is already a possibility. Just set delta_forecast to a bigger value, the default is 1, but you can fix it to 2, 3, etc...
Closing for now but reopen if needed.

[kcoffau]

So, let me start with david, you amazing, and thank you for pointing out the delta_forecast value. Cant believe I didnt see it.

ive extended that to 7. Im loading in 96 values, and my optimisation step is 30.

Im getting shape errors below. Cant understand why its referencing 44 or 22.

When I run 48 values in, it runs with no issue.

Trying to work out what I am doing wrong. Any ideas on starting points would be great.

CURL Source:
24 hour:
post_mpc_optim_ev_freelunch: "curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":[{% set current_time = now() %}{% set base_time = current_time.replace(minute=0, second=0, microsecond=0) if current_time.minute < 30 else current_time.replace(minute=30, second=0, microsecond=0) %}{% for offset in range(48) %}{% set nowish = base_time + timedelta(minutes=30*offset) %}{{ '0.081' if 0 <= nowish.hour < 6 else '0.00' if 11 <= nowish.hour < 14 else '0.40' }}{% if not loop.last %}, {% endif %}{% endfor %}], "prod_price_forecast":[{% for _ in range(48) %}0.08{% if not loop.last %}, {% endif %}{% endfor %}], "pv_power_forecast":{{(((state_attr('sensor.solcast_pv_forecast_forecast_today', 'detailedForecast') +(state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast'))) |selectattr('period_start', 'ge', now() - timedelta(minutes=29.99)) | map(attribute='pv_estimate')|map('multiply',1000)|map('int')|list)[:48])}}, "prediction_horizon":{{min(48,(state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)}},"soc_init":{{states('sensor.victron_battery_state_of_charge')|float(0)/100}},"def_total_hours":[0]}' http://localhost:5000/action/naive-mpc-optim"

48 Hour:
post_mpc_optim_ev_freelunch_ext: "curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":[{% set current_time = now() %}{% set base_time = current_time.replace(minute=0, second=0, microsecond=0) if current_time.minute < 30 else current_time.replace(minute=30, second=0, microsecond=0) %}{% for offset in range(96) %}{% set nowish = base_time + timedelta(minutes=30*offset) %}{{ '0.081' if 0 <= nowish.hour < 6 else '0.00' if 11 <= nowish.hour < 14 else '0.40' }}{% if not loop.last %}, {% endif %}{% endfor %}], "prod_price_forecast":[{% for _ in range(96) %}0.08{% if not loop.last %}, {% endif %}{% endfor %}], "pv_power_forecast":{{(((state_attr('sensor.solcast_pv_forecast_forecast_today', 'detailedForecast') +(state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast')) +(state_attr('sensor.solcast_pv_forecast_forecast_day_3', 'detailedForecast')) +(state_attr('sensor.solcast_pv_forecast_forecast_day_4', 'detailedForecast')) +(state_attr('sensor.solcast_pv_forecast_forecast_day_5', 'detailedForecast')) +(state_attr('sensor.solcast_pv_forecast_forecast_day_6', 'detailedForecast')) +(state_attr('sensor.solcast_pv_forecast_forecast_day_7', 'detailedForecast'))) |selectattr('period_start', 'ge', now() - timedelta(minutes=29.99)) | map(attribute='pv_estimate')|map('multiply',1000)|map('int')|list)[:96])}}, "prediction_horizon":{{min(96,(state_attr('sensor.solcast_pv_forecast_forecast_today', 'detailedForecast')|length + state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast')|length + state_attr('sensor.solcast_pv_forecast_forecast_day_3', 'detailedForecast')|length + state_attr('sensor.solcast_pv_forecast_forecast_day_4', 'detailedForecast')|length + state_attr('sensor.solcast_pv_forecast_forecast_day_5', 'detailedForecast')|length + state_attr('sensor.solcast_pv_forecast_forecast_day_6', 'detailedForecast')|length + state_attr('sensor.solcast_pv_forecast_forecast_day_7', 'detailedForecast')|length))}},"soc_init":{{states('sensor.victron_battery_state_of_charge')|float(0)/100}},"def_total_hours":[0]}' http://localhost:5000/action/naive-mpc-optim"

Curl Translated Output:
24 hour:
post_mpc_optim_ev_freelunch: "curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":[0.00, 0.00, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.00, 0.00, 0.00, 0.00], "prod_price_forecast":[0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08], "pv_power_forecast":[685, 1112, 409, 469, 847, 1182, 1581, 1275, 770, 250, 90, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 811, 1634, 2351, 2982, 3598, 4091, 4180, 4188, 4111, 3961, 3778], "prediction_horizon":32,"soc_init":0.716,"def_total_hours":[0]}' http://localhost:5000/action/naive-mpc-optim"

48hour:
post_mpc_optim_ev_freelunch_ext: "curl -i -H "Content-Type: application/json" -X POST -d '{"load_cost_forecast":[0.00, 0.00, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.40, 0.00, 0.00, 0.00, 0.00], "prod_price_forecast":[0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08], "pv_power_forecast":[685, 1112, 409, 469, 847, 1182, 1581, 1275, 770, 250, 90, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 811, 1634, 2351, 2982, 3598, 4091, 4180, 4188, 4111, 3961, 3778, 3531, 3252, 3078, 2921, 2668, 2288, 1822, 1314, 793, 310, 130, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 89, 329, 647, 902, 1070, 1140, 1217, 1293, 1357, 1274, 1037], "prediction_horizon":96,"soc_init":0.716,"def_total_hours":[0]}' http://localhost:5000/action/naive-mpc-optim"

EMHASS LOG:
2024-04-02 13:22:59,219 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.0, 0.0, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.081, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.0, 0.0, 0.0, 0.0], 'prod_price_forecast': [0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08], 'pv_power_forecast': [685, 1112, 409, 469, 847, 1182, 1581, 1275, 770, 250, 90, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 811, 1634, 2351, 2982, 3598, 4091, 4180, 4188, 4111, 3961, 3778, 3531, 3252, 3078, 2921, 2668, 2288, 1822, 1314, 793, 310, 130, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 89, 329, 647, 902, 1070, 1140, 1217, 1293, 1357, 1274, 1037], 'prediction_horizon': 96, 'soc_init': 0.705, 'def_total_hours': [0]}
2024-04-02 13:22:59,219 - web_server - INFO - >> Setting input data dict
2024-04-02 13:22:59,220 - web_server - INFO - Setting up needed data
2024-04-02 13:22:59,231 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-02 13:23:01,863 - web_server - INFO - Retrieving weather forecast data using method = list
2024-04-02 13:23:01,865 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-04-02 13:23:01,866 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-02 13:23:08,765 - web_server - INFO - >> Performing naive MPC optimization...
2024-04-02 13:23:08,765 - web_server - INFO - Performing naive MPC optimization
2024-04-02 13:23:08,776 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 145, in action_call
opt_res = naive_mpc_optim(input_data_dict, app.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 273, in naive_mpc_optim
df_input_data_dayahead = input_data_dict['fcst'].get_load_cost_forecast(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 715, in get_load_cost_forecast
forecast_out = self.get_forecast_out_from_csv(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 530, in get_forecast_out_from_csv
forecast_out = pd.DataFrame(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 758, in init
mgr = ndarray_to_mgr(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 337, in ndarray_to_mgr
_check_values_indices_shape_match(values, index, columns)
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 408, in _check_values_indices_shape_match
raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (44, 1), indices imply (22, 1)

Update:
I updated the 48 code to have a prediction horizon of 48. Even though it had 96 values. So the prediction horizon is my issue. continuing troubleshooting.

[rixxxx]

@kcoffau let us know if you figure it out. I have similar issue when trying 26h forecast with delta_forecast 2.
emhass-1 | ValueError: Shape of passed values is (4, 1), indices imply (2, 1)

[davidusb-geek]

Ok so I went down to this and this is actually a bug!
Actually I realized that we never tested the code well for delta_forecast > 1
Working on a fix right now

[davidusb-geek]

Done!
Normally solved here: #250
Should be available on the next release

[rixxxx]

My 34h forecast now works. Thanks.

[kcoffau]

0.8.6

2 days forecast!!!

Absolutely beautiful. Thank you David!

[purcell-lab]

After 12.30pm each day our energy market operator (AEMO) publishes 30 minute forecasts until 3.30am the day after the following day (ie for the next 39 hours), they then update this set of forecasts every 5 minutes, always finishing at 3.30am.

EMHASS can now consume the full 39 hours and with MPC can update it's optimisation every time the pricing changes.

Thanks @davidusb-geek .