Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible bug after upgrade from old version #50

Closed
ldotlopez opened this issue Jun 25, 2023 · 12 comments
Closed

Possible bug after upgrade from old version #50

ldotlopez opened this issue Jun 25, 2023 · 12 comments

Comments

@ldotlopez
Copy link
Owner

ldotlopez commented Jun 25, 2023 •
edited
Loading

I have detected some problems with dirty (vs. fresh installs) due new statistics calculation method.
This error is related to upgraded installations from previous version, not fresh ones.

Errors logs shows exceptions like KeyError related to 'sum' or incompatible types in '+' operator.

While I work on a stable solution here are some tips if you're confident with command line. Each one is individual, if option A works for you just stop mangling things. If not, try next option if you understand what are you doing.

Remember to backup everything before proceed. I can't give support beyond fixing the upgrade path, recovering from failed attempt to fix is outside limits.

A. Try to update to v2.0.1

B. Clean up your database:

  • Write down related historical sensors: everything like sensor.esXXXX_historical
  • Stop homeassistant
  • Open the sqlite prompt
    sqlite 'config/home-assistant_v2.db'
  • Get the metadata_id of each sensor:
    select id from statistics_meta where statistic_id = 'sensor.esXXXXX_historical;' (mine is 114)
  • Update statistics_meta:
    update statistics_meta set has_mean=1, has_sum=1 where id = <your_id>; (remember to replace <your_id> with yours)
  • Delete incorrect values from statistics:
    delete from statistics where (mean is null or sum is null or state is null) and metadata_id = <your_id>;
  • Start home-assistant

C. Wipe database: Just delete (ore temporally rename) the config/home-assistant_v2.db file, you will lose some history and stats, but anything else will stay.

D. Wait. Statistics are a complex stuff and importing data from past makes it worse. I'm trying my best to fix upgrade path
@ldotlopez ldotlopez mentioned this issue Jun 25, 2023
Closed
@sirdir1972
Copy link

I think you are missing a ‘ in the select id line? Anyways I don‘t even get there because:
Unable to open database "home-assistant_v2.db": file is encrypted or is not a database
I thought it might be corrupted and did ‚C‘, but so far it still doesn‘t seem to work. And I get the same message when I try to open the db with sqlite.

ldotlopez added a commit that referenced this issue Jun 25, 2023
@ldotlopez
Try to address bug #50
d364e64
@ldotlopez
Copy link
Owner Author

ldotlopez commented Jun 25, 2023 •
edited
Loading

Hi @sirdir1972

I pushed a commit into main that incorporates all that stuff into the integration and warns about the fixes. It doesn't fixes it yet, I need to confirm some stuff.

Can you try the "v2.0.2a1" following these instructions?

  1. Before upgrade please add those lines to your configuration.yaml:
logger:
  logs:
    homeassistant_historical_sensor: debug
    custom_components.ideenergy: debug
    custom_components.ideenergy.fixes: debug

  1. Download the beta release "v2.0.2a0"
    imagen
    imagen

  2. Post log lines with 'custom_components.ideenergy' or 'homeassistant_historical_sensor'

Thanks

@sirdir1972
Copy link

sirdir1972 commented Jun 25, 2023 via email

@sirdir1972
Copy link

same btw if I delete the db:
2023-06-25 20:13:04.301 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Registered sensor 'AccumulatedConsumption'
2023-06-25 20:13:04.304 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Request update for datasets: MEASURE
2023-06-25 20:13:04.305 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] update denied for MEASURE: update window is closed
2023-06-25 20:13:04.305 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Finished fetching xx/xxx coordinator data in 0.001 seconds (success: True)

@sirdir1972
Copy link

Later:
2023-06-25 20:53:34.433 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Request update for datasets: MEASURE
2023-06-25 20:53:34.434 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] update allowed for MEASURE
2023-06-25 20:53:35.294 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] update error for MEASURE: Forbidden (403)
2023-06-25 20:53:35.294 DEBUG (MainThread) [custom_components.ideenergy.datacoordinator] Finished fetching xxx coordinator data in 0.861 seconds (success: True)

@ldotlopez
Copy link
Owner Author

Some tips:

  • I think you don't have the historical consumption sensor enabled. There should be a "HISTORICAL_CONSUMPTION" request in the logs.
  • You have to configure energy panel to use some of the i-de sensor as source, i.e. sensor.es0022xxxxx_historical
  • Historical data is updated at startup and after 6 hours, also, i-de website can have a delay up to 24h for historical data.

@sirdir1972
Copy link

sirdir1972 commented Jun 26, 2023 via email

@sirdir1972
Copy link

sirdir1972 commented Jun 26, 2023 via email

@sirdir1972
Copy link

sirdir1972 commented Jun 26, 2023 via email

@ldotlopez
Copy link
Owner Author

Hi,

I'm actively working on this.

I just pushed some changes into 'main' that should log some warning messages if your stats are broken.
If you are affected by this bug, check the 'main' version and post your logs.

Warning: there are lots of situations where you are not getting statistics but not because of this bug. Check:

  • You have enabled historical sensors
  • Historical states can have a 24h delay, go one or two days back in the graphs to check it.
  • Historical states are refreshed at startup and each 6 hours.
  • Enable debug logs for this integration, check if there are errors
  • "HISTORICAL_GENERATION: Forbidden (403)" it's OK, i-de.es website is picky.
  • "MEASURE: update window is closed" it's OK.

ldotlopez added a commit that referenced this issue Jun 29, 2023
@ldotlopez
More on bug #50: More on bug #50: Implemented an idempotent mechanism…
beaaeaa 
… to fix statistics

Somehow somewhere Home Assistant writes invalid statistics into the
recorder, possibility because historical sensors are on a undefined
state forever.

The (unfinished) fixes.async_fix_statistics takes care of keeping stats healhty.
ldotlopez pushed a commit that referenced this issue Jun 30, 2023
@lopezl
More on #50: Check for continuously increasing sum values
cb61eab
ldotlopez added a commit that referenced this issue Jun 30, 2023
@ldotlopez
Bug #50: fix wrong int/float conversion
eab4dae
@gaRFio01
Copy link

Same problem here…

@gaRFio01
Copy link

Creo que el problema está en la lectura desde el contador. Hace algún tiempo que no puedo recoger datos ni desde la propia aplicación de i-DE. He abierto una incidencia e indican que hay problemas de comunicación con el contador…
Es posible que tenga algo que ver?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ldotlopez @sirdir1972 @gaRFio01