DSMR reader sensor has state class total_increasing, but its state is not strictly increasing

[hermanops]

The problem

Entity sensor.dsmr_reading_electricity_delivered_1 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing

What is version of Home Assistant Core has the issue?

core-2021.9.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

sensor recorder

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:262
Integration: Sensor (documentation, issues)
First occurred: 22:47:22 (1 occurrences)
Last logged: 22:47:22

Entity sensor.dsmr_reading_electricity_delivered_1 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+recorder%22

Additional information

No response

[probot-home-assistant]

dsmr_reader documentation
dsmr_reader source
_{^{(message by IssueLinks)}}

[probot-home-assistant]

Hey there @depl0y, mind taking a look at this issue as it has been labeled with an integration (dsmr_reader) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[probot-home-assistant]

recorder documentation
recorder source
_{^{(message by IssueLinks)}}

[kroona]

2021-12-14 16:05:10 WARNING (Recorder) [homeassistant.components.sensor.recorder] Entity sensor.dsmr_reading_electricity_delivered_2 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 2999.258 with last_updated set to 2021-12-14T15:00:56.217526+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22

[Diondk]

Same error here:

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:319
Integration: Sensor (documentation, issues)
First occurred: 09:55:10 (1 occurrences)
Last logged: 09:55:10

Entity sensor.dsmr_reading_electricity_delivered_2 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 4506.090 with last_updated set to 2021-12-21T08:50:31.180099+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22

[DavyRoswinkel]

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:319
Integration: Sensor (documentation, issues)
First occurred: 00:05:11 (1 occurrences)
Last logged: 00:05:11

Entity sensor.dsmr_reading_electricity_delivered_1 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 7801.788 with last_updated set to 2022-01-29T23:00:28.663342+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22

[hermanops]

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:311
Integration: Sensor (documentation, issues)
First occurred: 08:23:27 (3 occurrences)
Last logged: 08:25:10

Entity sensor.dsmr_reading_electricity_delivered_1 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 4553.003 (4553.003) with last_updated set to 2022-02-06T07:23:26.976102+00:00.

Entity sensor.dsmr_day_consumption_electricity1 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 2.551 (2.554) with last_updated set to 2022-02-06T07:23:26.077709+00:00.

Entity sensor.dsmr_day_consumption_electricity_merged from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 2.551 (2.554) with last_updated set to 2022-02-06T07:23:26.063985+00:00.

[kroona]

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:311
Integration: Sensor (documentation, issues)
First occurred: 13:40:10 (2 occurrences)
Last logged: 14:25:10

Entity sensor.dsmr_reading_electricity_returned_2 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 1082.053 (1082.055) with last_updated set to 2022-03-17T12:35:44.433586+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22
Entity sensor.dsmr_reading_electricity_delivered_2 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 3361.091 (3361.093) with last_updated set to 2022-03-17T13:23:05.915098+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22

[ArNRDS]

Same here:
Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:311
Integration: Sensor (documentation, issues)
First occurred: 7:30:10 PM (3 occurrences)
Last logged: 7:30:10 PM

Entity sensor.dsmr_reading_electricity_delivered_2 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 100.204 (100.205) with last_updated set to 2022-05-20T17:26:53.939223+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22
Entity sensor.dsmr_day_consumption_electricity2 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 1.157 (1.173) with last_updated set to 2022-05-20T17:27:36.669277+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22
Entity sensor.dsmr_day_consumption_electricity_merged from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 4.017 (4.033) with last_updated set to 2022-05-20T17:27:36.201841+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22

[Glodenox]

I fear this is going to be a difficult one to fix properly. We can't just change the total_increasing state class to the default measurement state class. That would make it so that meter resets are no longer seen as a reset by HA, but as an actual decrease in value, which will probably have negative side effects in the energy dashboard.

The cause of this issue seems to mostly be slightly invalid readings received from the net meters. The only workaround I can think of right now would be storing the previous value for those sensors and ignoring a new value if it's only off by a certain small percentage. I'm just not sure whether that would be an acceptable solution, as you'd be faking data.

I'm mostly surprised by the values from @ArNRDS, as these are larger jumps than any of the other values I've seen so far. The consumption_electricity2 is almost 2% off. If you still have the data: could you maybe check the graph of that entity's value around that time to see whether it was just a short downwards spike in the readings? I'm presuming it jumped up again quite soon afterwards.

[Glodenox]

I've got some code running locally that ignores any values that are less than 2% lower than the previous value. This has silenced the warnings completely, while still allowing the resets to 0. For my meter, it seems to have this issue like 5 times per hour. I'm just not certain yet whether this is a desirable solution.

[ArNRDS]

Had it again a few days back. Will check the graph

Logger: homeassistant.components.sensor.recorder
Source: components/sensor/recorder.py:311
Integration: Sensor (documentation, issues)
First occurred: June 5, 2022, 2:05:10 AM (1 occurrences)
Last logged: June 5, 2022, 2:05:10 AM

Entity sensor.dsmr_reading_electricity_delivered_1 from integration dsmr_reader has state class total_increasing, but its state is not strictly increasing. Triggered by state 217.372 (217.373) with last_updated set to 2022-06-05T00:01:02.860359+00:00. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+dsmr_reader%22

Graph does not show anything strange

[Glodenox]

This issue wasn't fully resolved with the recent change. There are still other entities that also encounter this behaviour, like sensor.dsmr_reading_electricity_delivered_2. Taking into account the issue described in #75924, I'm starting to become less and less in favor of the removal of the state classes from those entities.

[frenck]

@Glodenox 75924 is reported and an issue that is going to be resolved. That is a general thing.
In the end, this is actually a pretty serious bug in the DSMR Reader, as these meters its monitors are not capable of going backward...

So the real issue is in the upstream software.

This issue does not occur using the same energy meters using the native DSMR integration or, e.g., the native ESPHome components.

[Glodenox]

@frenck I'll check with the people of DSMR Reader then. I suspect they're just passing on the data frames they're getting as-is, but maybe there's something else going on. But then it would be best to temporarily(?) remove the TOTAL_INCREASING state class from those other entities as well, as I'm still getting warning logs for some of those. So far I've noticed sensor.dsmr_reading_electricity_delivered_2 and sensor.dsmr_reading_electricity_returned_1 giving the same error.

[frenck]

But then it would be best to temporarily(?) remove the TOTAL_INCREASING state class from those other entities as well

No, it is best if they fix the issue.

I've removed the state class from the daily ones. Besides them having this issue more obvious (as they are calculated values), they are also not needed in Home Assistant, as the total increasing lifetime meter values are also provided (and preferred to use for things like energy monitoring in Home Assistant). Removing the state class from daily aggregation is done from that perspective.

To me, it looks like they have rounding/calculation issues. I have not come across false readings from any DSMR meter in years. In more spectacular detail: they use the exact same library for parsing the telegrams as we do in our native integration.

I'm pretty sure it's a calculation/data processing issue.