OZW light transition values are no longer compatible when using OpenZWave addon v0.5.2

[kpine]

The problem

The OpenZWave addon v0.5.2 was updated to use the latest version of open-zwave. The latest open-zwave code has changed the API for dim duration values, so the light integration needs to be updated as well.

Environment

• Home Assistant Core release with the issue: All OZW versions, with OpenZWave addon v0.5.2. Standalone docker users with build-150 are not affected. Build-150 and later have the same problem.
• Last working Home Assistant Core release (if known): N/A
• Operating environment (OS/Container/Supervised/Core): N/A
• Integration causing this issue: ozw
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/ozw/

Problem-relevant configuration.yaml

N/A

Traceback/Error logs

N/A

Additional information

A recent commit in open-zwave changed the type of the dim duration value for dimmer switches.

Previously it was a byte value that was encoded by the application into the z-wave values. Now the value is seconds, and open-zwave encodes it to the correct z-wave value.

The value type change affects the factory transition value (255) and any transition values > 127. For example, when light.turn_on is used w/o a brightness, ozw encodes that as 255, which means factory default. With the new API that means 255 seconds.

The factory default value is not well defined. I think there are still bugs in this area of code, so it is unknown if any other changes will follow. In the current implementation, the integration could choose any value > 15240 to indicate factory default.

[kpine]

The problem can be worked around by setting transition in any automations to a sensible value, e.g. for toggle on/off use 0. Any values < 128 are fine and will coincidentally just be treated as seconds.

Toggling the switch on/off in the UI will still be affected.

[probot-home-assistant]

ozw documentation
ozw source
_{^{(message by IssueLinks)}}

[probot-home-assistant]

Hey there @cgarwood, @marcelveldt, @MartinHjelmare, mind taking a look at this issue as its been labeled with an integration (ozw) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[Fishwaldo]

Just FYI - Give me a bit to fix up a few things here, and I'll post when we have settled on the "factory default" value

[seandop]

Good find, @kpine! This explains why my lights all of a sudden are taking several minutes to respond...

[kpine]

Another observation, because of the value change the cache version was updated. Looks like the cache load does not gracefully handle incremental changes and discards the entire cache. This is basically the same as losing the cache file and it needs to be rebuilt. Mains-powered devices will be found automatically. For battery devices this means waking them up manually, or waiting for them to eventually wake up (if they do). Some users rely on the naming feature of OZW, and will notice those names are lost.

[warning]: Warning - Node: 0 WARNING: Driver::ReadCache - /opt/ozw/user/ozwcache_0xc85ce84b.xml is from an older version of OpenZWave and cannot be loaded.

[marcelveldt]

Another observation, because of the value change the cache version was updated. Looks like the cache load does not gracefully handle incremental changes and discards the entire cache. This is basically the same as losing the cache file and it needs to be rebuilt. Mains-powered devices will be found automatically. For battery devices this means waking them up manually, or waiting for them to eventually wake up (if they do). Some users rely on the naming feature of OZW, and will notice those names are lost.

[warning]: Warning - Node: 0 WARNING: Driver::ReadCache - /opt/ozw/user/ozwcache_0xc85ce84b.xml is from an older version of OpenZWave and cannot be loaded.

Oops, good catch, that will cause a lot of headache.

[Fishwaldo]

Oops, good catch, that will cause a lot of headache.

No choice unfortunately - I've had to change the Duration ValueID to a Int (from a byte) and if loading from a cache that has the old encoding, it would crash OZW.

[Fishwaldo]

Just FYI - Give me a bit to fix up a few things here, and I'll post when we have settled on the "factory default" value

So anything larger than 7620 will setup OZW to use the device default duration - OZW will actually report it as 4294967295.

This is pushed upstream and is pending a few tests from people to confirm all good.

[marcelveldt]

Oops, good catch, that will cause a lot of headache.

No choice unfortunately - I've had to change the Duration ValueID to a Int (from a byte) and if loading from a cache that has the old encoding, it would crash OZW.

That's too bad. Some users (including me) have lot's of battery powered nodes. Most of them do not wake up at all or very sporadic. Going to the trouble of manually refreshing them is a bit of a pain. Let alone the fact that they do not know about the fact this is needed after an update.

These kind of breaking change (if really not unavoidable) should be communicated. We can do that in the add-on release notes for example.

[Fishwaldo]

These cache invalidation are few and far between. Last time I did it was 1.4 -> 1.6 transition. So 1 1/2 years since then.

But yeah - Need better coordination between upstream and HA addon

[jerr-wol]

Aeotec Dimmer ZW-111 is also exhibiting dimming issues after update. I can still toggle the switch entity with out an issue, but dimming in its entirety is broken, including toggling on and off using the dimmer entity as well as % dimming.

[foxy82]

These cache invalidation are few and far between. Last time I did it was 1.4 -> 1.6 transition. So 1 1/2 years since then.

But yeah - Need better coordination between upstream and HA addon

Is it possible to transition the cache manually ourselves or is that too complex to explain?

[uSpike]

These cache invalidation are few and far between. Last time I did it was 1.4 -> 1.6 transition. So 1 1/2 years since then.
But yeah - Need better coordination between upstream and HA addon

Is it possible to transition the cache manually ourselves or is that too complex to explain?

You can copy the ozwcache_xxxxxxxx.xml file and reference it later.

[kpine]

@foxy82 Should be possible. Could probably be scripted via Python or using something like xmlstarlet. The challenge will be for addon users because the cache file is not easy to access. If you are using Docker standalone you can test it quite easily.

It might be easier to approach it the other way, let OZW upgrade the cache file. The mains-powered devices will be detected correctly (with loss of names) and the battery devices will be missing. Shut down OZW and copy the missing nodes from the previous version of the cache file into the newly upgraded one, and fix any names that were lost, if you use those. These will mostly be battery devices, and likely not be impacted by the changes (but you'll have to double check). Using a diff tool would be the way to go.

I think these are the changes between 4 and 5:

The config version changed from 4 to 5, version="5".

<Driver xmlns="https://github.com/OpenZWave/open-zwave" version="4" revision="109" home_id="0xc85ce84b" node_id="1" api_capabilities="8" controller_capabilities="28" poll_interval="30000" poll_interval_between="false">

<Driver xmlns="https://github.com/OpenZWave/open-zwave" version="5" revision="110" home_id="0xc85ce84b" node_id="1" api_capabilities="8" controller_capabilities="28" poll_interval="30000" poll_interval_between="false">

The MULTILEVEL command class "Dimming Duration" and COLOR command class "Duration" values changed from type byte to int. That's where the majority of changes would be. If you don't have any of these devices (usually dimmer switches and RGB bulbs) then there's probably nothing else to do. Here's an example.

byte (ignore the Help):

<Value type="byte" genre="system" instance="1" index="5" label="Dimming Duration" units="" read_only="false" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="255" value="255">
	<Help>Duration taken when changing the Level of a Device (Values above 15240 use the devices default duration)</Help>
</Value>

int:

<Value type="int" genre="system" instance="1" index="5" label="Dimming Duration" units="" read_only="false" write_only="false" verify_changes="false" poll_intensity="0" min="-2147483648" max="2147483647" value="0">
	<Help>Duration taken when changing the Level of a Device (Values above 15240 use the devices default duration)</Help>
</Value>

type, min and max are what you'd need to change.

[kpine]

So anything larger than 7620 will setup OZW to use the device default duration - OZW will actually report it as 4294967295.

@Fishwaldo This doesn't seem to be true. If you set 7621 it is reported as 7621. It's a signed integer so you also can't even set anything above 2147483647. If you set 2147483647 it reports 2147483647. Not sure it matters, since all those numbers result in the factory default, but is that a surprise?

Do you have a recommended number that should be used to set the factory default? 7621? The max value? -1?

[JT114881]

What release will this be included in?

[jef-pearlman]

What release will this be included in?

PR has been tagged for 0.113.2: #38254

[jef-pearlman]

Since this didn't make it in to 0.113.2 (#38254 (comment)), is there any chance of rolling back the ozw version in the addon until it's fixed in HA core? This issue has broken a bunch of my lighting. (Not sure if that means it should be reopened or not.) Thanks!

[jef-pearlman]

Since this didn't make it in to 0.113.2 (#38254 (comment)), is there any chance of rolling back the ozw version in the addon until it's fixed in HA core? This issue has broken a bunch of my lighting. (Not sure if that means it should be reopened or not.) Thanks!

Just noting that dropping in the dev branch of the ozw component into custom_components seems to be working fine as a workaround for me until the fix is released. (Thanks to the folks on Discord!)

[sdujolo]

Since this didn't make it in to 0.113.2 (#38254 (comment)), is there any chance of rolling back the ozw version in the addon until it's fixed in HA core? This issue has broken a bunch of my lighting. (Not sure if that means it should be reopened or not.) Thanks!

Just noting that dropping in the dev branch of the ozw component into custom_components seems to be working fine as a workaround for me until the fix is released. (Thanks to the folks on Discord!)

Can you tell med more how to do the workaround?

[Michsior14]

Since this didn't make it in to 0.113.2 (#38254 (comment)), is there any chance of rolling back the ozw version in the addon until it's fixed in HA core? This issue has broken a bunch of my lighting. (Not sure if that means it should be reopened or not.) Thanks!

Just noting that dropping in the dev branch of the ozw component into custom_components seems to be working fine as a workaround for me until the fix is released. (Thanks to the folks on Discord!)

Can you tell med more how to do the workaround?

Just copy the ozw folder from dev branch into yours custom_components folder and restart HA. It will be picked up instead of the system version. Remember to remove it once the commit will be cherry picked (probably in version 0.114).

[simulot]

I have used an other method to work around this issue.
The dimmer is correctly handled when calling the light.turn_on with duration=0

I have detailed the method here: OpenZWave/qt-openzwave#137 (comment)