[BUG] Zigbee2MQTT breaking changes

[fir3drag0n]

Bug report

As Zigbee2MQTT 1.42. or 2.0 come with breaking changes (Koenkk/zigbee2mqtt#24198), it also breaks some controllers, such as E1743

Description

The controller does not work anymore.

Additional information

• Devices involved:
  • Model: eg. IKEA E1743
  • ...
• Integration: z2m
• AppDaemon version: v4.4.2
• HACS version (if installed from there): 2.0.1
• Home Assistant Core version: 2024.11.3

AppDaemon app configuration

nachttisch_links:
  module: controllerx
  class: E1743Controller
  integration: z2m
  controller: sensor.dim_nachttisch_links_action
  light: light.hue_color_lamp_1

nachttisch_rechts:
  module: controllerx
  class: E1743Controller
  integration: z2m
  controller: sensor.dim_nachttisch_rechts_action
  light: light.schlafzimmer_nachttisch_rechts

wohnzimmer_dim:
  module: controllerx
  class: E1743Controller
  integration: z2m
  controller: sensor.dim_wohnzimmer_action
  light: light.wohnzimmer_decke
  
flur_dim:
  module: controllerx
  class: E1743Controller
  integration: z2m
  controller: sensor.dim_flur3_action
  light: light.flur_deckenlampe

kueche_dim:
  module: controllerx
  class: E1743Controller
  integration: z2m
  controller: sensor.dim_kueche_action
  light: light.unterschranke_kuche

[fir3drag0n]

https://www.zigbee2mqtt.io/guide/usage/integrations/home_assistant.html#via-mqtt-device-trigger-recommended

MQTT device triggers or event entities are suggested

[xaviml]

Hi @fir3drag0n ,

Z2M is removing the connection via sensors, and recommend using events. ControllerX allows connecting directly via MQTT, so I recommend you changing to that this type of configuration.

You can read more about mqtt config migration here:

https://controllerx.netlify.app/controllerx/others/zigbee2mqtt-light-controller/

Regarding E1743, I still have to see what has it changed, so I leave this ticket open.

Regards,
Xavi M.

[Crocmagnon]

v2.0.0 of Z2M has been released.

Here's what I had to do:

1. Enable MQTT plugin, see below for a config example
2. Restart AppDaemon
3. Migrate my z2m apps to mqtt

Appdaemon config

appdaemon:
  # ...
  plugins:
    # ...
    MQTT:
      type: mqtt
      namespace: mqtt
      client_host: core-mosquitto
      client_user: mqtt
      client_password: password
      # Listen only to zigbee2mqtt messages
      client_topics:
        - zigbee2mqtt/#

Apps configuration

Before

livingroom_controller:
  module: controllerx
  class: E1810Controller
  controller: sensor.livingroom_controller_action
  integration: z2m
  light: light.bedroom

After

livingroom_controller:
  module: controllerx
  class: E1810Controller
  controller: Livingroom Controller # this is the "friendly name" of the device in Z2M
  integration:
    name: z2m
    listen_to: mqtt
  light: light.bedroom

Suggestions

I suggest updating the second code sample here, it wasn't immediately clear that the controller key had changed, nor that the device's friendly name was expected.
Maybe also setup a Z2M 2.0 migration page in the docs?

MQTT error

I got the following errors in the logs. Looks like it ended up successful (Plugin initialization complete) but still, a bit scary.

The warning disappeared when listening only to z2m messages (client_topics config key).

Logs

2025-01-03 23:48:22.165394 INFO MQTT: Connected to Broker at URL core-mosquitto:1883
2025-01-03 23:48:22.555183 INFO MQTT: Unable to decode MQTT message
2025-01-03 23:48:22.557665 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
2025-01-03 23:48:22.560667 INFO MQTT: Unable to decode MQTT message
2025-01-03 23:48:22.562073 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
2025-01-03 23:48:22.565533 INFO MQTT: Unable to decode MQTT message
2025-01-03 23:48:22.566949 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
2025-01-03 23:48:22.702510 INFO AppDaemon: Got initial state from namespace mqtt
2025-01-03 23:48:22.703181 INFO MQTT: MQTT Plugin initialization complete

Edited on 2025-01-04: add appdaemon config example + update on MQTT warning

[AlexL00]

Here's what I had to do:

1. Enable MQTT plugin
2. Restart AppDaemon
3. Ignore MQTT errors in logs (see note below)
4. Migrate my z2m apps to mqtt

Thanks for the guidance! As I'm and likely many others, using the mqtt add-on it might be a good idea to add information about the client_host configuration key in the appdeamon.yaml file. I had to do it like this:

appdaemon:
  latitude: xx.xxxxxx
  longitude: xx.xxxxxx
  elevation: x
  time_zone: xx/xx
  plugins:
    HASS:
      type: hass
    MQTT:
      type: mqtt
      namespace: mqtt
      client_host: core-mosquitto
      client_user: mqtt
      client_password: strong_secret_from_app_deamon_add-on_configuration_page

[jeroenbeuz]

MQTT warning note: I got the following errors in the logs. Looks like it ended up successful (Plugin initialization complete) but still, a bit scary.
Logs

2025-01-03 23:48:22.165394 INFO MQTT: Connected to Broker at URL core-mosquitto:1883
2025-01-03 23:48:22.555183 INFO MQTT: Unable to decode MQTT message
2025-01-03 23:48:22.557665 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^

The errors you're getting might be related to message formats which AppDaemon does not understand, like frigate streams. You could try and specify the 'client_topics' so it only subscribes to what it needs. Here's what I have:

client_topics:
    - zigbee2mqtt/#

I still can't get this to work unfortunately. It successfully subscribes to z2m, but the actions are not executed somehow. Nothing in the logs. Any ideas?

My app config:

studiekamer_controller:
  module: controllerx
  class: Philips929002398602LightController
  controller: Studiekamer Dimmer Switch
  integration:
    name: z2m
    listen_to: mqtt
  light: light.studiekamer_lampen_dimmer

[Crocmagnon]

Thank you @AlexL00 & @jeroenbeuz, I updated my message above.

the actions are not executed [...] Any ideas?

Unfortunately no, sorry.

[phdelodder]

I have now changed it, however the response is much slower. Also dimming doesn't seem to working any more. Using:

dressing_spots_controller_night:
  module: controllerx
  class: LightController
  controller: Dressing Switch
  integration:
    name: z2m
    listen_to: mqtt
  light: light.dressing_spots
  constrain_input_boolean: input_boolean.night_mode
  mapping:
    press_1:
      service: light.toggle
      data:
        entity_id: light.dressing_spots
        brightness_pct: 20
    press_2: hold_brightness_toggle

[fribse]

@xaviml Isn't it a question of "just" using the same mechanism as for ZHA, I beleive ZHA uses events as well?
Not that I expect you can just enter ZHA in the integration instead of z2m?
I think going with MQTT is a poor solution, and will make it too complicated as well as slow things down quite a bit.
You know I've been a big advocate for controllerx for a long time, but this is not a good solution I think, and will make it very unattractive for new users.

[jeroenbeuz]

I still can't get this to work unfortunately. It successfully subscribes to z2m, but the actions are not executed somehow. Nothing in the logs. Any ideas?

My app config:

studiekamer_controller:
  module: controllerx
  class: Philips929002398602LightController
  controller: Studiekamer Dimmer Switch
  integration:
    name: z2m
    listen_to: mqtt
  light: light.studiekamer_lampen_dimmer

Finally fixed the issue. I reinstalled AppDaemon and ControllerX after it stopped working due to the z2m update. I then added my applications to the \addon_configs\a0d7b954_appdaemon\apps\apps.yaml file in stead of the /homeassistant/appdaemon/apps/apps.yaml file. To fix this, I copied the content to the file in the homeassistant folder and added the following line to the appdaemon.yml file:

app_dir: /homeassistant/appdaemon/apps

[audun-v]

MQTT warning note: I got the following errors in the logs. Looks like it ended up successful (Plugin initialization complete) but still, a bit scary.
Logs

2025-01-03 23:48:22.165394 INFO MQTT: Connected to Broker at URL core-mosquitto:1883
2025-01-03 23:48:22.555183 INFO MQTT: Unable to decode MQTT message
2025-01-03 23:48:22.557665 ERROR MQTT: Unable to decode MQTT message, with Traceback: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/plugins/mqtt/mqttplugin.py", line 271, in mqtt_on_message
    payload = payload.decode()
              ^^^^^^^^^^^^^^^^

The errors you're getting might be related to message formats which AppDaemon does not understand, like frigate streams. You could try and specify the 'client_topics' so it only subscribes to what it needs. Here's what I have:

client_topics:
    - zigbee2mqtt/#

I still can't get this to work unfortunately. It successfully subscribes to z2m, but the actions are not executed somehow. Nothing in the logs. Any ideas?

My app config:

studiekamer_controller:
  module: controllerx
  class: Philips929002398602LightController
  controller: Studiekamer Dimmer Switch
  integration:
    name: z2m
    listen_to: mqtt
  light: light.studiekamer_lampen_dimmer

Which apps.yaml file did you edit?
I had the same problem. I had to edit the apps.yaml located here:
\addon_configs\a0d7b954_appdaemon\apps
not here:
\config\appdaemon\apps
Restart AppDeamon after config change.

You can see if the config has taken effect if you go here:
your.home.assistant.ip:5050/aui/index.html#/state?tab=apps

[deadphill]

Hi @fir3drag0n ,

Z2M is removing the connection via sensors, and recommend using events. ControllerX allows connecting directly via MQTT, so I recommend you changing to that this type of configuration.

You can read more about mqtt config migration here:

https://controllerx.netlify.app/controllerx/others/zigbee2mqtt-light-controller/

Regarding E1743, I still have to see what has it changed, so I leave this ticket open.

Regards, Xavi M.

Hi @xaviml

Issue here is changing to Z2M means that colour looping doesn't work:
https://community.home-assistant.io/t/controllerx-bring-full-functionality-to-light-and-media-player-controllers/148855/1417

Are there any suggestions to enable this using any other method?

Thanks

Phill

[deadphill]

All

I have just found an option within zigbee2mqtt that allows for the old action entities to continue to work. You just have to enable it.

Go to zigbee2mqtt, and select settings - home assistant integration and then select the option "Home Assistant legacy action sensors". This will then allow your setup to continue to work. How much longer for I have no idea as this might go in the future but I hope it doesn't. I have multiple actions set for my remotes and reverting to z2m controls means I somehow have to get the devices to work with mqtt which at this moment in time I have no idea about, or somehow work out how to call an event rather than an action and I have no idea if this would even work with controllerx.

Hopefully this helps someone out there.. Ive been pulling what hair I have left out until I found this in a reddit forum!

Cheers

Phill

[Crocmagnon]

That's already documented here: Koenkk/zigbee2mqtt#24198

All action sensors are now disabled by default (sensor.*_action entities). It's recommended to use the MQTT device trigger instead.

But since it's a legacy option, we should probably find a way to not rely on it too much.

[Crocmagnon]

@xaviml I'm facing an issue with the following config:

ampoule_chambre:
  module: controllerx
  class: E1810Controller
  controller: Interrupteur chambre
  integration:
    name: z2m
    listen_to: mqtt
  light: light.ampoule_chambre
  merge_mapping:
    "toggle$2": on_full_brightness
    arrow_left_click:
      - service: switch.toggle
        data:
          entity_id: switch.guirlande_chambre
    arrow_right_click:
      - service: switch.toggle
        data:
          entity_id: switch.guirlande_chambre
    toggle_hold: []
    arrow_left_hold: []
    arrow_left_release: []
    arrow_right_hold: []
    arrow_right_release: []

The intent is to use a single press on the "power" button to toggle the light and a double press to trigger full brightness.
However, a single click on the button shows this in the logs:

2025-01-06 13:31:44.560420 INFO ampoule_chambre: 🎮 Button event triggered: `toggle$2`
2025-01-06 13:31:44.567770 INFO ampoule_chambre: 🏃 Running `Predefined (on_full_brightness)` now
2025-01-06 13:31:44.580925 INFO ampoule_chambre: 
🤖 Service: light.turn_on
  - entity_id: light.ampoule_chambre
  - brightness: 255

If I remove the toggle$2 config, then the button works properly.

This didn't happen with the action sensor.

[fribse]

I ran into trouble with node-red, and found out that there is a setting in Z2M that should be activated:

According to KennethLavrsen it is no longer experimental.
It's a rather cool addition, it will create an entity for the event.
So instead of a sensor, it's doable with an event entity.
The value of the button in Node Red is in: data.new_state.attributes.event_type
A cool benefit is that if you go to Developer Tools -> States and look up the device, it shows the possible values provided by the button.

Maybe ControllerX could use this as well @xaviml ?

[Batou38]

Hi @fir3drag0n ,

Z2M is removing the connection via sensors, and recommend using events. ControllerX allows connecting directly via MQTT, so I recommend you changing to that this type of configuration.

You can read more about mqtt config migration here:

https://controllerx.netlify.app/controllerx/others/zigbee2mqtt-light-controller/

Regarding E1743, I still have to see what has it changed, so I leave this ticket open.

Regards, Xavi M.

This was very helpfull for me, after a couple of hours I realised that friendly names in Z2M are :
- Case sensitive
- Can have "space" instead of underscore....
- and when switching to mqtt we have to be very cautious with that.

Now everything is working perfectly fine.

Here below 2 examples

ChambreParents:
  module: controllerx
  class: HueSmartButtonZ2MLightController
  integration:
    name: z2m
    listen_to: mqtt
  controller: Hue button chambre parents      # should be exactly the friendlty name from Z2M (case sensitive and spaces)
  light: 
     name: Hue light chambre parents          # should be exactly the friendlty name from Z2M (case sensitive and spaces)
     mode: mqtt
  
  
  
ChambreMathys:
  module: controllerx
  class: E2002Z2MLightController
  integration:
    name: z2m
    listen_to: mqtt
  controller: Telecommande_Mathys              # should be exactly the friendlty name from Z2M (case sensitive and spaces)
  light: 
      name: light_mathys                       # should be exactly the friendlty name from Z2M (case sensitive and spaces)
      mode: mqtt