All devices not reachable after reboot / power loss

[robertlandes]

Environment

• Raspberry PI 3B+ (3.0A power supply)
• Conbee II (latest firmware 264A0700)
• deCONZ 2.05.72 inside Docker (HASS.io add-on version 5.0)
• Home Assistant 0.103.5
• USB devices connected are Conbee II and Z-Wave (Sigma) sticks, both (each) connected via 1m USB extension cable to different USB ports
• USB devices configured by their well known name in HA for both Conbee II and Z-Wave stick

Situation

I recently setup the complete environment from scratch to mainly do some lighting automation. so I started initially with deCONZ add-on version 5 and didn't have all the problems of migrating from older versions of the HA add-on mentioned here and in the HA forums.

After I finished everything, I took a complet backup (HASS.io snapshot) and left my parents home after christmas (which is a 600 km drive from my current location, so repairing everything would be a super bad situation for me).
After a power outage, the raspberry rebooted and I ended up with all devices being unavailable. Restoring backups etc. didn't help and after investigating logs etc, I am actually out of options where to look at.

Some lights seem to be online and shown, but if you toggle them, nothing is happening, although deCONZ / HA show a change of state.

When I start the add-on with VNC enabled the weird thing is the network looks like this:

and my devices look like this:

Having the network open for quite some time didn't change anything. The only thing I saw is that one motion sensor seems to connect:

[SwoopX]

Hey, I faced a comparable situation two days ago. Around noon, my whole network went dead (which is a bit larger). I checked the network settings in deconz GUI and what was presented to me somehow felt fishy. Due to whatever reason, I had a different PAN-ID, network key and zigbee channel. Luckily, I was able to sniff the traffic which allowed me to see my device communication (after having changed the channel back from 15 to 25, still had the network key included in wireshark). However, I had no success via deconz GUI to restore the initial settings, it simply didn't work. Then I recalled something I've read in one of the release notes which might be able to solve it, and it did the trick indeed.

Created a wiki page for it so it doesn't keep on dwelling in the depths: Network lost issues. Maybe you want to give it a shot anyway.

@manup you may want to pin the wiki article or a comparable issue. That feature was a great help to me and it might aid in resolving other people's network issues. Were quite some over the past couple of months...

[robertlandes]

@SwoopX Thanks for the info, but that doesn't apply in my case. As you can see in my screenshot above I only have a single network config. And as I started the complete setup with the version no. mentioned above, I think I don't suffer from any past issues from old firmware/software issues.

I read through most of the older issues here, on HA issues and forums and I think my situation is not affected by most of the problems as they are upgrade related in some way.

As I said, this is a brand new install with everything fresh from the beginning.

I also have almost the same setup running at my place (only difference is a Raspberry 4 instead of 3b+) with more than 100 devices (Ikea, Hue, Innr, Xiaomi, etc) running perfectly fine. That is why I decided to create a new issue and didn't "reuse" one of the existing ones.

[SwoopX]

And as I started the complete setup with the version no. mentioned above, I think I don't suffer from any past issues from old firmware/software issues.

Well, it must not necessarily be due to some legacy config stuff. I had running my config for several months and it happened out of a sudden. The last update I made was 3 months ago or so.

Point I wanted to make here is that configuration looked ok but it did not work. Applying configuration from the "secret" Phoscon page resolved it. Why not give it a try, it cannot go any more wrong 😐

[robertlandes]

@SwoopX I should have mentioned that before, but I already tried loading that one config I have available there without any success.

[SwoopX]

I see, too bad. Somehow, it feels familiar. Have you checked #2245 already?

[robertlandes]

@SwoopX yes, I did search and have a look at most of the related issues including this one.

I did a reboot right now and disabled autostart of the deCONZ container and then started it manually when everything was up in HA. I noticed the following on startup:

starting version 232
[21:42:04] INFO: Waiting for device...
[21:42:05] INFO: Starting VNC server...
[21:42:09] INFO: Starting the deCONZ gateway...
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib/arm-linux-gnueabihf/dri:${ORIGIN}/dri:/usr/lib/dri)
libEGL warning: DRI2: failed to open swrast (search paths /usr/lib/arm-linux-gnueabihf/dri:${ORIGIN}/dri:/usr/lib/dri)
libpng warning: iCCP: known incorrect sRGB profile
[21:42:11] INFO: Starting Nginx...
[21:42:11] INFO: Running Hass.io discovery task...
[21:42:11] INFO: Running the deCONZ OTA updater...
[21:42:11] INFO: Running the IKEA OTA updater...
[21:42:11] INFO: deCONZ is set up and running!
2020/01/02 21:42:11 [notice] 390#390: using the "epoll" event method
2020/01/02 21:42:11 [notice] 390#390: nginx/1.10.3
2020/01/02 21:42:11 [notice] 390#390: OS: Linux 4.19.88-v7
2020/01/02 21:42:11 [notice] 390#390: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/01/02 21:42:11 [notice] 390#390: start worker processes
2020/01/02 21:42:11 [notice] 390#390: start worker process 419
21:42:11:410 HTTP Server listen on address 0.0.0.0, port: 40850, root: /usr/share/deCONZ/webapp/
21:42:11:434 CTRL. 3.16.221:42:11:636 dev /dev/ttyAMA0
21:42:11:636 COM: /dev/ttyACM1 / serialno: DE2132210
21:42:11:636 COM: --dev: /dev/ttyACM1 (ConBee II)
21:42:11:636 ZCLDB init file /data/.local/share/dresden-elektronik/deCONZ/zcldb.txt
21:42:12:057 parent process bash
21:42:12:057 gw run mode: docker/hassio
21:42:12:057 GW sd-card image version file does not exist: /data/.local/share/dresden-elektronik/deCONZ/gw-version
21:42:12:058 sd-card cid: 035344535033324780ffffffff0127d1
21:42:12:098 DB sqlite version 3.16.2
21:42:12:101 DB PRAGMA page_count: 35
21:42:12:101 DB PRAGMA page_size: 4096
21:42:12:101 DB PRAGMA freelist_count: 2
21:42:12:101 DB file size 143360 bytes, free pages 2
21:42:12:102 DB PRAGMA user_version: 6
21:42:12:102 DB cleanup
21:42:12:105 DB create temporary views
21:42:12:166 don't close database yet, keep open for 900 seconds
21:42:12:167 started websocket server at port 8081
21:42:12:182 found node plugin: libde_rest_plugin.so - REST API Plugin
21:42:12:189 found node plugin: libde_signal_plugin.so - Signal Monitor Plugin
21:42:15:573 found node plugin: libstd_otau_plugin.so - STD OTAU Plugin
21:42:15:607 dev /dev/ttyAMA0
21:42:15:607 COM: /dev/ttyACM1 / serialno: DE2132210
21:42:15:607 COM: --dev: /dev/ttyACM1 (ConBee II)
PROTO: CRC error
PROTO: CRC error
21:42:15:679 dev /dev/ttyAMA0
21:42:15:679 COM: /dev/ttyACM1 / serialno: DE2132210
21:42:15:679 COM: --dev: /dev/ttyACM1 (ConBee II)
21:42:15:715 DEV config changed event
21:42:15:787 Device firmware version 0x264A0700
21:42:15:794 unlocked max nodes: 200
21:42:15:875 Device protocol version: 0x010B
21:42:15:895 new node - ext: 0x00212effff050392, nwk: 0x0000
21:42:16:006 don't close database yet, keep open for 900 seconds
21:42:16:006 LightNode 5: Wohnzimmer Tischlampe added
21:42:16:023 don't close database yet, keep open for 900 seconds
21:42:16:024 LightNode 6: Wohnzimmer Schrank added
21:42:16:047 don't close database yet, keep open for 900 seconds
21:42:16:048 LightNode 7: Schlafzimmer Decke added
21:42:16:064 SensorNode 2 set node 0xccccccfffea6f966
21:42:16:082 SensorNode 3 set node 0xccccccfffe3eeefc
21:42:16:100 SensorNode 4 set node 0xec1bbdfffe23ab3c
21:42:16:119 SensorNode 5 set node 0xccccccfffea7a8db
21:42:16:135 SensorNode 6 set node 0xccccccfffe3d9446
21:42:16:152 SensorNode 7 set node 0xccccccfffee589dc
21:42:16:169 SensorNode 8 set node 0xccccccfffee2e978
21:42:16:185 SensorNode 9 set node 0x14b457fffed4253d
21:42:16:202 SensorNode 10 set node 0xccccccfffe54452d
21:42:16:278 ZDP node descriptor request to 0x00212EFFFF050392
21:42:16:278 APS-DATA.request id: 4, addrmode: 0x02, addr: 0x0000, profile: 0x0000, cluster: 0x0002, ep: 0x00 -> 0x00 queue: 0 len: 3 tx.options 0x04
21:42:16:278 ZDP send request id: 0x03 to 0x00212effff050392
21:42:16:329 Current channel 25
21:42:16:350 CTRL ANT_CTRL 0x03
21:42:16:382 Device protocol version: 0x010B
21:42:16:440 Current channel 25
21:42:16:460 CTRL ANT_CTRL 0x03
21:42:16:490 Device protocol version: 0x010B
21:42:16:549 Current channel 25
21:42:16:569 CTRL ANT_CTRL 0x03
21:42:16:600 Device protocol version: 0x010B
21:42:16:659 Current channel 25
21:42:16:679 CTRL ANT_CTRL 0x03
21:42:16:753 APS-DATA.confirm id: 4, status: 0x00 SUCCESS
21:42:16:753 APS-DATA.confirm request id: 4 -> confirmed, timeout 38155024
21:42:16:758 APS-DATA.indication srcAddr: 0x0000, srcEp: 0x00 dstAddrMode: 2, profile: 0x0000, cluster: 0x8002, lqi: 241, rssi: 19
21:42:16:758 APS-DATA.indication request id: 4 -> finished
21:42:16:758 APS-DATA.request id: 4 erase from queue
21:42:16:758 ZDP status = 0x00 -> SUCCESS
21:42:16:758 ZDP Node_Descriptor_rsp 0x00212EFFFF050392 - 0x0000
21:42:16:837 Mgmt_Lqi_req zdpSeq: 1 to 0xEC1BBDFFFE33B91E start index 0
21:42:16:837 APS-DATA.request id: 10, addrmode: 0x03, addr: 0xec1bbdfffe33b91e, profile: 0x0000, cluster: 0x0031, ep: 0x00 -> 0x00 queue: 0 len: 2 tx.options 0x00
21:42:17:367 dev /dev/ttyAMA0
21:42:17:371 GW update firmware found: /usr/share/deCONZ/firmware/deCONZ_ConBeeII

I was wondering about the following lines in my log:
PROTO: CRC error
after initializing the Conbee II and searching for similar log lines I found #1996 which seems to be the exact problem I am experiencing.

[easybeat]

Hi
since tonight I'm facing exactly the same situation. All my devices do not react. Absolut nightmare!

My whole house is not to use anymore. And I thought I'm on the right site with deconz!

Tried to restore the netword thing but didn't help.

Any inputs what I could do?

Thanks for any help!
Beat

[easybeat]

Hi
Ok after last nights total crash, I did restore my latest backup this morning. For sure still nothing working. Then I did reset every single light (have 32 of different brands) with Touchlink and run the add light process. The lights came back one after another. But unfortunately all the scenes were deleted.

If I'm not the only one having this issue then I think this is a massive problem of deconz!

I have no idea what caused that problem as the system was running with this firmware since July and my last change is 10 days ago. It started on Monday night when some lights started randomly blinking. I just managed to turn all lights off and then about 5 lights didn't show connection. So I tried to reconnect but failed and on Tuesday night after some more trying to fix it at once all my light were disconnected and gone.

@robertlandes : How did you fix your problem?

Kind regards
Beat

[mountainsandcode]

I have exactly the same problem, it's rather annoying to have to completely reset the network every time I happen to reboot my docker host device (in my case a Synology). @manup: Any ideas? Anything we can do to facilitate the troubleshooting?

[easybeat]

I just had the same problem again. Some lights didn't react, some started randomly flashing. Had to disconnect each light from power and restart Deconz.

Now everything is working again.

@manup: Any ideas what could cause this. It is getting critical for me now with 2 events like that within 2 weeks!

Kind regards
Beat

[nodefeet]

@robertlandes I am having the same issue.

After reboot there is no connection possible. And its easy to reproduce:
Reset the gateway -> add lights in phoscon, create a group -> reboot --> no connection to lights or groups.

I am developing with deconz for quite a while now and it always worked fine after reboot.
Right now I am only adding 2 lights from my desk but nothing works after reboot.

The only thing I changed compared to the last months is that I was using a Conbee 2 now. But it does not work with my old Conbee 1 either now.

deconz Version: 2.05.64
Conbee 1 Firmware: 26330500

Edit: I upgraded to the latest version 2.05.72 same issue
I am running deconz headless on a RPi. the deconz service runs fine

[jcaron23]

@nodefeet Exactly the same issue here. On reboot, all connections are gone. The settings are correct (channel, network ID, security key...), but the devices are no longer connected. It looks like the coordinator is not correctly saving details of its neighbours or something similar.

The only exception from the devices I have at hand are a Philips Hue Motion Sensor, which will manage to re-join each time.

[nodefeet]

I can even recreate it on my deCONZ GUI App on my windows machine.

Step 0: Reset the Gateway
Step 1: Add lights
Step 2: reboot

Result: no connection even after several hours

To be clear I am using these lights for almost a year without any connectivity issues.

[jcaron23]

@nodefeet In my case, even after many days the connections don't come back up (except that Philips Hue Motion Sensor). Definitely something missing in what is saved and restored after a reboot.

[nodefeet]

@jcaron23 same for me. The connection is lost permanently.

At least in my case it has to do something with Conbee's non-volatile memory.
I was able to get another Conbee 2 and it works now after reboot or shutdown, although I have no idea how I supposedly should have damaged the EEPROM of my old Conbee 1 and the new Conbee 2.

But here is what I don't understand:
If network settings are lost in the Conbee after reboot. Shouldn't it be possible for me to at least reload the configuration from the hidden network site (ALT+Click on Advanced in Gateway page) for one of my "damaged" Conbees?

After reboot the networks settings (Channel, PANID, Network Key) are the same as before. So as expected loading these settings does not reenable the connection. Backups do not help as well.

[SwoopX]

Applying the configuration took a while in my case. I havent touched anything for like 5 minutes. Maybe you were too impatient?

[nodefeet]

@SwoopX unfortunately there is no connection possible even after hours.

I was about to write how I discovered that this issue happens when you start the raspberry with deconz without an Ethernet network connection. And indeed I was able to "destroy" multiple Conbees this way. I could even reproduce it on a fresh raspbian install with deconz. But thanks to the support of dresden-elektronik the following seems to work for me!

Solution

All I did were the first three steps from the technical support page from deconz:
[Point 6 under Troubleshooting for ConBee II with Linux ](https://phoscon.de/en/support#conbee2

1. Close deCONZ if it’s running
   -> I did: sudo systemctl stop deconz
2. Unplug the ConBee II and wait 10 seconds
3. Connect the ConBee II again and wait 10 seconds
   -> I did: sudo systemctl start deconz afterwards

I am currently not able to reproduce the connection error after reboot and my "destroyed" Conbees seem to work again. Everything works fine 😀.

[SwoopX]

Great. Hm, Im my case, it is a Raspbee. Might be that I unplugged my Pi Zero running it for a while as part of the (at least my) solution... Anyhow, those steps should be remembered!

[cooperaj]

I'm seeing this issue and the above fix by @nodefeet did not work.

I have rebooted everything multiple times in different orders and at best I can get a single connection to a switch. :(

Next step is reset it all and start from scratch. I had such high hopes after the initial setup being so seamless. Hopefully this can be figured out soon.

[nebbiadigiorno]

same problem here. At every restart all my network goes down.

[nodefeet]

Just to make sure:

Did you try repowering the lights?

Sometimes my lights need to be restarted as well after deconz restarts before there is any communication possible again.

[RosaEinhorn]

I am having the exact same issue after a restart of the host system and had to reconfigure all devices today.

[jcaron23]

Is there a way to download, install and use older versions of the firmware? From what I understand this is a recent issue, so finding in which firmware version the problem was introduced should help to track it down.

[manup]

Hi everybody, I hope with the next release 2.05.73 the reboot issue gets addressed for some cases. But there seems to be some issue deeper.

Can you please try in deCONZ GUI:

• Leave
• Join

To restart the Zigbee network, to see if that brings the network back.

[cooperaj]

Can you please try in deCONZ GUI:

* Leave

* Join

To clarify, should I have a broken network to start with?

[manup]

The steps are meant for the case when deCONZ shows "In Network" state but for some reason "all" devices are not reachable.

[mycanaletto]

For me I just loose the 3 last device registered. Others devices (11) are present ! Any idea ?

In fact it seems that it happens on Xiaomi sensors that were previously paired with the GW Xiaomi, then removed from it.
I have just paired an Aqara sensor (opening) and I find it after restarting Deconz.

The other possibility is that in the meantime I have updated to version 2.05.74

[andreasscherbaum]

I'm still thinking about it, but also need to explore alternatives.
All in all it's a nice device, and if they can fix this grave bug it works fine.

[TheWizz]

This lengthy discussion is a bit concerning. Apparently, there are numerous paying customers experiencing loss of basic functionality after a restart/power loss. There is ZERO official response. Neither here, nor to any support emails as far as what's reported above. Perhaps DEs line of business is really something else, and this is just a little side project for them, where they really couldn't care less?

-JM

[randomlyalex]

If i'd known about this before I purchased I would definitely chosen a competing product. So whilst it doesnt solve our problem, i've left and edited my reviews accordingly so others buying now will know. I would recommend everyone else leaves reviews on amazon and where ever describing their expereinces and lack of offical response and support.

[easybeat]

I contacted them several times and never got a response!

[vekexasia]

Still no news on this? wtf

[nodefeet]

I also would appreciate some input from dresden elektronik about this issue.

@manup: I am not sure if you are aware of the significance of this.
This issue renders complete home installations useless with hours of work wasted to release and reconnect everything.

I do not understand why this issue does not have a higher priority to you.

[gregorschulz]

Holy cow. My whole home automation is built on sand?!
I‘m using a RaspBee @ RaspPi 3B+ and had just one problem with the virtual light sensor of the Phoscon Gateway showing only the „Moon“ all the time.
No answer from DE yet, but I just get discouraged to receive one.

Moreover I found this issue where I will loose my complete setup due to a power cycle?!

[Smanar]

@gregorschulz in the JSON of your virtual sensor, you have "true" at the "config/configured" field ?

[gregorschulz]

@Smanar
This virtual daylight sensor is a built-in sensor in the RaspBee Phoscon-Gateway. It shows the „Moon“-symbol in the own GUI.
Which json do you mean?

[Mimiix]

@nodefeet @gregorschulz

I'll be closing this issue for now. Please open own issues on your personal situation. As this issue is open since Jan, new issues cannot be related and should be given their own topics.

[cooperaj]

If this issue is not fixed then why close the ticket (very recent side tracking not-withstanding)?

[Mimiix]

@cooperaj In my personal experience in reading over 400 issues the past week, i've noticed that there is allot of reasons why devices are gone. The personal experience in moderating learned that having threads like these never lead to a proper solution.

From the user perspective, a issue that is fully related to their environment helps solving this issue faster than having 1 thread with 20 people with 15 different configurations.

From the developers perspective, having all these cases seperated helps to see paterns. This thread have become a mess in which you can't figure out what happend where with what parameters.

I want your issue solved, hence i want you to open a own issue.

[TheWizz]

I did file a separate issue for this quite a while back as #2015 However, that one was merely closed as "stale" with no answer or resolution, so I'm not sure filing individual issues works either. But if you prefer that method, please have my individual issue re-opened and look into it now.

-JM

[Mimiix]

I did file a separate issue for this quite a while back as #2015 However, that one was merely closed as "stale" with no answer or resolution, so I'm not sure filing individual issues works either. But if you prefer that method, please have my individual issue re-opened and look into it now.

-JM

@TheWizz
I would like to ask you to be careful on how to approach things here. Demanding stuff like this is not done in any situation. I am in no way employed by DE and just doing this on voluntary basis.

[andreasscherbaum]

Ok, I also don't see how it is helping when the same problem is discussed in several different issues, but I opened #2915 for my case.

Given the lack of feedback from the vendor I'm seriously considering throwing work away and just return the device.

[Mimiix]

@andreasscherbaum I think i made myself clear on this topic by staying positive and polite, but apparently there still people that do not understand that negativity only makes things worse. You are asking for change but yet you are not contributing to do make the change.

If people keep doing this, i will be locking this issue.

[andreasscherbaum]

@Mimiix How is opening the other issue as requested not helping?

[Mimiix]

It is, but the negativity isn't helping ;)

[andreasscherbaum]

@Mimiix Others complained about the lack of feedback from the vendor as well, and you made it clear that you are not working for them. I dunno, how else can someone express frustration about this lack of feedback? I spent time writing a couple Ansible Playbooks for this, I submitted a PR or two, and sent in a bug report. That's fine, and I got feedback there. But when it comes to serious issues like this, am I forbidden to say anything about it?

[Mimiix]

@andreasscherbaum Allot of things have been said. And trust me, i'm forwarding all of this to @manup and talk about it with him. Stuff needs to be better and stuff needs to be done. Hence i am still talking to you, because i know how you feel. I'm also the guy in trying to fix this thing by doing my part and trying to make it a better place. Last week i've closed over 250 of the 480 issues. That aint fun either, but i'm doing my part.

Do i think DE needs to step up? Yes
Do i think DE Dropped the ball? Yes
Do i think stuff needs to be fixed? Yes
Does complaining about this all day long over and over again solve issues? Absolutely not.

This stuff needs time and i'm on it. Trust me on this, yesterday also included allot of stuff for the community as that is my only concern at this point. I am not a developer, i just like to see this place grow.

[TungstenE2]

Hi all, just called DE (Tel.: +49351318500) and talked to the support. The issue is know and a fix/workaround is in place.

Mail from DE:
Mit der deCONZ 2.05.77 und der Firmware 26580700 sollten die Bugs behoben sein. Evtl. muss einmal unten genannter Workaround durchgeführt werden. Wenn die Geräte nicht von selbst nach einer Wartezeit wieder kommen müssen sie evtl. einmal neu angelernt werden. Danach aber nicht wieder und ein Neustart sollte kein Problem darstellen.

Update Firmware:
wget http://deconz.dresden-elektronik.de/deconz-firmware/deCONZ_ConBeeII_0x26580700.bin.GCF
sudo GCFFlasher_internal -d /dev/ttyACM0 -t 60 -f deCONZ_ConBeeII_0x26580700.bin.GCF

Update deCONZ:
wget -N http://deconz.dresden-elektronik.de/raspbian/deconz-latest.deb
sudo dpkg -i deconz-latest.deb

Details auch hier:
https://github.com/dresden-elektronik/deconz-rest-plugin/wiki/Update-deCONZ-manually

Workaround:

1. im deCONZ „Leave“-Network
2. im deCONZ Netzwork Settings öffnen
3. mehrmals den „Save“-Button betätigen (Parameter werden neu gesetzt)
4. Network-Setting mit „Done“ – verlassen
5. „Join“- Network

[SwoopX]

The magic around the firmware update might be, and I wouldn't have known about it either, to leave and rejoin here.

[TheWizz]

Would it be possible for someone to translate the above into proper English? I know I can do a Google Translate, but since this is mainly an engish-speaking place, it would be prudent to have any workaround described clearly so everyone can benefit.

-JM

[TheWizz]

Can it be assumed that a new version of Deconz will be made available that contains the updated firmware, and can apply this without going through the GCFFlasher_internal dance? Any ETA on such a proper solution?

-JM

[Mimiix]

@TheWizz Please update as mentioned in your issue. After that, let's see what happens. You can't start to apply this without your updates first. It's essentially the same as is recommended to you: update.

The only difference is the leave and join network part. I will translate that later.

In regard to your request to use without GCFFlasher, open a feature request in the Phoscon git. Afaik you can update from phoscon, but need to manually download still: https://dresden-elektronik.github.io/deconz-rest-doc/configuration/#import

@TungstenE2 please open your own issue in your regards if you have issues after that. I will make sure what you posted gets translated.

[Mimiix]

From now, i'll lock this issue as people keep going on. The addition from @TungstenE2 is great, i'll make sure to make a page in Wiki later today or tommorow.

[Mimiix]

To make things go faster, i've added the page:
https://github.com/dresden-elektronik/deconz-rest-plugin/wiki/Possible-fix-for-devices-not-connecting

Thanks again for your effort @TungstenE2 !