Nest Protect stops updating sensors after losing connection with Internet - requires reload #385
Comments
puterboy
changed the title
|
Are you using NabuCasa or some fancy DIY solutions for the callback? |
|
Duplicate of #63 |
|
Could be that it has always been "broken" but that the router reboot time is borderline in terms of causing loss of connectivity. Interesting, though the standard Nest Integration only started to lose connectivity on reboot at the start of November so whatever is causing loss of connectivity may be different between the two integration. Regardless though the bugs for both the standard Nest Integration and Nest Protect are totally repeatable. Given that per #63 the bug already dates back a couple of years, do you plan to fix it? |
|
@puterboy if we figure out the exact issue, I am happy to see if I have time to fix it. Or perhaps any of the other contributors like @GSzabados. I will close this issue, let's continue the discussion in #63. |
|
@puterboy, are you sure that you are on the latest beta? If yes, could you please turn on debug logging before your router reboots and check the logs for Also, which version of protect do you have? I have had a router reboot a few days ago and I have not seen any issue like that. |
|
v.0.40b9 I will turn on debug logging and check back... |
|
Turn it on in the configuration.yaml not on the Integration's page, so it would be persistent. |
|
How long does it last when turned on via the GUI? It seems to have survived a |
|
I have mixed experience, that's why I suggested the yaml way. |
|
OK. |
|
Ok, could you provide some details. Which Nest Protect do you have wired or wireless? How long did you wait after rebooting the router before restarting the integration? What kind of router do you have as it requires regular reboots? And why does it require regular reboots? How is the IP assignment from your ISP. Do you have IP4 or IP6, fixed or dynamic, have at all a public IP or sitting behind a NAT? How is your IP assignment on your internal network? How is IP assigned to HA,how does it assigned to your Nest Protects? I really think that your issue is likely a wrong a network configuration, but not an integration issue. As I cannot reproduce it with a router reboot. And mine does take some time to reboot. Otherwise, do you have any error message in your logs when you reboot your router, that relates to the integration? Or any log from nest_protect regarding fetching new token, unknown exception, etc... |
I have 3 (wireless) Nest Remote Thermostat Sensors (one each for my 3 Nest thermostats)
I waited about 4 hours. By then, I knew the temperature had changed several times based on other temperature sensors in the house.
I use dd-wrt and reboot weekly as it is a recommended best practice.
IPv4 only.
IP address for HA is static.
My network configuration is pretty simple and stable -- I am not doing anything crazy and I understand my setup very well. Have you tried this with a Nest thermostat remote sensor?
No logs after I reboot. Only get logs again after restarting the integration -- at which point everything is working again normally. Again, I think this is probably similar to the bug identified with the standard Nest integration (that I use for the thermostats themselves) Google Nest Integration loses connection with Nest Thermostats... #130407 What seems to happen at least with the Nest Integration is that after rebooting, the Google pub/sub api loses connection with the thermostat and aborts. Perhaps a similar issue occurs with Nest Protect NOTE: I need to use both the standard "Nest Integration" plus "Nest Protect" since "Nest Integration" is required for the Nest thermostats themselves (using the published Google Nest API) while "Nest Protect" is needed to capture the temperature of the Nest remote sensors (which is not included in the Google API) BTW, thank you so much for helping me to troubleshoot this!!!!! |
|
The following is quoted from the update released to fix the problem with the classic Nest Integration:
Perhaps that is helpful... |
|
First of all, this is not the same API as the Nest integration. Second, we need some log to see what is happening. How it does disconnect. If your HA loose connection, how do you know that your Nest thermostat is really publishing changes as well. For me the odd thing, that I cannot reproduce it with a router restart or configuration reload, even by dropping all states from the router's table. Which is like a full disconnect. Otherwise, I have not heard of this suggested router restart thing, but sounds quite weird and suggests that something is not working well in the internals, if you need to clear out everything. Do you have any other integration which fails like this? Edit: Happy New Year! 🥳 |
|
And thinking a bit more. Are you using NabuCasa? And can you access remotely your HA just after the router is rebooted? Because I think that after rebooting the router it is likely just blocks any incoming connection to your HA server. |
|
Most happiest of New Years to you too!
Well the point is that there are NO Nest Protect debug log entries UNTIL I reload the Nest Protect integration. It's like the integration is just sitting there waiting for messages but never receiving them. Is it possible to create a stand-alone python script that subscribes to the messages and prints them to stdout for example.
Well, I can see the Nest remote temperature sensor still updating in both the Google Nest App and Google Home.
Perhaps it depends on how long the timeout is before reconnecting?
Per the DD-WRT wiki
Only the regular Nest Integration |
Yes - I can immediately access HA after reboot via both web and ssh -- in fact, it doesn't even break existing ssh connections from within my LAN. |
I am asking remote access. And that it is not broken on lan, I guess you might have some other network equipment as well on the LAN side, either a Mesh in bridge mode or a switch. Anyhow, how do you access HA remotely? Can you access it remotely, NOT on local LAN but for example 4G? I still think that your router just closes the connection and even if a callback happens it does not let it through due to the reboot and whatever it does... |
Sure... I can access it via my phone on 4G/5G using nginx and https
The reboot takes a couple of minutes... so it is possible that the callback occurs while the reboot is still occuring. Can you share with me the logic of how connections and callbacks are supposed to be maintained and restarted? |
Really unlikely.
I guess your router closes the connection and not notifying the participants. So there isn't reconnect. I have an idea, which I will discuss with @iMicknl to make it more resilient. |
|
I was able to isolate and trigger the problem by simulating a WAN disconnect by adding and deleting the following firewall rules that block connection between my HA instance and Google Nest Cloud. No reboot or other router changes required.
Note: vlan2 = wlan Doing the above stops all updates from my Nest Remote Temperature Sensors from registering in HA. This is 100% reproducible and should confirm that the issue is caused by HA (and thus the Google Nest Protect Integration) losing connection to Google Nest Cloud (and/or vice-versa). This also should rule out any user issues with my Router or my general WAN connectivity or my LAN setup & topology, or how I reboot the router or how I access HA etc... |
|
You just proved, that it is not a HA issue, it is a router configuration issue. The reboot should not drop those forwarding rules. I would check after reboot, what is happening with your iptables. You might have set up some rules which are not applied correctly, and blocking access remotely. |
I am confused about why you are so insistent on attributing the problem to my specific choice or configuration of router. The reboot doesn't drop any forwarding rules. It just temporarily breaks Internet connectivity. The root cause is that the Nest Protect Integration fails to recover from temporary lost Internet connectivity, regardless of cause -- a router reboot is just one of many ways to lose temporary connectivity (as any router does) while rebooting. Regarding my previous post, I only dropped and re-added the iptables forwarding rules to simulate a temporary disconnection Indeed, just to be sure, I confirmed that temporarily physically unplugging the Ethernet cable to the HA device and restoring the physical connection also causes the Nest Protect Integration to fail and not recover (in contrast, other services like SSH recover when network connectivity is restored) - -- nothing to do with my router At a minimum, this uncovers a lack of robustness in the HA Nest Protect Integration. This silent failure to reconnect is insidious because unless one is regularly observing the temperature traces for Nest sensors, one may never know that the integration needs restarting -- and many users may experience temporary Internet disruptions without even being aware of it. Bottom line is that the root cause has nothing to do with my particular router choice, router configuration, network topology etc... Can we focus on how one might go about fixing this? As a temporary workaround, I have set up cron jobs that use the REST API to restart the integration after every (scheduled) reboot. It would seem that the right solution though is to make the integration itself more robust and self-aware. If it's not fixable, the documentation should at least warn the user that the integration may need to be restarted any time that HA loses connection with the Internet. |
|
Your router after reboot blocks the incoming message from the Nest API. The integration waits for incoming messages. The problem is your router. It is not your internet connection, it is not your HA connecting to your router. It is your router.
Or Elon Musk can rule the world from tomorrow...
Yes, there is a keepalive message coming from the Nest API which is blocked by your router.
FYI. It was reconnecting every 5 minutes and meanwhile it was not updating anything, breaking the whole purpose of the integration. #347 (comment)
Feel free to contribute.... |
We seem to be talking past each other :( I know my router iptables rule thoroughly.
This is all pretty basic. Presumably though the problem is:
That would explain the behavior since:
So if truly the Nest Protect Integration is just sitting around waiting for an incoming message, then a reboot or any of the other Internet disconnect use cases listed below will cause the integration to fail silently when the
Indeed, I left out losing Starlink satellite coverage which would be another possible cause :) Of course all the above use cases do happen -- more frequently than you might imagine (Google Nest even had an outage a couple of weeks ago)
My router only blocks Can you enlighten me on what IP addresses, protocols, ports and/or states you think my router is uniquely blocking versus all the other routers out there on the market? But without any specifics of what IP address/protocols/ports/states need to be allowed, it's really impossible for me to address the situation -- other than to say that my router does not block any incoming messages that are not blocked by pretty much any other consumer router on the market.
That would explain why it used to work!!! Good catch!!!
Happy to... which is why I asked previously if there is any way to abstract this code from the Nest Protect Integration so I could play with it independent of the more complicated HA environment. That being said, I am not a coder and am a rank beginner at Python. |
|
How often do you receive updates from the Nest API? And how long is a reboot takes on your router? |
|
The updates are asynchronous -- only when temperature changes by 1 degree'ish. A reboot takes about 3-5 minutes I would say. Presumably the Nest thermostat itself has to also deal with and recover from intermittent Internet disconnects -- and it always does manage to do so without requiring a HW or SW reset... in fact, you can see a delay in that the thermostat seems to take a couple (1-3 minutes?) to recover from an offline event caused by losing Internet connection. So the Nest thermostat must be getting a heartbeat and/or pinging the server periodically. (Note the Bluetooth-based Nest Remote Sensor communicates to the Internet via the WiFi-based thermostat rather than directly) |
|
Ok, what if you would try a different router? I have disconnected my HA server for more than 5 minutes from my network, updates still came in as normal. Forced updates during the disconnect, and once connected those updates came in immediately. The integration is resilient enough to survive these two cases on my side, so I can narrow it down that the issue is with your router and network setup. Don't tell me, that the Nest Thermostat has a heartbeat and reconnects, that must communicate with the Nest server, that is its purpose, to get weather forecast and adjust your heating pattern. This integration is like the Nest app on your phone. It shouldn't continuously ping the servers. As you keep rebooting your router, you can keep rebooting your HA as well or reloading the integration once you rebooted the router. It might help HA as well, if you keep it rebooting. And just to give you some background on the router. I used to have a 4G modem/router, which was disconnecting from the internet continuously, so I set up a script to check connectivity, and once it has failed for 15 seconds, then it rebooted the router. I was convinced that one of the base stations in the area must be faulty, and drops connections. It went this way, until I bought a different modem/router, and to my biggest surprise it did not drop connections anymore, and it did not require regular reboots, even it was from the same manufacturer. |
Per your advice, I swapped the router for another physical one (same popular, well-tested Netgear R6700 model) and got the same behavior so it's not the HW.
The real question (in my mind) is why in my situation, the Nest Protect integration doesn't try to re-initiate connection with the Google FCM server when Internet connectivity is resumed -- while in your case, the HA Nest Protect integration seems to detect the disconnect and re-initiate connection. Indeed, I don't see how my router firewall rules (or router behavior more generally) would allow for an initial (new) outbound connection to the Google FCM server but then block subsequent (new) connections when attempting to reconnect after a disconnect or reboot -- the router firewalls can't possibly know the difference since they just look at IP addresses, protocols, and ports when establishing new connections (and after a reboot, a new connection is needed for sure) Without understanding further how the Nest Protect integration is supposed to reconnect to the Google server after losing connectivity, it's really hard for me to troubleshoot.
The Nest Thermostat app behavior is rightly different from the Nest integration with HA. In contrast, if the Nest Protect Integration relies on 'push' communications from Google servers, then it needs to maintain constant connectivity via an
Again to be clear. My router doesn't require weekly reboots - just people I respect in the router dev community have told me that it's a good practice to reboot periodically to clear out an accumulating "rot". Indeed, that is probably why most routers have a feature to allow regular reboots. Also my router's connectivity is rock solid - I track it regularly... |
|
I was just looking through the pynest codebase and I don't see any code that checks for network connectivity and could detect a network disconnect. The code can detect server disconnections that are signaled by the server but not network disconnects themselves. To do so it would need to have a heartbeat or keep-alive or network ping or OS-level check that the TCP connection is alive etc. In practice:
Are you using a Nest Remote Sensor? Are you using HAOS? In summary
|
Please add it all, just need to handle the right exceptions and need to add more debug logging, feel free to read the documentation of asyncio and aiohttp, plus you can rewrite the whole logic of the code to take care of checking the connection state. Here is some unsupported documentation on the subject: https://www.home-assistant.io/more-info/unsupported/connectivity_check/ Otherwise meanwhile you are digging yourself into the codebase and reached an expert level in the disconnection problems, try to change in the
And to note, I am not a code myself either, but your essay writing skills are amazing me, just to see a bloody cloud connected temperature sensor... Maybe you should put this effort somewhere else, and should contribute to something instead of whining about it. |
|
Crickets... I can hear crickets... |
The problem
This is probably related to the same problem identified and verified for the standard Nest integration Google Nest Integration loses connection with Nest Thermostats... #130407
.
The problem is when the router reboots or otherwise loses connection to the Internet, the integration stop receiving pub/sub messages from the Google Cloud.
The temporary workaround is to manually reload the integration every time you reboot the router or otherwise lose Internet connection
Interestingly, the problem with the standard Nest Integration seemed to start for me about 6 weeks ago (I do automatic router reboots every Friday night) BUT the problem with the Nest Protect integration only first affected me a week ago and again last night.
What version of this integration (ha-nest-protect) has the issue?
0.4.0b9
What version of Home Assistant Core has the issue?
2024.12.5
Device / Model
Nest Protect Temperature sensors
Diagnostics information
As above.
Home Assistant log
Logs
Additional information
No response
The text was updated successfully, but these errors were encountered: