Connectivity to Hass suddenly stopped - no known changes occured #362

c2mfj · 2024-05-20T14:55:56Z

Problem/Motivation

Today I noticed I could not reach my Home assistant while external to my local and tailscale network. This configuration has been working for 4 or 5 months without issue.

Expected behavior

Successfully access HASS through iOS app or browser from the internet

Actual behavior

Timeout, debug logs in add on do not show the attempted connection

lmagyar · 2024-05-20T17:50:52Z

So the add-on is running. Can you see it on the admin page as connected? https://login.tailscale.com/admin/machines Did you disable key expiry? 4-5 months ago?

You can retrieve the add-on logs even after multiple restarts with sg. like ha host log --identifier addon_a0d7b954_tailscale --lines 500 Aren't there any errors or warnings?

Without logs, without the add-on's config, it is hard to say anything more.

esenliyim · 2024-05-20T18:26:12Z

Same here. Updated today and the addon's stopped working. Previously I'd had it running as a funnel for a while now, with no issues.

There's only the following entry in the logs that appears to be anything questionable:

2024/05/20 21:10:53 rename of "/etc/resolv.conf" to "/etc/resolv.pre-tailscale-backup.conf" failed (rename /etc/resolv.conf /etc/resolv.pre-tailscale-backup.conf: device or resource busy), falling back to copy+delete

The rest just look like success messages.

I can also see the node appears as "connected" on Tailscale admin panel. In other words everything appears to be in order, but no connection.

Doubt it'll help, but here's the add-on config anyway:

funnel: true
log_level: debug
proxy: true
userspace_networking: false
proxy_and_funnel_port: 443

josefsedy · 2024-05-20T18:45:19Z

I have the same problem after updating to version 0.19.0

c2mfj · 2024-05-20T18:48:51Z

addon_a0d7b954_tailscale --lines 500
Yes addon is running. My key is set to never expire and no errors or warnings in the logs. Interestingly, unlike the others that have replied, my issue started before I updated to version 0.19.0. I updated to the latest but my issue persists.

My config:
funnel: true log_level: debug proxy: true userspace_networking: false proxy_and_funnel_port: 443

The nodes is online and connected in the tailscale admin panel

lmagyar · 2024-05-20T19:34:47Z

Quite simple configs. Hmmm.

my issue started before I updated to version 0.19.0

So there is a chance, that it is not related to this version. Strange coincidence.

userspace_networking: false

This is not needed to access HA or anything on the device from the tailnet and/or from the internet (funnel), this is needed only if HA or anything on the LAN wants to access the tailnet. If anything on the LAN (not HA on the device, but other devices on the LAN) wants to access the tailnet, with TS v1.66.x you need stateful_filtering: false (https://tailscale.com/security-bulletins#ts-2024-005 and lmagyar#127), but I think something else is going on. So 2 tips for testing:

try userspace_networking: true
try stateful_filtering: false (only if anything from the LAN wants to access the tailnet)

lmagyar · 2024-05-20T19:41:19Z

And I have rename of "/etc/resolv.conf" ... in the logs also, and everything works. So that shouldn't be a problem.

c2mfj · 2024-05-20T19:45:17Z

Quite simple configs. Hmmm.

my issue started before I updated to version 0.19.0

So there is a chance, that it is not related to this version. Strange coincidence.

userspace_networking: false

This is not needed to access HA or anything on the device from the tailnet and/or from the internet (funnel), this is needed only if HA or anything on the LAN wants to access the tailnet. If anything on the LAN (not HA on the device, but other devices on the LAN) wants to access the tailnet, with TS v1.66.x you need stateful_filtering: false (https://tailscale.com/security-bulletins#ts-2024-005 and lmagyar#127), but I think something else is going on. So 2 tips for testing:

try userspace_networking: true

try stateful_filtering: false (only if anything from the LAN wants to access the tailnet)

funnel: true proxy: true userspace_networking: true log_level: debug stateful_filtering: true

No improvement, still nothing in the logs showing the attempted connection as well.

lmagyar · 2024-05-20T19:57:44Z

Could you test stateful_filtering: false temporarily? There is a non-zero chance it can be a solution (tailscale/tailscale#12108).

esenliyim · 2024-05-20T20:15:23Z

I have tried all four combinations of stateful_filtering and userspace_networking. No changes. Still can't see anything in any logs.

c2mfj · 2024-05-20T20:44:20Z

Could you test stateful_filtering: false temporarily? There is a non-zero chance it can be a solution (tailscale/tailscale#12108).

no change with stateful filtering = false

lmagyar · 2024-05-20T21:29:22Z

Does this return an IP? dig <machine-name>.<tailnet-name>.ts.net +short This would be TS's TCP proxy for the funnel.

Can you access the https://<machine-name>.<tailnet-name>.ts.net from a tailnet connected browser? To test TS's reverse proxy inside the add-on is working correctly?

Can you acces the http://<tailnet IP of HA>:<port where HA is accessible with plain http> from a tailnet connected browser?

I'm running out of ideas. In the past, when nothing worked that should, I've deleted all the site related cookies and everything from the browser, and usually that helped.

c2mfj · 2024-05-21T01:30:43Z

Does this return an IP? dig <machine-name>.<tailnet-name>.ts.net +short This would be TS's TCP proxy for the funnel.

Can you access the https://<machine-name>.<tailnet-name>.ts.net from a tailnet connected browser? To test TS's reverse proxy inside the add-on is working correctly?

Can you acces the http://<tailnet IP of HA>:<port where HA is accessible with plain http> from a tailnet connected browser?

I'm running out of ideas. In the past, when nothing worked that should, I've deleted all the site related cookies and everything from the browser, and usually that helped.

I sat down to respond to these and it was working again. Feels like something on the tailscale side changed and reverted. Anyone else's begin working without a real solution?

esenliyim · 2024-05-21T04:29:59Z

Yeah it's back on its own.

I've done some googling, and yeah this might have something to do with the latest Tailscale version. I found a few posts on other websites where others' setups, completely unrelated to HA and this addon, suddenly stopped working for no apparent reason as well in the last 24 hours.

c2mfj · 2024-05-21T12:36:31Z

Closing it out since it appears unrelated to this addon. Thanks everyone!

c2mfj · 2024-05-21T17:14:21Z

Welp, now its down again. Still feels like a tailscale thing but just wondering if we're all still in the same boat.

itmokel · 2024-05-22T05:18:11Z

Got the same issue after the update this night. Can ping my own tailscale IP but notihing else. The connections between all other Tailscale devices are working as intended. Even incoming connections to HomeAssistant from other Tailscale devices. Without running Uptime Kuma on HA i wouldn't be able to see the issue because the connection to the App and Webinterface are fine.

I will let it run for a while and report if it changes itself without any configuration change. But i can't imagine that this is not related to the update at all.

Edit: I suggest to leave the ticket open for a while, if it's related to the update. Not everybody is updating on release day ;) I disabled them aswell for now.

robert10055 · 2024-05-22T05:42:54Z

I use it on three machines with HA, not one of them is functional since the last update

ovizii · 2024-05-22T08:44:33Z

I have a possibly related issue with the same update: #365 since this update, I can no longer create backups via the samba backup HA add-on which have been perfectly working for years before this TS (add-on) update.

gentertain · 2024-05-22T08:52:26Z

From what i have gathered as a layperson in my setup the issue lies with the routing in the containers: The tailscale addon container is entered as default gateway in other containers.

josefsedy · 2024-05-22T09:26:33Z

My problem is that since the update to version 0.19.0, the container with the add-on keeps shutting down and I don't even have the option to open its web interface. I haven't found a solution yet. Everything was working until the update. I don't know if my description is understandable, I apologize for that.

lmagyar · 2024-05-22T09:40:58Z

ha host log --identifier addon_a0d7b954_tailscale --lines 500

@josefsedy You can see the past logs with ha host log --identifier addon_a0d7b954_tailscale --lines 1000 Even if the add-on is stopped.

lmagyar · 2024-05-22T09:46:20Z

I feel 3 different issues:

TS v1.66.x related, suddenly stops working, suddenly starts.
Outgoing traffic related, when stateful_filtering: false can help. Even TS reverted this in v1.66.4 where this will be default off (as in the next add-on version also). Stateful filtering is now off by default #368
A plain bug in the add-on logic that causes the add-on not to start. This is already fixed (will be fixed) in the next release. Skip default networks without a gateway to enable UDP GRO for forwarding #366

itmokel · 2024-05-22T10:00:18Z

This fixed it for me:

Enable stateful_filtering, save, restart Tailscale
Disable stateful_filtering, save, restart Tailscale
Restart the whole system

Had the same when setting it up with Userspace networking mode. It shows disabled, but you have to enable it, save, and disable it to get it disabled. Don't know if this is related to Homeassistant or the Tailscale Addon but it feels just wrong.

epposan · 2024-05-22T10:05:37Z

@ovizii On one of my machines it seems related to name resolution,

If I change the name of the backup destination into its current ip address, the addon starts up fine.

On that specific machine I can ping the backup server using its ip address, but not using its network name
On two other machines that have been installed identically, I can also ping the backup server by its network name

It's just a temporary workaround, but worth a try.

ovizii · 2024-05-22T10:08:01Z

@ovizii On one of my machines it seems related to name resolution,

If I change the name of the backup destination into its current ip address, the addon starts up fine.

On that specific machine I can ping the backup server using its ip address, but not using its network name On two other machines that have been installed identically, I can also ping the backup server by its network name

It's just a temporary workaround, but worth a try.

Thanks but my backup target has always been an IP address which was advertised as a subnet by another tailscale node. And this upgrade broke that. :-(

lmagyar · 2024-05-22T11:23:07Z

Enable stateful_filtering, save, restart Tailscale

Disable stateful_filtering, save, restart Tailscale
It shows disabled, but you have to enable it, save, and disable it to get it disabled. Don't know if this is related to Homeassistant or the Tailscale Addon but it feels just wrong.

Yes, this is a HA UI bug (I don't know whether is there an issue for it), the UI can't distinguish between unset and disabled options, both displayed as disabled, I always use the YAML editor. As I remember thsi works in the UI editor: enable, save, disable, save, [check the yaml], restart :(

lmagyar · 2024-05-22T11:26:23Z

If I change the name of the backup destination into its current ip address, the addon starts up fine.

Seems to be related: tailscale/tailscale#12108

itmokel · 2024-05-22T11:31:32Z

Enable stateful_filtering, save, restart Tailscale

Disable stateful_filtering, save, restart Tailscale
It shows disabled, but you have to enable it, save, and disable it to get it disabled. Don't know if this is related to Homeassistant or the Tailscale Addon but it feels just wrong.

Yes, this is a HA UI bug (I don't know whether is there an issue for it), the UI can't distinguish between unset and disabled options, both displayed as disabled, I always use the YAML editor. As I remember thsi works in the UI editor: enable, save, disable, save, [check the yaml], restart :(

Wouldn't i be possible to add everything to the .yaml and set the default values to off? Otherwise every new feature must be actively enabled to be disabled. I very much prefer an auto update with disabled new features, instead of checking the release notes, adding new features to be disabled in the .yaml and update manually.

epposan · 2024-05-22T11:47:03Z

The UI not being able to distinguish between unset and disabled is indeed very confusing (one really has to read the textual explanation to see what the default is) and what makes it even more confusing is, that when you do edit the yaml file, you don't get to see the default settings, only the ones that have been modified.

I imagine this is related to the way Tailscale itself handles the option; if you don't specify an option, then use the default.

It would take quite a bit of effort for the official maintainer (@frenck ? ) of the addon for Homeassistant to scan every new release of Tailscale for new options, its defaults and then add them to the default yaml file.

For the name resolution issue I have to dig in deeper. My three identical machines apparently are not as identical as I thought.

frenck · 2024-05-22T11:48:09Z

The UI not being able to distinguish between unset and disabled is indeed very confusing

This is feedback for the Home Assistant project, not this add-on.

epposan · 2024-05-22T11:52:22Z

It was not meant as criticism :-) only to clarify my understanding of the extra work it would bring.

frenck · 2024-05-22T11:55:45Z

Oh it is fine, but I do think such feedback is helpful in a way (at the right place) 😄

As for extra work... that is not what this is about. I don't think it is right to force a default from this end, instead, I think following upstream is the correct path.

Lastly, it hides configuration one like doesn't need from the UI (as it is optional), which IMHO is a big plus.

ovizii · 2024-05-22T15:52:35Z

Btw. the v0.19.1 update fixed everything. I updated the add-on, restarted it and voilà, my backups are working again. No configuration changes.

robert10055 · 2024-05-22T16:58:18Z

I don't have the same experience
after installing 0.19.1 it behaves the same

2024/05/22 18:46:37 got LocalBackend in 14ms
[18:46:37] INFO: Service NGINX exited with code 256 (by signal 15)
[18:46:37] INFO: Service tailscaled exited with code 256 (by signal 15)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service init-nginx: stopping
s6-rc: info: service web: stopping
s6-rc: info: service tailscaled successfully stopped
s6-rc: info: service init-nginx successfully stopped
[18:46:37] INFO: Service Tailscale web exited with code 256 (by signal 15)
s6-rc: info: service web successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service base-addon-log-level: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service base-addon-log-level successfully stopped
s6-rc: info: service base-addon-banner: stopping
s6-rc: info: service base-addon-banner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

lmagyar · 2024-05-22T17:05:52Z

We need the beginning of the log, there must be another error, this is only a consequence of that error.

lmagyar · 2024-05-22T17:09:52Z

You can access the past logs with ha host log --identifier addon_a0d7b954_tailscale --lines 1000, there have to be a real ERROR message somewhere.

rlad78 · 2024-05-22T17:22:21Z

@lmagyar having the same issue as everyone else when upgrading to >=0.19.0. This contains the errors I see at the top of my log. Not sure if this is useful to you.

s6-rc: info: service base-addon-banner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service base-addon-log-level: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service base-addon-log-level successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-nginx: starting
s6-rc: info: service web: starting
s6-rc: info: service tailscaled: starting
s6-rc: info: service web successfully started
s6-rc: info: service tailscaled successfully started
s6-rc: info: service post-tailscaled: starting
[13:12:29] INFO: Starting Tailscale...
[13:12:29] INFO: Starting Tailscale web...
[13:12:29] INFO: Allow UDP GRO for forwarding on enp0s25
[13:12:29] NOTICE: Tailscale logs will be suppressed after 200 lines, set add-on's configuration option 'log_level' to 'debug' to see further logs
netlink error: bit name not found (offset 56)
netlink error: Not supported
s6-rc: warning: unable to start service post-tailscaled: command exited 92
s6-rc: info: service init-nginx successfully started
2024/05/22 13:12:29 You have disabled logging. Tailscale will not be able to provide support.
2024/05/22 13:12:29 logtail started
2024/05/22 13:12:29 Program starting: v1.66.4-t067defc64-g49481b7d0, Go 1.22.3: []string{"/opt/tailscaled", "--state=/data/tailscaled.state", "--statedir=/data/state", "--no-logs-no-support"}
2024/05/22 13:12:29 LogID: c0f9341f5954375b5ae34008b022ee2db41cbb5a82764f4ab39b0d74ae274431
2024/05/22 13:12:29 logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
s6-rc: info: service nginx: starting
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
2024/05/22 13:12:29 dns: [resolved-ping=yes rc=unknown ret=direct]
2024/05/22 13:12:29 dns: using "direct" mode
s6-rc: info: service nginx successfully started
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
/run/s6/basedir/scripts/rc.init: fatal: stopping the container.
s6-rc: info: service nginx: stopping
s6-rc: info: service tailscaled: stopping

The messages that seem concerning to me are:

netlink error: bit name not found (offset 56)
netlink error: Not supported
s6-rc: warning: unable to start service post-tailscaled: command exited 92

...and

/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
/run/s6/basedir/scripts/rc.init: fatal: stopping the container.

Edit: My config, in case it's useful:

userspace_networking: false
taildrop: false
advertise_exit_node: false
advertise_routes: []
accept_routes: false

josefsedy · 2024-05-22T17:31:54Z

Unfortunately, even today's update didn't help. This is an extract from the Supervisor log. The container keeps stopping and cannot be started.

File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-05-22 19:22:50.939 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-05-22 19:22:50.940 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/init.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 207, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-05-22 19:22:55.782 INFO (SyncWorker_4) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:23:00.228 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:23:04.217 ERROR (MainThread) [supervisor.api.ingress] Ingress error: Cannot connect to host 172.30.32.1:62128 ssl:default [Connect call failed ('172.30.32.1', 62128)]
2024-05-22 19:23:08.454 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:23:08.494 INFO (SyncWorker_3) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:23:12.375 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:23:19.046 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-05-22 19:23:19.046 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/init.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 207, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-05-22 19:23:19.326 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:23:19.345 INFO (SyncWorker_4) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:23:19.437 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-05-22 19:23:19.438 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/init.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 207, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available
2024-05-22 19:23:23.364 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:23:30.281 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:23:30.286 INFO (SyncWorker_3) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:23:34.350 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:23:41.041 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:23:41.052 INFO (SyncWorker_1) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:23:44.622 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:23:50.480 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:23:50.486 INFO (SyncWorker_0) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:23:54.580 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:24:01.135 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:24:01.139 INFO (SyncWorker_4) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:24:04.849 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:24:11.291 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:24:11.297 INFO (SyncWorker_5) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:24:14.886 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:24:20.664 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:24:20.668 INFO (SyncWorker_1) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:24:24.385 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:24:30.918 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:24:30.923 INFO (SyncWorker_3) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:24:34.392 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:24:40.214 WARNING (MainThread) [supervisor.addons.addon] Watchdog found addon Tailscale is failed, restarting...
2024-05-22 19:24:40.218 INFO (SyncWorker_2) [supervisor.docker.manager] Cleaning addon_a0d7b954_tailscale application
2024-05-22 19:24:43.691 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tailscale/amd64 with version 0.19.1
2024-05-22 19:24:48.357 ERROR (MainThread) [asyncio] Task exception was never retrieved
future: <Task finished name='Task-4686' coro=<Addon.watchdog_container() done, defined at /usr/src/supervisor/supervisor/addons/addon.py:1429> exception=AddonsJobError('Rate limit exceeded, more than 10 calls in 0:30:00')>
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/addons/addon.py", line 1443, in watchdog_container
await self._restart_after_problem(event.state)
File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 290, in wrapper
raise on_condition(
supervisor.exceptions.AddonsJobError: Rate limit exceeded, more than 10 calls in 0:30:00
2024-05-22 19:29:07.812 ERROR (MainThread) [supervisor.host.logs] No systemd-journal-gatewayd Unix socket available
2024-05-22 19:29:07.812 ERROR (MainThread) [supervisor.api] Failed to get supervisor logs using advanced_logs API
Traceback (most recent call last):
File "/usr/src/supervisor/supervisor/api/init.py", line 404, in get_supervisor_logs
return await self._api_host.advanced_logs_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/api/host.py", line 207, in advanced_logs_handler
async with self.sys_host.logs.journald_logs(
File "/usr/local/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/supervisor/supervisor/host/logs.py", line 143, in journald_logs
raise HostNotSupportedError(
supervisor.exceptions.HostNotSupportedError: No systemd-journal-gatewayd Unix socket available

lmagyar · 2024-05-22T17:41:41Z

@rlad78 @josefsedy What are the outputs of the below 2 commands?

ip -4 route show 0/0
ip -6 route show ::/0

lmagyar · 2024-05-22T17:47:12Z

@josefsedy Could you please check for TS add-on's errors with this: ha host log --identifier addon_a0d7b954_tailscale --lines 1000 Dont copy the whole, just some error related lines.

rlad78 · 2024-05-22T17:52:26Z

@rlad78 @josefsedy What are the outputs of the below 2 commands?
ip -4 route show 0/0
ip -6 route show ::/0

richard@snoothome:~$ ip -4 route show 0/0
default via 10.0.2.1 dev enp0s25 proto dhcp metric 100 
richard@snoothome:~$ ip -6 route show ::/0
richard@snoothome:~$

josefsedy · 2024-05-22T18:01:52Z

I don't know where and how to look for it.

josefsedy · 2024-05-22T18:13:06Z

It could be this:

2024/05/22 19:24:45 | stderr | s6-rc: info: service s6rc-oneshot-runner successfully stopped
2024/05/22 19:24:45 | stderr | s6-rc: info: service s6rc-oneshot-runner: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service base-addon-banner successfully stopped
2024/05/22 19:24:45 | stderr | s6-rc: info: service base-addon-banner: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service base-addon-log-level successfully stopped
2024/05/22 19:24:45 | stderr | s6-rc: info: service fix-attrs successfully stopped
2024/05/22 19:24:45 | stderr | s6-rc: info: service base-addon-log-level: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service fix-attrs: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service legacy-cont-init successfully stopped
2024/05/22 19:24:45 | stderr | s6-rc: info: service legacy-cont-init: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service web successfully stopped
2024/05/22 19:24:45 | stderr | [19:24:45] INFO: �[32mService Tailscale web exited with code 256 (by signal 15)�[0m
2024/05/22 19:24:45 | stderr | s6-rc: info: service tailscaled successfully stopped
2024/05/22 19:24:45 | stderr | [19:24:45] INFO: �[32mService tailscaled exited with code 256 (by signal 15)�[0m
2024/05/22 19:24:45 | stderr | s6-rc: info: service init-nginx successfully stopped
2024/05/22 19:24:45 | stderr | s6-rc: info: service web: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service init-nginx: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service nginx successfully stopped
2024/05/22 19:24:45 | stderr | [19:24:45] INFO: �[32mService NGINX exited with code 256 (by signal 15)�[0m
2024/05/22 19:24:45 | stderr | s6-rc: info: service tailscaled: stopping
2024/05/22 19:24:45 | stderr | s6-rc: info: service nginx: stopping
2024/05/22 19:24:45 | stderr | /run/s6/basedir/scripts/rc.init: fatal: stopping the container.
2024/05/22 19:24:45 | stderr | /run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
2024/05/22 19:24:45 | stderr | s6-rc: warning: unable to start service post-tailscaled: command exited 1
2024/05/22 19:24:45 | stderr | For more information run ethtool -h
2024/05/22 19:24:45 | stderr | ethtool: bad command line argument(s)
2024/05/22 19:24:45 | stderr | [19:24:45] INFO: �[32mAllow UDP GRO for forwarding on eth0�[0m

josefsedy · 2024-05-22T18:39:40Z

it seems that the whole problem is on my Synology NAS where I have the container manager running. Advanced SSH & Web Terminal behaves as non-standard as Tailscale.

lmagyar · 2024-05-22T19:50:20Z

OK, when I saw ethtool: bad command line argument(s) and netlink error: bit name not found (offset 56) I gave up and reverted the root cause. This was a suggested optimization from the TS log.

Tisoza · 2024-05-23T06:43:18Z

both v0.19.1 and v0.19 no longer work for me either. Everything up to 0.18 worked without problems
In Home Assistant, the situation seems fine, however in Synology NASs container, Tailscale addon does not start.

It that any way to come back to v0.18 version?

lmagyar · 2024-05-23T07:32:00Z

Is that any way to come back to v0.18 version?

It should. I think add-on version 0.19.0 had some general tailscale issues and some add-on bugs, 0.19.1 has only add-on bugs, reverted with #370.

josefsedy · 2024-05-23T09:01:49Z

Unfortunately, version 0.19.1 is just as broken in Container Manager as version 0.19.0. Version 0.18.0 worked fine.

Tisoza · 2024-05-23T10:38:31Z

Unfortunately, version 0.19.1 is just as broken in Container Manager as version 0.19.0. Version 0.18.0 worked fine.

do you know a simple way to install version v0.18.0?

frenck · 2024-05-23T10:41:41Z

do you know a simple way to install version v0.18.0?

Restore a backup is the only way.

josefsedy · 2024-05-23T13:15:26Z

Unfortunately, I don't have the 0.18.0 version backed up. But version 0.17.0 is working.

tdejneka · 2024-05-23T20:21:43Z

FWIW, version 0.19.1 failed to connect my instance of Home Assistant to my tailnet. The addon starts then immediately stops. I don't recall seeing an explicit error message, just that nginx would report it was stopping mere seconds after it started. I needed this fixed ASAP so I restored my backup of version 0.18 and it now works properly (i.e. my Home Assistant server is now visible in the latest version of the Tailscale Android app).

Obviously something that changed between 0.18 and 0.19 appears to be the culprit.

lmagyar · 2024-05-23T22:30:22Z

Yes, #370 is under review, final version will try to keep the optimization, but the add-on will start even if it fails in any way.

c2mfj closed this as not planned Won't fix, can't repro, duplicate, stale May 21, 2024

c2mfj reopened this May 21, 2024

lmagyar mentioned this issue May 22, 2024

Failsafe enabling of UDP GRO for forwarding #370

Merged

frenck closed this as completed in #370 May 24, 2024

github-actions bot locked and limited conversation to collaborators Jun 23, 2024

Connectivity to Hass suddenly stopped - no known changes occured #362

Connectivity to Hass suddenly stopped - no known changes occured #362

Comments

c2mfj commented May 20, 2024

Problem/Motivation

Expected behavior

Actual behavior

lmagyar commented May 20, 2024

esenliyim commented May 20, 2024 • edited Loading

josefsedy commented May 20, 2024

c2mfj commented May 20, 2024 • edited Loading

lmagyar commented May 20, 2024

lmagyar commented May 20, 2024

c2mfj commented May 20, 2024

lmagyar commented May 20, 2024

esenliyim commented May 20, 2024 • edited Loading

c2mfj commented May 20, 2024

lmagyar commented May 20, 2024

c2mfj commented May 21, 2024

esenliyim commented May 21, 2024

c2mfj commented May 21, 2024

c2mfj commented May 21, 2024

itmokel commented May 22, 2024 • edited Loading

robert10055 commented May 22, 2024

ovizii commented May 22, 2024

gentertain commented May 22, 2024

josefsedy commented May 22, 2024

lmagyar commented May 22, 2024

lmagyar commented May 22, 2024

itmokel commented May 22, 2024

epposan commented May 22, 2024

ovizii commented May 22, 2024

lmagyar commented May 22, 2024

lmagyar commented May 22, 2024

itmokel commented May 22, 2024

epposan commented May 22, 2024

frenck commented May 22, 2024

epposan commented May 22, 2024

frenck commented May 22, 2024 • edited Loading

ovizii commented May 22, 2024

robert10055 commented May 22, 2024

lmagyar commented May 22, 2024

lmagyar commented May 22, 2024

rlad78 commented May 22, 2024 • edited Loading

josefsedy commented May 22, 2024

lmagyar commented May 22, 2024 • edited Loading

lmagyar commented May 22, 2024

rlad78 commented May 22, 2024

josefsedy commented May 22, 2024

josefsedy commented May 22, 2024

josefsedy commented May 22, 2024

lmagyar commented May 22, 2024

Tisoza commented May 23, 2024

lmagyar commented May 23, 2024

josefsedy commented May 23, 2024

Tisoza commented May 23, 2024

frenck commented May 23, 2024

josefsedy commented May 23, 2024

tdejneka commented May 23, 2024

lmagyar commented May 23, 2024

esenliyim commented May 20, 2024 •

edited

Loading

c2mfj commented May 20, 2024 •

edited

Loading

esenliyim commented May 20, 2024 •

edited

Loading

itmokel commented May 22, 2024 •

edited

Loading

frenck commented May 22, 2024 •

edited

Loading

rlad78 commented May 22, 2024 •

edited

Loading

lmagyar commented May 22, 2024 •

edited

Loading