Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Studio Code Server doesn't work on access to HA by https with NGINX proxy #584

Closed
vdemidov opened this issue Mar 6, 2023 · 29 comments
Closed

Comments

@vdemidov
Copy link

vdemidov commented Mar 6, 2023

Problem/Motivation

After addon update to version 5.5.3 it doesn't work when HA accseed over https.

Expected behavior

Expected normally work with ingress by http and https

Actual behavior

Work only with direct access to HA by http.

When accessed by https with NGINX proxy error on screen after very long time and errros in log
Error text:

An unexpected error occurred that requires a reload of this page.
The workbench failed to connect to the server (Error: Time limit reached)

Steps to reproduce

Home Assistant Core: 2023.3.1
Home Assistant Supervisor: 2023.01.1
Home Assistant config:

http:
  server_port: 8123
  use_x_forwarded_for: true
  trusted_proxies:
    - 172.30.33.0/24

Addon NGINX Home Assistant SSL proxy
Config:

domain: mydomain.duckdns.org
hsts: max-age=31536000; includeSubDomains
certfile: fullchain.pem
keyfile: privkey.pem
cloudflare: false
customize:
  active: false
  default: nginx_proxy_default*.conf
  servers: nginx_proxy/*.conf

Addon's log

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service base-addon-banner: starting
-----------------------------------------------------------
 Add-on: Studio Code Server
 Fully featured Visual Studio Code (VSCode) experience integrated in the Home Assistant frontend.
-----------------------------------------------------------
 Add-on version: 5.5.3
 You are running the latest version of this add-on.
 System: Debian GNU/Linux 11 (bullseye)  (amd64 / qemux86-64)
 Home Assistant Core: 2023.3.1
 Home Assistant Supervisor: 2023.01.1
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
s6-rc: info: service base-addon-banner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service base-addon-timezone: starting
s6-rc: info: service base-addon-log-level: starting
s6-rc: info: service fix-attrs successfully started
[10:33:46] INFO: Configuring timezone (Europe/Kiev)...
s6-rc: info: service base-addon-log-level successfully started
s6-rc: info: service base-addon-timezone successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-user: starting
s6-rc: info: service init-mysql: starting
s6-rc: info: service init-mosquitto: starting
s6-rc: info: service init-code-server: starting
s6-rc: info: service init-code-server successfully started
s6-rc: info: service init-user successfully started
s6-rc: info: service code-server: starting
s6-rc: info: service code-server successfully started
[10:33:47] INFO: Starting code-server...
s6-rc: info: service init-mysql successfully started
s6-rc: info: service init-mosquitto successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
[2023-03-06T08:33:47.604Z] info  Wrote default config file to ~/.config/code-server/config.yaml
[2023-03-06T08:33:48.053Z] info  code-server 4.10.1 d477972c68fc8c8e8d610aa7287db87ba90e55c7
[2023-03-06T08:33:48.054Z] info  Using user-data-dir /data/vscode
[2023-03-06T08:33:48.067Z] info  Using config file ~/.config/code-server/config.yaml
[2023-03-06T08:33:48.067Z] info  HTTP server listening on http://0.0.0.0:1337/
[2023-03-06T08:33:48.067Z] info    - Authentication is disabled
[2023-03-06T08:33:48.067Z] info    - Not serving HTTPS
[10:35:40] 
[10:35:40] Extension host agent started.
[2023-03-06T08:35:40.457Z] error Forbidden HttpError: Forbidden
    at ensureOrigin (/usr/local/lib/code-server/out/node/http.js:288:15)
    at wrapped (/usr/local/lib/code-server/out/node/wsRouter.js:64:24)
    at Layer.handle [as handle_request] (/usr/local/lib/code-server/node_modules/router/lib/layer.js:102:15)
    at next (/usr/local/lib/code-server/node_modules/router/lib/route.js:144:13)
    at Route.dispatch (/usr/local/lib/code-server/node_modules/router/lib/route.js:109:3)
    at handle (/usr/local/lib/code-server/node_modules/router/index.js:515:11)
    at Layer.handle [as handle_request] (/usr/local/lib/code-server/node_modules/router/lib/layer.js:102:15)
    at /usr/local/lib/code-server/node_modules/router/index.js:291:22
    at param (/usr/local/lib/code-server/node_modules/router/index.js:368:14)
    at param (/usr/local/lib/code-server/node_modules/router/index.js:379:14)
    at Function.process_params (/usr/local/lib/code-server/node_modules/router/index.js:424:3)
    at next (/usr/local/lib/code-server/node_modules/router/index.js:285:10)
    at Function.handle (/usr/local/lib/code-server/node_modules/router/index.js:184:3)
    at router (/usr/local/lib/code-server/node_modules/router/index.js:59:12)
    at Layer.handle [as handle_request] (/usr/local/lib/code-server/node_modules/router/lib/layer.js:102:15)
    at trim_prefix (/usr/local/lib/code-server/node_modules/router/index.js:330:13)
    at /usr/local/lib/code-server/node_modules/router/index.js:294:7
    at Function.process_params (/usr/local/lib/code-server/node_modules/router/index.js:349:12)
    at Immediate.next (/usr/local/lib/code-server/node_modules/router/index.js:285:10)
    at Immediate.<anonymous> (/usr/local/lib/code-server/node_modules/router/index.js:671:15)
    at processImmediate (node:internal/timers:468:21)

Home Assistant Supervisor log:
23-03-06 10:42:05 ERROR (MainThread) [supervisor.api.ingress] Ingress error: 403, message='Invalid response status', url=URL('http://172.30.33.3:1337/stable-441438abd1ac652551dbe4d408dfcec8a499b8bf?reconnectionToken=ed7a7470-a38d-4b38-ac08-25ba3d197d48&reconnection=false&skipWebSocketFrames=false')

@vdemidov vdemidov changed the title Doesn't work on access to HA by https with NGINX proxy Studio Code Server doesn't work on access to HA by https with NGINX proxy Mar 6, 2023
@vdemidov
Copy link
Author

vdemidov commented Mar 6, 2023

After rollback to previous version 5.5.2 all works fine:
Log:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service base-addon-banner: starting
-----------------------------------------------------------
 Add-on: Studio Code Server
 Fully featured Visual Studio Code (VSCode) experience integrated in the Home Assistant frontend.
-----------------------------------------------------------
 Add-on version: 5.5.2
 There is an update available for this add-on!
 Latest add-on version: 5.5.3
 Please consider upgrading as soon as possible.
 System: Debian GNU/Linux 11 (bullseye)  (amd64 / qemux86-64)
 Home Assistant Core: 2023.3.1
 Home Assistant Supervisor: 2023.01.1
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
s6-rc: info: service base-addon-banner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service base-addon-timezone: starting
s6-rc: info: service base-addon-log-level: starting
s6-rc: info: service fix-attrs successfully started
[10:45:33] INFO: Configuring timezone (Europe/Kiev)...
s6-rc: info: service base-addon-log-level successfully started
s6-rc: info: service base-addon-timezone successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-user: starting
s6-rc: info: service init-mysql: starting
s6-rc: info: service init-mosquitto: starting
s6-rc: info: service init-code-server: starting
s6-rc: info: service init-code-server successfully started
s6-rc: info: service init-user successfully started
s6-rc: info: service code-server: starting
s6-rc: info: service code-server successfully started
[10:45:33] INFO: Starting code-server...
s6-rc: info: service init-mysql successfully started
s6-rc: info: service init-mosquitto successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
[2023-03-06T08:45:34.031Z] info  Wrote default config file to ~/.config/code-server/config.yaml
[2023-03-06T08:45:34.403Z] info  code-server 4.9.1 f7989a4dfcf21085e52157a01924d79d708bcc05
[2023-03-06T08:45:34.405Z] info  Using user-data-dir /data/vscode
[2023-03-06T08:45:34.424Z] info  Using config file ~/.config/code-server/config.yaml
[2023-03-06T08:45:34.424Z] info  HTTP server listening on http://0.0.0.0:1337/ 
[2023-03-06T08:45:34.424Z] info    - Authentication is disabled 
[2023-03-06T08:45:34.424Z] info    - Not serving HTTPS 
[10:45:43] 
[10:45:43] Extension host agent started.
[10:45:44] [172.30.32.2][1ad6f571][ManagementConnection] New connection established.
[10:45:44] [172.30.32.2][fbd6cb91][ExtensionHostConnection] New connection established.
[10:45:44] [172.30.32.2][fbd6cb91][ExtensionHostConnection] <434> Launched Extension Host Process.

@raphaabreu

This comment was marked as off-topic.

@L3st86
Copy link

L3st86 commented Mar 6, 2023

@vdemidov , how did you did the rollback? Happening to me also.

Thks!!

@frenck
Copy link
Member

frenck commented Mar 6, 2023

There is no rollback, the only thing Home Assistant offers is restoring backups.

../Frenck

@L3st86
Copy link

L3st86 commented Mar 6, 2023

@frenck , yep, thanks. Restored from backup and working!

@frenck
Copy link
Member

frenck commented Mar 6, 2023

Please stop commenting +1 anywhere on GitHub, and just use an emoji reaction on the opening post instead.

Those "same here" and +1 posts are highly polluting, annoying and absolutely add zero value to triaging an issue or the conversation around it.

../Frenck

@thor0215
Copy link

thor0215 commented Mar 8, 2023

I think this update to code-server is causing the issue:
Add origin checks to web sockets
coder/code-server#6048

This is confirmed when you do a browser network trace, the ws connection never comes back successfully.
2023/03/08 12:06:06 [error] 453#453: *74 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.21, server: homeassistant.domain.com, request: "GET /api/hassio_ingress/wnwxqjP0OT33gh-mNT2ZR2mMURC1_upU77nkdlXpn6I/stable-441438abd1ac652551dbe4d408dfcec8a499b8bf?reconnectionToken=64945fd5-291e-4790-b2ab-389090fdb666&reconnection=false&skipWebSocketFrames=false HTTP/1.1", upstream: "http://172.30.32.1:48123/api/hassio_ingress/wnwxqjP0OT33gh-mNT2ZR2mMURC1_upU77nkdlXpn6I/stable-441438abd1ac652551dbe4d408dfcec8a499b8bf?reconnectionToken=64945fd5-291e-4790-b2ab-389090fdb666&reconnection=false&skipWebSocketFrames=false", host: "homeassistant.domain.com:8123"

@thor0215
Copy link

thor0215 commented Mar 8, 2023

Also:
https://github.com/coder/code-server/releases/tag/v4.10.1
Code v1.75.1

Security
Added an origin check to web sockets to prevent cross-site hijacking attacks on
users using older or niche browser that do not support SameSite cookies and
attacks across sub-domains that share the same root domain.

The check requires the host header to be set so if you use a reverse proxy
ensure it forwards that information otherwise web sockets will be blocked.

@thor0215
Copy link

thor0215 commented Mar 9, 2023

I was able to fix this by adding this to the Nginx Proxy Manager Custom Nginx Configuration:

location / {
# Needed to workaround VSCode Web Socket Origin error
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
proxy_set_header Origin "";

# Proxy!
include conf.d/include/proxy.conf;
}

@TheJulianJES
Copy link

In that case, proxy_set_header Origin ""; is likely enough. It's not really a "fix" though if you completely remove the origin. The added checks are all disabled then:
https://github.com/coder/code-server/blob/be40eca5d92ac2edbd3196e50df1493272431ded/src/node/http.ts#L341-L346

@thor0215
Copy link

thor0215 commented Mar 9, 2023

In that case, proxy_set_header Origin ""; is likely enough. It's not really a "fix" though if you completely remove the origin. The added checks are all disabled then: https://github.com/coder/code-server/blob/be40eca5d92ac2edbd3196e50df1493272431ded/src/node/http.ts#L341-L346

Oh, I'm aware of that. I shouldn't say it's a fix as it's really a workaround. I just can't see where the differences are between my host and origin with the Nginx logging turned up to trace. I would love a better solution.

@thor0215
Copy link

thor0215 commented Mar 9, 2023

Here's another workaround that doesn't strip the Origin header but instead makes sure to send the x-forwarded-host header

location / {
# Needed to workaround VSCode Web Socket Origin error
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-Host $http_host;

# Proxy!
include conf.d/include/proxy.conf;
}

@tomasbedrich
Copy link

tomasbedrich commented Mar 12, 2023

I had the same problem with my reverse proxy setup. Turns out to be misconfiguration on my side, where I had the following in Caddyfile:

...
    header_up Host {upstream_hostport}
    header_up -X-Forwarded-*
...

Effectively I was stripping a proxy info and pretending that I'm connected locally.

I removed these two directives altogether (default Caddy reverse_proxy config handles it well) and instead fixed my configuration.yaml according to the standard reverse proxy guide.

What a noob mistake.. Thank you guys for the kick! 😊

Based on this, I guess the issue can be closed. It's likely that the recent VS code release only reveals a proxy misconfiguration.

@vdemidov
Copy link
Author

Based on this, I guess the issue can be closed. It's likely that the recent VS code release only reveals a proxy misconfiguration.

Could you help with create issue for Home Assistant Add-on: NGINX Home Assistant SSL proxy?

@tomasbedrich

This comment was marked as outdated.

@adamkoch
Copy link

In my case, I have a separate Nginx server (not using the HA addon) - the proxy config looks the same as the HA addon one already.

What do I need to set in the Nginx config files to mirror trusted_proxies in the HA addon yaml?

@adamkoch
Copy link

What do I need to set in the Nginx config files to mirror trusted_proxies in the HA addon yaml?

Oh actually I realized this is HA http config - I actually have trusted_proxies set correctly already, but vscode still no longer works after upgrading to the latest as per this bug. Any ideas?

@vdemidov
Copy link
Author

No need to create issue for Nginx addon. The proxy is already configured properly. All you need to do is to setup trusted_proxies according to the guide.

If you watch to start message i have Nginx addon with default config and trusted_proxies in HA config. And have problems with VSCode.

@tomasbedrich

This comment was marked as off-topic.

@thor0215
Copy link

@vdemidov did you look at my suggestion from my comment above

@vdemidov
Copy link
Author

vdemidov commented Mar 13, 2023

@vdemidov did you look at my suggestion from my comment above

Where is this config should be in Nginx addon?

Could you help with creating issue for Home Assistant Add-on: NGINX Home Assistant SSL proxy?

@thor0215
Copy link

I've created an issue for nginx-proxy-manager
hassio-addons/addon-nginx-proxy-manager#407

@TheHolyRoger
Copy link

There is NO support for X-Forwarded-Host in the official Nginx Home assistant SSL proxy addon.

I fiddled with various things in the config and in the end gave up and reverted to 5.5.2.

@frenck don't mean to sound like an arse but this should really go as a big warning in the release notes mate...

Based on the above comments, I suspect changing the base config for the nginx addon here:
https://github.com/home-assistant/addons/blob/master/nginx_proxy/data/nginx.conf#L66

And adding the X-Forwarded-Host line would solve it. An easy fix for the maintainer who also happens to be @frenck ;)

I have no need for the addon-nginx-proxy-manager addon so I don't plan to set it up just to fix this issue...

For anyone else reading this and using the official nginx addon, save yourself a headache and do a partial restore on vscode to downgrade it. There is no easy fix until a new version of the nginx addon is released.

@adamkoch
Copy link

adamkoch commented Mar 14, 2023

Oh actually I realized this is HA http config - I actually have trusted_proxies set correctly already, but vscode still no longer works after upgrading to the latest as per this bug. Any ideas?

For those who have a separate NGINX setup that is basically stock config, I was able to fix vscode loading by changing:

proxy_set_header X-Forwarded-Host $host;

to

proxy_set_header X-Forwarded-Host $http_host;

(as per @thor0215's post)

@charettepa
Copy link

what about those with the HA nginx.
I no longer have my backup as its over 5 days old.

How can we solve this?

@TheHolyRoger
Copy link

@charettepa all I can suggest is to make yourself heard here home-assistant/addons#2912

Or use the other nginx addon I guess

@TheHolyRoger
Copy link

I have confirmed the fix for stock nginx addon and opened a PR - if anyone wants steps to manually patch the nginx addon config until the addon is updated let me know.

I'm not too sure if the patch persists through a restart though :) I'll have to find out ;)

@frenck
Copy link
Member

frenck commented Mar 15, 2023

Closing this issue, as this is not an issue of this add-on.

../Frenck

@frenck frenck closed this as not planned Won't fix, can't repro, duplicate, stale Mar 15, 2023
@kubiksamek
Copy link

kubiksamek commented Mar 21, 2023

@charettepa
In the meantime you can hotfix it like this:

  1. ssh to HA as root
  2. cd /mnt/data/supervisor/addons/data/a0d7b954_nginxproxymanager/nginx/proxy_host/
  3. find config for your HA
  4. edit desired conf file with proxy_set_header X-Forwarded-Host $http_host; in location / block
  5. restart nginx addon

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests