Unable to connect over webockets with TLS (wss) from Firefox

[stepanmracek]

Hi,
I'm running Mosquitto v1.5.8 with enabled websockets, but I'm unable to connect from Firefox when the broker is running over TLS. Strange thing is that It works from Chrome/Chromium. I tried both MQTT.js as well as Paho javascript client libraries with no success.

Connecting always ends with error Firefox can’t establish a connection to the server at wss://localhost:9001/mqtt.

Environment:

• Mosquitto: 1.5.8
• libwebsockets: 3.1.0
• openssl: 1.1.1.b

Here is my broker configuration:

allow_anonymous false
password_file /home/stepo/tmp/mqtt_pwd

listener 9001
protocol websockets

cafile /home/stepo/tmp/mqtt-certs/ca.crt
certfile /home/stepo/tmp/mqtt-certs/server.crt
keyfile /home/stepo/tmp/mqtt-certs/server.key

(When I comment the cafile, certfile and keyfile entries and thus running on ws:/ everything in Firefox works)

Here you can find GIST for JS Paho client: https://gist.github.com/stepanmracek/9b6d4a6ee9ef9d4d6efe738c069a9577
I always outputs following error to the console: AMQJS0007E Socket error:undefined.

HINT

• I've also tried Mosquitto 1.4.11 with libwebsockets 1.6.3 and it works
• On another machine I have Mosquitto 1.4.15 but this time with libwebsockets 3.0.0 and there I'm facing the same issue -> does not work from Firefox.

Therefore I'm suspecting that this is some regression in libwebsockets that is unfortunately reproducible only in Firefox.

[ralight]

Thanks for the hints, I'll take a look. I'd suggest using libwebsockets 2.4.x, that has proved to be a pretty reliable version.

[timothy-godfrey]

Hi!
I can confirm similar behaviour: secure Websockets works between chrome/chromium and Mosquitto, but not between Firefox(v68) and Mosquitto, using Mosquitto v1.6.4 and libwebsockets-3.1. For the time being I will revert to libwebsockets-2.4

I have uploaded a couple of filtered Wireshark captures to Firefox Send here one for Firefox not working and one for Chrome working. Chrome appears to be using Websockets over TLS, whereas Firefox appears to be using Websockets over http2.

In the Firefox capture things seem to go awry at packet 34 when the Mosquitto host (192.168.3.28) sends RST_STREAM followed by GO_AWAY. The host client running firefox (192.168.3.23) replies with its own GO_AWAY (for the control stream?), and after that a TLS 'Close Notify' Alert. Then the Mosquitto host sends the TCP RST packets.

I don't know how to get more debugging information without rebuilding Mosquitto and libwebsockets, so I don't have any insights into what is causing the Mosquitto host to reset the http2 stream. My impression is that something from Firefox is malformed/unexpected from the libwebsockets point of view and so libwebsockets responds by resetting the stream.

I hope that is useful!

[timothy-godfrey]

Writing down the above led me to consider that it was a WSS over http/2 problem. So I recompiled libwebsockets-3.1 with http2 support disabled, and now it works! I can connect to my Mosquitto broker over WSS from Firefox and Chrome.

[ralight]

@timothy-godfrey Are you able to try the current mosquitto fixes branch with libwebsockets compiled with http2 support? There are some libwebsockets related changes in fixes. Or maybe with libwebsockets 3.2?

[vincent-lefevere]

Today, I download the last version of libwebsockets-3.2 and I compile it with http2 support. Next, I compile mosquitto (version 1.6.8). Result: Firefox(V71.0) does not want to use mqtt over wss. No trouble with Chrome!
When I compile libwebsockets-3.2 (as well as libwebsocket-3-1) without http2 support and next I compile mosquitto, then Firefox work as well as Chrome.

[jonjonarnearne]

You can workaround this in firefox by setting network.http.spdy.websockets to false in about:config.

This seems to be a bug in mosquitto, as the libwebsockets-test-server is working fine with wss over http/2.

[jonjonarnearne]

Just tested this with a connection to wss://test.mosquitto.org:8081/mqtt. And this fails if we are using wss over http/2.

[jonjonarnearne]

The problem seems to be be that libwebsockets claim to support http/2, so firefox tries that,
but then mosquitto closes the connection?

[ralight]

@jonjonarnearne libwebsockets on test.mosquitto.org isn't compiled with http2 support, so if it's advertising it as available, that was a bug in libwebsockets. I use firefox all the time, and don't have a problem connecting to https://test.mosquitto.org/sys/ssl.html , for example - I haven't modified about:config.

What is the best way for me to test this?

One thing that may be causing an issue with lws, is that in more recent versions the WITH_EXTERNAL_POLL option is turned off by default, so Mosquitto does not receive any callbacks for the network traffic.

[jonjonarnearne]

@ralight It seems you are completely right.
I'm not able reproduce the failing connection to wss://test.mosquitto.org:8081/mqtt.

There must have been something wrong with my test setup when this connection failed on me.

I'm sorry for the noise.

[gary-hammer]

This is an issue on my website where I use MQTT(wss) to show live data.
With Firefox, unless I use the about:config workaround, the connection is reset and the data is not updated.
If I use the about:config workaround, the connection is established and data is updated.

The problem is, users of the website aren't going to dig to find out why, they'll just leave.

[wiebeytec]

I wiresharked some TLS websocket traffic to a server of ours that has this issue. Ubuntu 20.04, libwebsocket 3.2.1-3, mosquitto 1.6.12, compiled ourselves, using standard Ubuntu libs.

Chromium never tries http2. Firefox does (by starting with PRI * HTTP/2.0\r\n\r\nSM). There are also answers in http2, so it's not just a bug in reporting it can handle http2 while it can't.

Could be the WITH_EXTERNAL_POLL. I don't know how Ubuntu's version is compiled, nor do I have an MQTT packet parser handy to see where it gets stuck.

I fixed it by putting an Nginx proxy in front of it, proxying using HTTP 1.1. I can afford it on this server, but it's not ideal, because you'll run into maximum number of connections, because of TCP ephemeral port exhaustion.

[ralight]

@wiebeytec Is there anything special to reproduce it? The easiest you can make it for me so I don't have to guess anything would be great.

The Ubuntu package doesn't have external poll support. I know there are problems around the 3.2.x versions, there are version specific fixes in Mosquitto to deal with them, but I know people have still had other problems with them.

[wiebeytec]

No, just use Ubuntu 20.04, install buildessential and libsockets-dev, compile 1.6.12 and use Firefox and look in the console (CTRL-SHIFT-K). I didn't save the error messages unfortunately.

There may be a chance it only happens after a few messages. I don't have any other basis to say this than that our webapp got past the first step and then stalled. If you can't reproduce it, you can contact me privately, and I may be able to set up a dev server and have you connect to it.

[mister-teddy]

Hello everyone, what is the status of this issue? I'm having a project using mqtt over https. And our Firefox user have to use the about:config workaround in order to use the website 😞

[wiebeytec]

Is there a reason Mosquitto should support HTTP2 websockets? It seems like unnecessary overhead. MQTT over websocket is already packing frames in frames. With HTTP2 it's frames in frames in frames.

HTTP2 combines multiple requests into one connection to avoid connection overhead (and introduce new stalling problems). There is no reason for a MQTT web client to (repeatedly) open multiple connections to the same server, so packing data in yet an extra layer of frames seems like unnecessary overhead.

[ralight]

I've been able to figure out the cause of the first part of the problem. Mosquitto uses the incorrect wsi structure to try and get the network address when the client connection is established. This fails because it is the incorrect wsi and so the connection is terminated with no other problems. Unfortunately if I do use the correct wsi, then it still doesn't work for reasons I can't disclose at the moment.

I'm afraid the only solution right now is to compile libwebsockets without http2 support.

[mike-sandstrom]

Unfortunately if I do use the correct wsi, then it still doesn't work for reasons I can't disclose at the moment.
I'm afraid the only solution right now is to compile libwebsockets without http2 support.

I ran into this problem as well in firefox and safari and, as stated above, the workaround was turning off http2.

So if we already have mosquitto installed, do we have to remove libwebsockets and re install it without http2 support?

As stated on libwebsockets website https://libwebsockets.org/lws-api-doc-main/html/md_READMEs_README_8build.html
Enable -DLWS_WITH_HTTP2=1 in cmake to build with http/2 support enabled. is the option enabling http2. So now, we have to switch this to 0 when installing libwebsockets via cmake?

Have any other solutions to this been found? @ralight I saw you plan to move away from libwebsockets in favor of wslay. Is your recommendation to move to the mosquitto implementation using wslay?

[mike-sandstrom]

How can we utilize this newly merged commit to resolve this issue?

[jmico]

The current Debian 11 package of Mosquitto (2.0.11-1) has this issue.
TLS webockets work fine on Chrome, but does not on Firefox (unless "network.http.spdy.websockets" is set to false).
I was using 2.0.11 from http://repo.mosquitto.org on Debian 10 and had no problems with it.

[mike-sandstrom]

I believe the option is to build mosquitto using cmake and turn libwebsockets http2 off but I have not done that yet, also checking that the polling is set to on in LWS_WITH_EXTERNAL_POLL=ON

[Izerous]

I have 2 running instances of Mosquitto and both exhibit this issue with FF. However both instances were installed via http://www.steves-internet-guide.com/install-mosquitto-linux/ basically using apt-get not using go/make

i'm not really sure how to modify the currently installed versions when installed in that fashion to in turn disable http/2 to get around this issue. I don't want to keep telling people to use network.http.spdy.websockets false and would like to address this.

[jmico]

To get around this issue you need to install a libwebsockets version compiled with http2 support disabled.

If you are using Debian (or can switch to Debian) you just need to install both mosquitto 2.0.12 and libwebsockets 4.2.1 from http://repo.mosquitto.org (see the README).

Please note that, due to issue #2322, on Debian 11 the libwebsockets dependency is not installed automatically as it should and you must download manually the correct libwebsockets18 for your platform and then install it using dpkg.

[wiebeytec]

@Izerous do you have more than 30k clients? If not, you can install Nginx to proxy. You don't expose Mosquitto's websocket port, but use this example Nginx config to reach it:

server {
  listen 1.2.3.4:443 ssl reuseport;
  server_name webmqtt.mycompany.com;
  
  ssl_certificate     /etc/letsencrypt/live/webmqtt.mycompany.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/webmqtt.mycompany.com/privkey.pem;

  location / {
    proxy_pass http://127.0.0.1:4400;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";

    # Perhaps not necessary
    proxy_set_header Sec-Websocket-Protocol "mqtt";
  }

  include letsencrypt.inc;

  access_log /var/log/nginx/webmqtt.mycompany.com/access.log combined_plus;
  error_log  /var/log/nginx/webmqtt.mycompany.com/error.log;
}

If you do have more than 30k clients, you run into the TCP ephemeral port limit. You'll then have to work with multiple (bind) addresses.

[jmico]

As of version 2.0.14, Debian packages from http://repo.mosquitto.org have this issue fixed.

[hspaay]

@jmico, I was quite excited to read your note about 2.0.14.

However, installing the Ubuntu ppa from https://mosquitto.org/download/ (at the bottom), and installing 2.0.14, the problem is still there on Ubuntu 21.10 (Impish).

Where did you find the package with the fix?

[jmico]

As it is explained before to you need a libwebsockets compiled with http2 support disabled.

On Debian I've got libwebsockets 4.2.1 installed automatically from http://repo.mosquitto.org and it works fine (the actual packages can be browsed at https://repo.mosquitto.org/debian/pool/main/libw/libwebsockets/).

I can see on Ubuntu PPA that libwebsockets 4.2.1 package for 21.10 (Impish) is not available yet (the last one published is for 21.04).

[tombjarne]

Apparently the issue still persists even with the latest version (2.0.15). Chrome seems to be no issue, but Firefox is still rejecting the connection. Any news on the issue and the libwebsockets package? It would be interesting to hear about the workarounds you guys used to make it work in the long haul (if that's the case).

[jlambright]

We've found putting Mosquitto behind a reverse proxy seems to solve this problem. It's a stop-gap solution though.

[Rednaxela-beep]

Hello!
cmake .. -DLWS_WITH_HTTP2=0
does not works for me :(
same as cmake -LH LWS_WITH_HTTP2:BOOL=OFF ..
Is that correct?
How to check that http/2 already off?