The widget fails to proxy in WebKit-based browsers

[noahlevenson]

UPDATE 12/02/2022 PART 2: In conjunction with #12, it sure seems that this issue is actually about WebKit.

UPDATE 12/02/2022 PART 1: This issue was previously focused on all mobile platforms, but after some basic sleuthing it seems that the bug is related to iOS only. See the conversation below...

In pursuit of https://github.com/getlantern/engineering/issues/78, I fired up the widget in both Chrome and Safari on my iPhone 8. The widget completes signaling as expected -- but when I try to proxy Firefox through it, it doesn't work.

The UI Current throughput and Lifetime data proxied values do bounce around a bit in the way I'd expect them to, indicating that we're pushing bytes through the thing.

Inspecting the mobile Chrome log, it looks like the widget's connection to the egress server is borking.

Because of the way we mux and demux protocols at the egress server, sometimes WebSocket borkage actually reflects an error at a higher level of the stack -- for example, it's possible that something is going wrong with an smux stream.

Attached: the widget in mobile Chrome on my iPhone, and the related chrome://inspect output.

[noahlevenson]

Update: the problem is racy. I can get an HTTP (not HTTPS) request to work via cURL about 50% of the time:

curl --proxy http://127.0.0.1:1080 http://info.cern.ch

I haven't been able to get an HTTPS request to work yet.

EDIT 12/02/2022: Given our recent focus on WebKit as the culprit -- and the fact that this bug is reproducible in desktop Safari -- it seems unlikely to be related to the old bug I described below.

We had a problem that exhibited identical symptoms during early development, and it was caused by the channel buffers being too small to keep up with the client data rate. This was before we'd panic when a nonblocking send failed (as a canary to warn us to increase the size of our channel buffers).

What'd happen is that we'd try to push more chunks than the client could keep up with, a send would encounter no listener at the other end of the channel (because the listener was still busy dealing with the previous chunk), we'd silently enter the default case for the send, and ultimately skip sending a chunk. It was easy to watch this happen for HTTPS requests -- the bytes you expect to see during a TLS handshake were arriving in a wacky order.

It's interesting that the same symptoms have reappeared when testing on resource constrained mobile devices, because our mobile phones are unlikely to maintain the same data rate as our desktop computers. What's weird, though, is that we don't seem to be triggering the panic.

I gotta keep digging...

[noahlevenson]

It gets a bit weirder.

The bug isn't reproducible in Chrome on my old Android tablet (Galaxy Tab A). On this particular mobile device, the widget just works.

[noahlevenson]

I fired up the widget in mobile Firefox on my iPhone, and it exhibits the same bug as it does while running under mobile Safari and mobile Chrome on that iPhone.

Since the widget runs fine in mobile Chrome on my Android tablet but has issues running under all mobile browsers on my iPhone, I've changed the focus of this ticket to iOS only.

My phone is an iPhone 8 running iOS 16.1.1.

Update: I bumped my iPhone 8 to iOS 16.1.2 and the issue persists.

[noahlevenson]

@woodybury wisely pointed out that the unifying theme here is WebKit, since all mobile browsers on iOS are mandated by Apple to use WebKit under the hood.

[noahlevenson]

In light of #12, I've changed the focus of this ticket to WebKit-based browsers.

Here's the error reported by @woodybury in desktop Safari:

WebSocket connection to 'ws://localhost:8080/ws' failed: The operation couldn't be completed. (kNWErrorDomainPOSIX error 100 - Protocol error)
writeSync — main.js:2:252752

[noahlevenson]

@woodybury solved it:

It's caused by an experimental WebKit feature, NSURLSession WebSocket, that Apple has turned on by default 🙄

If you disable that feature, everything works as expected.

See:

https://developer.apple.com/forums/thread/685403?page=2

I'm currently investigating a serverside workaround that doesn't require WebKit-based users to disable an experimental feature.

[noahlevenson]

Another good thread about the pain of NSURLSession WebSocket:

https://learn.microsoft.com/en-us/answers/questions/564726/ios-15-safari-set-nsurlsession-websocket-on.html

Arrest Tim Cook

[noahlevenson]

The underlying WebKit bug is related to Apple's broken implementation of WebSocket compression.

Fixed with a workaround. We disable WebSocket compression in the egress server: adb7c82

A couple more interesting discussions about the WebKit bug to leave you with:

gorilla/websocket#731

coder/websocket#218