Raw pass-through option

[asthomas]

This is an extremely useful module. Thanks for sharing it.

I am using the "vncproxy" plugin from here: https://github.com/abligh/apache-websocket
This plugin is providing a WebSocket-to-plain-TCP proxy that effectively turns any back-end TCP server into a WebSocket enabled server. It works very well.

I have run into a situation where the back-end server is expecting a WebSocket connection, not a plain TCP connection. I understand that it would be possible to modify the plugin to re-wrap the TCP data back into WebSockets frames, but that seems pretty wasteful, particularly on the client leg of the journey where it would mean unpacking and then re-packing the data as base64.

How hard would it be to modify apache-websocket to add an option that basically just says "pass the headers and data intact without modification to the plugin". It seems like one possibility would be to extend the interface between mod-websocket and the plugin to include two functions:

1. pre-process headers. Here the plugin could process the headers in some way. In the case of a websocket proxy that would mean transmitting them unmodified to the back-end server.
2. pre-process raw data. Here the plugin could see the raw data, including framing, from the client. It could choose to transmit it unmodified to the back-end server. A return code could indicate to mod-websocket that the data had been dealt with and that no further processing (like base64 decoding) is required.

I suppose it's reasonable to ask - why not use mod_proxy? I think there are two reasons. First, mod_proxy does a lot of header modification, and second, mod_proxy does not spawn a thread for each connection so it can handle only a small number of simultaneous connections.

I know there is a project called mod_websocket on github, but it is non-functional and single-threaded. I think it's the wrong starting point for a raw websocket proxy.

[disconnect]

I have a local branch that has support for WebSocket proxying. It isn't ready for general use, and I haven't been able to work on it for over a month now. If all goes well, perhaps I can find some time soon.

[asthomas]

I got something working based on the vncproxy plugin. It's a bit of a hack in the sense that it takes the decoded websocket packets and re-encodes them to send to the back-end server. It turns out that I needed to add options to mod_proxy to avoid sending the headers twice, and to give plugins access to the WebSocket procotol version and the draft76 "key3" data from the message body. On the up-side the vncproxy code has solved all the problems around handling the back-end TCP socket in a separate thread. I haven't quite worked out how to do a volume test yet - I want to have 1000 WebSockets open at once. I think mod_proxy + vncproxy may allow it.

[asthomas]

Having said that, I'm stuck on proxying SSL connections through to the back-end server. I very much look forward to seeing this done properly. Please let me know if I can offer any assistance.

[pwaller]

@disconnect, any ETA? Any harm of you putting up the branch on github? I'd love to try and use this.

[disconnect]

OK, I finally got around to pushing my experimental "proxy" branch (switch to that branch to give it a try). It may not work on all platforms. I should update the README, but for now hopefully this will help...

Include a WebSocketProxy line instead of a WebSocketHandler line. Specify the URL to the WebSocket server that will be handling the request. Make sure that the scheme is "ws" ("wss" isn't currently supported). Also, it only uses the hostname and the port for now (the path will be ignored). If you have problems with hostname lookups, specify the IP address. Here is an example:

<IfModule mod_websocket.c>
  <Location /echo>
    SetHandler websocket-handler
    WebSocketProxy "ws://127.0.0.1:8080/"
  </Location>
</IfModule>

[asthomas]

Thank you for posting your proxy code. I looked at the changes and I do not understand the threading model. Is a new thread being created for each proxied WebSocket, or is a single thread handling multiple WebSocket proxies?

Do you plan to add wss: support? That's the part that has me stumped. Will this code accept an incoming SSL connection and produce an outgoing plain-text connection? That would actually be usable for some applications.

Do you plan to add support for a path in the target address? If not then I can take a crack at that when I have some time. That's necessary for any back-end server that is supplying both WebSocket and HTTP services, as in my case.

[disconnect]

Each proxy connection is handled in the thread/process in which the request was created from Apache. No additional threads are created to support proxying.

Yes, I intend to add support for proxying to secure WebSocket servers eventually. The module supports incoming SSL connections, just not outgoing to the underlying target server right now.

I just pushed an update that has support for a path in the target URL (e.g., ws://127.0.0.1:8080/blah). It does not support a query after the path, however (e.g., /blah?x=1).

[asthomas]

A WebSocket is a long-lived connection, lasting hours or days. If each WebSocket proxy holds the original Apache thread, won't that consume all of the Apache workers very quickly? It looks as if my Apache server, with 16 workers, will be unable to handle either the 17th WebSocket proxy or any other HTTP request. That's not really practical in a production system. A WebSocket server could potentially handle hundreds or even thousands of simultaneous connections, which would mean the same number of proxies in Apache.

How would we go about spawning a new thread for each WebSocket proxy?

[pengc99]

I'm having a strange problem with the raw pass through branch (proxy)

On config for the location (/socket.io) I add a ws://127.0.0.1:7676/socket.io as the destination

However, Apache continually tells me 404 not found and the request is never forwarded to the socket.io application.

Any known problems?

        <IfModule mod_websocket.c>
                <Location /socket.io>
                        SetHandler websocket-handler
                        WebSocketProxy "ws://127.0.0.1:7676/socket.io"
                </Location>
        </IfModule>

[disconnect]

Since you are using socket.io as your server, I'm assuming that you are using standard Apache proxying to get the other socket.io resources. I think the problem is with your Location entry. Socket.io sends more than just /socket.io in the WebSocket request, so you will need to put a wildcard on that location. In this case, ideally the path and query passed to Apache in a request would be forwarded to the socket.io server. I don't have code in place for that, as you have to specify the hard-coded URL that you want to use. If you want to see if this is the problem, try replacing <Location /socket.io> with <Location ~ "/socket.io/*">. Then change the WebSocketProxy to "ws://127.0.0.1.7676/socket.io/1/websocket/". That may "partially" work, but it won't be passing the client id in the request URL. To handle it properly for your specific case, modify the source code so that the proxy path comes from the the requested path, not the hard-coded one in the config file. It seems that handling wildcards in the location request and then allowing those values to be included in the WebSocketProxy string would be something useful to implement.

[ntisservices]

Thanks a lot for sharing this disconnect. I've got it working using a tornado server, my plan is to use Stomp in HornetQ as the web socket implementation so that it will fit in well with our Java architecture. There are a few developers using Windows on the team so it'll be interesting whether the module compiles under Windows or not. Fortunately our target infrastructure is Linux.

[mantesap]

Thanks for the branch. Using WebSocketProxy "ws://127.0.0.1:8080/path/to/websocket" I can now pass the websocket handling to a tomcat instance but for some reason it fails when the initial request to apache is made on wss://. Your echo sample works fine for wss but when passing to tomcat echo websocket something happens and there is no response, although tomcat seems to receive and reply to the initial request. Any ideas ?

[jej2003]

I also am seeing tomcat terminating requests that start as wss:// and are proxied to ws://. I tried to step into the code a little, but it appears that tomcat is closing the request after trying to read the WebSocket frame and reaching the end of the input stream. Unfortunately, I don't know enough about the protocol or either implementation to truly understand where things are going wrong though. Maybe someone will have a better idea....

[mantesap]

If anyone else is still looking for a solution, our problems were solved using the mod_proxy_wstunnel module which is currently part of the apache httpd trunk

[jej2003]

Were you able to connect to an ssl web socket on the backend?
On Apr 5, 2013 1:54 AM, "mantesap" [email protected] wrote:

If anyone else is still looking for a solution, our problems were solved
using the mod_proxy_wstunnel module which is currently part of the apache
httpd trunk

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/24#issuecomment-15940105
.

[asthomas]

Looking at the source for mod_proxy_wstunnel, it doesn't appear that there is a separate thread being created to handle long-lived tunnels. Can anybody confirm this? It looks like I could trivially mount a denial of service attack on Apache through mod_proxy_wstunnel just by opening a web page using a websocket as many times as there are worker threads in Apache (10? 16?). After that, won't Apache stop responding to connections?

[mantesap]

jej2003, no but I can try and let you know. In our case SSL is terminated on apache and we have the same
ProxyPass from front to back in both cases. My guess is that it will work though.

asthomas, I will also try and test but I consider mod_proxy_wstunnel the long term viable option being part of the apache trunk