Seeking advice for optimal multi-node socket.io server setup behind an AWS ALB

[janeklb]

I have an application comprised of 3 socket.io server instances all behind a single AWS Application Load Balancer, and the ALB has session stickiness enabled (using "duration-based stickiness").

The application logs a message on connection events emitted by the socket.io Server. In that log message I include the "sid" (via socket.client.id), as well as "server instance identifier" (the kubernetes pod name / replica) to see which server instance is handling the socket.io connection.

The application also logs a message on connection_error events emitted by engine.io. Those log messages include the context (which includes a sid for "Session ID unknown" errors), as well as the "server instance identifier".

I often see a connection log with a given sid coming from one replica, with a "Session ID unknown" connection_error (with the same given sid) coming from another replica. This suggests that (sometimes) session stickiness isn't working.

Is my ALB + socket.io setup sufficient for running multiple socket.io instances?
Does the load balancer have to use "application-based stickiness" instead, and if so why?

[darrachequesne]

I often see a connection log with a given sid coming from one replica, with a "Session ID unknown" connection_error (with the same given sid) coming from another replica. This suggests that (sometimes) session stickiness isn't working.

That sounds weird indeed. Does it happen regularly? Or for specific clients/browsers?

Is my ALB + socket.io setup sufficient for running multiple socket.io instances?

Yes, your setup looks good.

Do you use the same domain for your backend and your frontend? If not, did you set the withCredentials option to true, so the browser includes the AWSALBCORS cookie when reaching the backend?

References:

• https://socket.io/docs/v4/client-options/#withcredentials
• https://docs.aws.amazon.com/elasticloadbalancing/latest/application/sticky-sessions.html

[janeklb]

Hi @darrachequesne - thanks for the reply

That sounds weird indeed. Does it happen regularly? Or for specific clients/browsers?

"Session ID unknown" errors happen pretty regularly; however, for most of those occurrences I'm unable to cross-reference the sid to a "connection established" log event from another pod. In the cases where I am able to do this, I haven't noticed any patterns around specific clients/browsers but this is not something I've focused on.

Here's a screenshot of our logs to illustrate

note:

• time is descending (ie. read from bottom up)
• ive highlighted in orange a group of log entries that have the same sid
• the socket connection established message gets logged at the top of my server.on('connection', () => ... handler.
• the connection error is logged from server.engine.on('connection_error, () => ...`

Interestingly, there's a similar behaviour pattern that occurs right before the group of highlighted log messages except that the sid is different between the on('connection' and on('connection_error' events

Is it the client or the server that end up setting the sid? Is there any event that could be hooked into to see when the sid is actually being generated or when it's being re-used?

Do you use the same domain for your backend and your frontend?

Yes, same domain.

[janeklb]

Is there anything special that needs to be configured on the ALB routing rule that points at my socket.io server's path? Here's how we have them set up for our domain(s)

[darrachequesne]

According to the logs, it seems the session stickiness is not properly enabled. All requests with the same sid should reach the same pod.

Could you please check whether the AWSALB cookie is properly set in the headers?

Is it the client or the server that end up setting the sid?

The server sends the sid in the response of the first HTTP request, and the client includes it in the query parameters for subsequent HTTP requests.

[janeklb]

Could you please check whether the AWSALB cookie is properly set in the headers?

The cookie is definitely set. Here's a logs screenshot which shows (again, from bottom up) a socket.io connection getting established, followed by two "Session ID unknown" connection_error logs, followed by another socket.io connection getting established and then proceeding to function correctly.

The interesting thing here is that the server logs 3 different AWSALB cookie values. It also doesn't receive an AWSALB cookie on the very first request (but I'm assuming that's because the ALB attaches a Set-Cookie header on the response)