Server responding with html error message causes an exponential number of retries

[moritz157]

I have my server running behind Apache as a reverse proxy. If the server is running everything is working fine. If I then stop the server the SSE connection dies and after 5 seconds it tries to reconnect. However if the server is still down Apache responds with a regular html error message. The SSEClient then, line by line, tries to parse the response to single events. Of course this fails for every single line. The problem is that if the parser cannot parse a line it calls _retryConnection while completely ignoring if a previous line already triggered a reconnect. Therefore a single response consisting of for example 10 lines triggers 10 connection retries simultaneously. If the server remains offline, each of those retries then trigger another 10 retries since Apache is still responding with an error. One minute of server downtime therefore results in 10^12 = 1 trillion requests at once (retry interval 5 seconds).

I accidentally kinda DOS'ed myself a number of times before I discovered the problem 😅
I think I am not the only one running my server behind a reverse proxy that responds like this if the server is down and I think this should be prevented.

The simplest solution to this would be to just cancel the response line stream subscription if an error occurs while parsing and therefore only allow one retry attempt. One giant downside of this approach though is that this only logs the first line of the response which isn't really helpful for debugging. So we should think about catching non-event-stream responses early and logging them in full before it could be split up line-by-line.