-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add WebSocketOptions.Timeout to ClientWebSocket #48729
Comments
Tagging subscribers to this area: @dotnet/ncl Issue DetailsI think that adding a WebSocketOptions.Timeout property would be a good thing. Today, the ClientWebSocket is closed when the TCP Retransmission times out. This timeout seems to be different on Windows and Linux. By default, on Windows it takes about 21 seconds and on Linux it takes about 15 minutes before the WebSocket is closed. It would be good if we could have a separate Timeout setting, so that we get the same behavior regardless of the platform. The ClientWebSocket today only sends "Pong" to the server, and therefore it does not wait for a "Pong" response to arrive. I think that the ClientWebSocket shall send a "Ping" instead, and then verify that a "Pong" is received within the Timeout set. If not received, the connection shall be closed. Or at least it should expose the "Ping" and "Pong" messages in the WebSocketReceiveResult.MessageType. Then the application could use these Ping/Pong message types to implement its own timeout logic.
|
Related: dotnet/aspnetcore#24614 |
This is a duplicate of: #35473 runtime/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs Lines 576 to 579 in b121bca
We might revisit the original decision to not to send Pings. |
I think it should be revisited. Today's implementation relies on the OS TCP/IP timeout setting to be able to determine when the WebSocket has been disconnected. It is not always wanted to change this setting in the OS as it effect all applications running there. Also. this OS setting differs by default between Windows and Linux. By default on Linux it takes about 15 minutes before the client realizes that the server is not there anymore. I think that it would be good to be able to set a timeout value on the ClientWebSocket itself(and use the PING/PONG frames internally to detect if a timeout occurs). Exposing the PONG messages is not needed if this is handled internally(preferred). |
Triage: we are open to adding proper ping/pong sending. At the moment, there's only this one issue asking for it so we'll leave it to future. |
If adding a new Timeout setting/property maybe the following would be something to consider when implementing:
|
Not so. Issue dotnet/aspnetcore#24614 is already asking for the same (and duplicate dotnet/aspnetcore#26117) |
Strong +1 for this issue. This is an important feature for applications like games where detecting a broken connection quickly is important for user experience. The WebSocket API does not expose the ability to send ping or pong messages, so there isn't even really a way to add this functionality in our applications without creating our own works-like ping/pong messages (which aren't really ping/pong messages but regular websocket messages). Just one point of clarification in the original issue — OP mentions that the current implementation only sends pong, rather than ping. I don't think that's true, from what I can see in ManagedWebSocket:518. I think the issue is simply that the implementation doesn't wait for a pong in response to the ping. The WebSocket RFC does not explicitly specify that clients should disconnect in the case a pong is not received in response to a ping within a certain timeframe. I think this would be a good thing to do though, as a user-configurable option. It should probably be default-off, so the change doesn't impact any applications which are somehow depending on very long retry times. Just adding an extra |
Note: there's another ask for this feature #60620 |
@gdalsnes @aroman can you please upvote top post? That will help us track number of people who need this behavior. Thanks! |
I guess SignalR uses its own pingpong, and is covering over for this flaw for most use cases? But I use it with vs-streamjsonrpc and ended up implementing my own pingpong, and used 99.9% time on that and the rest on actual work... My testing involved killing processes, pulling network cables, supending/resuming/killing VM's, "pulling" network "cables" within VM's etc. I easily got into situations with infinite hang (waited over night). |
I've updated the root description with the API proposal |
But will this cover aspnetcore websocket as well? It should also have the same timeout option, else it can hang forever and stall request pipelines (potentional DOS problem?) |
@gdalsnes what do you mean by "aspnetcore WebSocket"? |
@gdalsnes good point. This will need an additional API from ASP.NET Core side as well. I suspect it should be added to |
|
We need to ping from client and server every few seconds to make sure connection is still alive and restart asap if needed. Since this is not possible with official ClientWebSocket, I use modified websocket-sharp for now |
I would also like to see this implemented. Our setup is running a websoket server with lots of embedded devices as clients. We run a standardized protocol based on websocket and therefore cannot implement our own ping mechanism or use signalR. It's important since if you run linux for your servers it will take about 15 min for it to detect a disconnect as the default TCP RTO setting in linux. |
Triage: given that the customer ask is high, and it should be relatively easy to implement, optimistically putting it to 9.0. |
We'll likely have to manually test with the browser. These kinds of tests aren't great to add to automation since they by design would need to last 20+ seconds due to no APIs in the browser that let us influence the keep alive timeout. Assuming we add good enough unit tests the one-off manual test should be fine. |
It's unfortunate that we can't implement this effectively because ManagedWebSocket.SendFrameAsync is private, preventing us from sending the PING frames ourselves in the application. Additionally, ClientWebSocket.ReceiveAsync doesn't allow the reception of PONG frames in the application. |
@mar1u50 is there a reason you would still need this access on the application level, after the proposed API gets implemented? |
I think the only missing part (or maybe I missed this from the API change description) is a sequence number sent in the PING to be able to pair it with the PONG. |
The proposed |
In systems where there may be other machines between the client and server, such as a CloudFlare node, it is possible that if the client-CloudFlare connection dies, the CloudFlare-server connection will not also die. Likewise, it is possible for the CloudFlare-server connection to die while the client-CloudFlare connection remains active. To prevent either end from remaining in a zombie state, I was thinking that the server could also check if pings are still coming and force a close if they are not. For this, the server should be able to detect if the ping is the one requested by the client (through its payload). It is true that this could be implemented in another way too: the client initiates pings and waits for pongs, and at the same time, the server also initiates pings and waits for pongs. |
Yes, I believe this would be the expected approach by RFC. If the server is on Kestrel, the timeout APIs will be introduced as a follow-up to this one (see #48729 (comment)), within the same release. @BrennanConroy is there a tracking issue for this? |
I've been using this as the tracking issue, until there is something to do on the Kestrel side. |
Looks good as proposed. namespace System.Net.WebSockets;
public partial class WebSocketCreationOptions
{
public TimeSpan KeepAliveTimeout { get; set; } = Timeout.InfiniteTimeSpan;
}
public partial class ClientWebSocketOptions
{
[UnsupportedOSPlatform("browser")]
public TimeSpan KeepAliveTimeout { get; set; } = Timeout.InfiniteTimeSpan;
} |
The proposal is exactly what's needed for my company's servers. With .NET 5-8, the replacement of the unidirectional Ping with a unidirectional Pong broke tens of thousands of clients that didn't have a 3rd party implementation that could understand Pong frames as a heartbeat. The result was chaos, with disconnections about every minute or so. |
Will this be ready for dotnet 9? |
@Madguy93 it is currently planned for .NET 9, yes. |
I wonder... Maybe you could also add two new DateTimes, so we can better get a view of what is happening internally in the WS?
These would be updated whenever data (including ping\pong) is transmitted\recieved. |
@CarnaViire any estimate on which preview it would make? (I won't hold you to it) Right now our options seem to be as follows and would like to make an informed decision.
|
Most possibly Preview 7, scheduled for the middle of August @chrsin |
@osexpert these timestamps, or rather, the difference between BUT if you already need go and check the It seems redundant to check the timestamps, "do the math" (and manual comparison with a timeout) to deduce the state, when the state would be already computed and available in |
Update: this didn't make it in time for Preview 7; it is now targeting RC1 instead (September) |
Edited by @CarnaViire
Justification
This API allows the users to identify and close idle WebSocket connections without having to rely on the TCP KeepAlive timeout, which can be way too big (e.g. ~15 min on Linux), and can be impossible or very hard to configure.
// see the original issue in the end of the post
API Proposal
Usage
For ClientWebSocket:
For ManagedWebSocket:
Notes on behavior
Timeout.InfiniteTimeSpan
orTimeSpan.Zero
would disable the timeout logic and fall back to the existing "unsolicited"Pong
keep-alive (per understanding in #35473 (comment) we cannot sendPing
s without waiting for the reply)Any other positive timeout value would turn on
Ping-Pong
logic; we will be sendingPing
s instead ofPong
s and track whether we've receivedPong
s in time; if not, the connection is treated as stale and aborted. The behavior and implementation should be similar to HTTP/2KeepAlivePingTimeout
.Alternative Naming
We could name the new property KeepAlivePingTimeout. This will "break" it's similarity to KeepAliveInterval, but it might help highlight that
Ping
s, notPong
s, are being sent (if we care enough about this distinction).Existing API shape for other keep-alives
For comparison:
Original issue by @bjornen77
I think that adding a WebSocketOptions.Timeout property would be a good thing.
Today, the ClientWebSocket is closed when the TCP Retransmission times out(if the connection to the server drops for some reason). This timeout seems to be different on Windows and Linux. By default, on Windows it takes about 21 seconds and on Linux it takes about 15 minutes before the WebSocket is closed. It would be good if we could have a separate Timeout setting, so that we get the same behavior regardless of the platform. And also that this could be configured by the application without having to modify the TCP Timeout settings in the OS... The websocket should still be disconnected if the OS TCP Timeout is reached.
The ClientWebSocket today only sends "Pong" to the server, and therefore it does not wait for a "Pong" response to arrive. I think that the ClientWebSocket shall send a "Ping" instead, and then verify that a "Pong" is received within the Timeout set. If not received, the connection shall be closed. Or at least it should expose the "received "Pong" message in the WebSocketReceiveResult.MessageType. It would be good if an received "Ping" message from a server is also exposed. Then the application could use these Ping/Pong message types to implement its own timeout logic.
The text was updated successfully, but these errors were encountered: