MsQuic is used as the basis for several different protocols (HTTP, SMB, etc.), but they all have several things in common when it comes time to deploy them. This document outlines the various things that must be taken into account whenever deploying an MsQuic based solution.
Generally, for any existing TCP based deployments that are adding QUIC support, there are a number of things to be considered. Many things are different between a TCP based solution and a QUIC based one, including breaking some pretty "core" assumptions made for TCP:
- QUIC uses UDP instead of TCP.
- Any firewalls or other network devices must take this into account, and make sure this traffic is allowed.
- QUIC traffic is designed to be generally indistinguishable from other UDP traffic.
- Network devices must not assume UDP traffic on any port is QUIC unless explicitly configured.
- Current QUIC based protocols primarily use port 443 on the server, but not necessarily exclusively.
- HTTP and SMB use this port, but other protocols (e.g. DNS over QUIC) likely will not.
- QUIC is versioned and extensible, and thus is expected to be very dynamic on the network.
- Network devices must not assume anything about the structure of a QUIC packet beyond what is stated in the Invariants draft.
- QUIC is completely encrypted end to end.
- Most information that might have been viewable on a TCP connection is now only visible to the endpoints.
- A single UDP flow or tuple (address + port) does not necessarily map to a single connection.
- A single QUIC connection may span multiple flows.
- Multiple QUIC connections may share a single flow.
- NAT bindings for UDP flows on the internet generally timeout much quicker than TCP; resulting in flow changes much more often.
- QUIC, as a protocol, is able to survive these changes, unlike TCP.
For more details, please see the Manageability draft.
MsQuic supports a number of configuration knobs (or settings). These settings can either be set dynamically (via the QUIC_SETTINGS structure) or via persistent storage (e.g. registry on Windows).
Important - Generally MsQuic already choses the best / most correct default values for all settings. Settings should only be changed after due diligence and A/B testing is performed.
| Setting | Type | Name | Description |
|---|---|---|---|
| Max Partition Count | uint16_t | MaxPartitionCount | The maximum processor count used for partitioning work in MsQuic. Restart is required. |
| Max Bytes per Key | uint64_t | MaxBytesPerKey | |
| Handshake Idle Timeout | uint64_t | HandshakeIdleTimeoutMs | |
| Idle Timeout | uint64_t | IdleTimeoutMs | |
| Max TLS Send Buffer (Client) | uint32_t | TlsClientMaxSendBuffer | |
| Max TLS Send Buffer (Server) | uint32_t | TlsServerMaxSendBuffer | |
| Stream Receive Window | uint32_t | StreamRecvWindowDefault | |
| Stream Receive Buffer | uint32_t | StreamRecvBufferDefault | |
| Flow Control Window | uint32_t | ConnFlowControlWindow | |
| Max Worker Queue Delay | uint32_t | MaxWorkerQueueDelayMs | The maximum queue delay (in ms) allowed for a worker thread |
| Max Stateless Operations | uint32_t | MaxStatelessOperations | The maximum number of stateless operations that may be queued at any one time |
| Initial Window | uint32_t | InitialWindowPackets | The size (in packets) of the initial congestion window for a connection |
| Send Idle Timeout | uint32_t | SendIdleTimeoutMs | |
| Initial RTT | uint32_t | InitialRttMs | |
| Max ACK Delay | uint32_t | MaxAckDelayMs | |
| Disconnect Timeout | uint32_t | DisconnectTimeoutMs | |
| Keep Alive Interval | uint32_t | KeepAliveIntervalMs | |
| Peer Stream Count (Bidirectional) | uint16_t | PeerBidiStreamCount | |
| Peer Stream Count (Unidirectional) | uint16_t | PeerUnidiStreamCount | |
| Retry Memory Limit | uint16_t | RetryMemoryFraction | The percentage of available memory usable for handshake connections before stateless retry is used |
| Load Balancing Mode | uint16_t | LoadBalancingMode | |
| Max Operations per Drain | uint8_t | MaxOperationsPerDrain | The maximum number of operations to drain per connection quantum |
| Send Buffering | uint8_t | SendBufferingEnabled | |
| Send Pacing | uint8_t | PacingEnabled | |
| Client Migration Support | uint8_t | MigrationEnabled | |
| Datagram Receive Support | uint8_t | DatagramReceiveEnabled | |
| Server Resumption Level | uint8_t | ServerResumptionLevel |
TODO - Finish table above
On Windows, these settings can set via the registry and will persist across reboots and build upgrades. For most settings, a reboot is not required for them to immediately take effect. Also note that updated settings will only affect new connections (not existing ones).
The main registry path for the keys is:
HKLM:\System\CurrentControlSet\Services\MsQuic\Parameters
The settings can also be set per "app-name" (as indicated in RegistrationOpen):
HKLM:\System\CurrentControlSet\Services\MsQuic\Parameters\app-name
The DWORD type should be used for all 32-bit or less types. For 64-bit types, DWORD or QWORD may be used. If invalid types or values are used, they will be ignored and the built-in default will be used instead.
For example, to set the Initial Window Size setting to 20 packets, you may do the following:
reg.exe add "HKLM\System\CurrentControlSet\Services\MsQuic\Parameters" /v InitialWindowPackets /t REG_DWORD /d 20
By default, the new cipher suite TLS_CHACHA20_POLY1305_SHA256 is disabled. It can be enabled via the following command:
Enable-TlsCipherSuite -Name TLS_CHACHA20_POLY1305_SHA256In order to configure the Windows firewall to allow inbound QUIC traffic efficiently, use a command such as the one below. Generally, the firewall rule should be applied for all scenarios, unless a layer below you (e.g. IIS) is already doing it on your behalf.
New-NetFirewallRule -DisplayName "Allow QUIC" -Direction Inbound -Protocol UDP -LocalPort 443 -Action Allow -LocalOnlyMapping $trueNote the use of the -LocalOnlyMapping $true argument. This is a performance optimizing feature that should be used for UDP based protocols (like QUIC). See MSDN for additional details.
MsQuic currently supports a load balancing mode where the server encodes the local IPv4 address or IPv6 suffix into bytes 1 through 4 of the connection IDs it creates. You can read more details about the general encoding in Load Balancers draft.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| First octet | Server ID (X=8..152) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Any (0..152-X) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Server ID is 4 bytes long, as encodes the complete IPv4 address OR the last 4 bytes of the IPv6 address.
This encoding is not enabled by default. To configure it, set the LoadBalancingMode setting to 1. Note - The server must be restarted for this setting to take effect.
To use this load balancing model, the load balancer must support the model described above and be explicitly configured to enable it for your endpoint.
Client migration is a key feature in the QUIC protocol that allows for the connection to survive changes in the client's IP address or UDP port. MsQuic generally supports this but it requires QUIC load balancing support (when using a load balancer). QUIC encodes a connection identifier (connection ID or CID) in every packet it sends. This CID allows a server to encode routing information that a coordinating load balancer can use to route the packet, instead of using the IP tuple as most existing load balancers currently use to route UDP traffic.
If your deployment does not have QUIC load balancing support then you will not be able to make sure of the client migration feature described above to survive any NAT rebindings that change the client's IP tuple (from the server's perspective). This can be especially painful for any services migrating from a TCP based solution to QUIC, since most middleboxes on the internet have a much smaller timeout period for UDP (20 to 30 seconds) compared to TCP. This means any QUIC connection that goes idle for greater than ~20 seconds runs the risk of getting rebound by the NAT the next time the client sends a packet, resulting in a tuple change, and then likely resulting in the packet getting routed to the incorrect load balanced server.
The mitigation to this problem is to enable QUIC keep alives. They can be enabled on either the client or server side, but only need to be enabled on one side. They can be enabled either dynamically in the code or globally via the settings. To enable keep alives via the settings, set the KeepAliveIntervalMs setting to a reasonable value, such as 20000 (20 seconds).
MsQuic has a few built-in denial of service mitigations (server side).
MsQuic tracks the number of outstanding connections currently in the handshake state. When that reaches a certain threshold, MsQuic will start forcing clients to retry before the connection will be accepted. This entails the following:
- The server sends back a "Stateless Retry" packet with an encrypted token to the client.
- The server drops the incoming packet and doesn't save any state.
- The client must then reply back with its initial packet, this time including the encrypted token.
- The server validates the token, and only if successful, accept the connection.
This protects the server from naive attackers trying to flood the server with new connection attempts; especially in scenarios where the client is spoofing its source IP address in an attempt to avoid attribution.
The threshold mentioned above is currently tracked as a percentage of total avaialble (nonpaged pool) memory. This percentage of avaiable memory can be configured via the RetryMemoryFraction setting.
MsQuic uses worker threads internally to execute the QUIC protocol logic. For each worker thread, MsQuic tracks the average queue delay for any work done on one of these threads. This queue delay is simply the time from when the work is added to the queue to when the work is removed from the queue. If this delay hits a certain threshold, then existing connections can start to suffer (i.e. spurious packet loss, decreased throughput, or even connection failures). In order to prevent this, new connections are rejected with the SERVER_BUSY error, when this threshold is reached.
The queue delay threshold can be configured via the MaxWorkerQueueDelayMs setting.
For details on how to diagnose any issues with your deployment at the MsQuic layer see Diagnostics.