Enlarge TCP recv_buf to improve throughput #6690
Conversation
wfchandler
changed the title
| info!( | ||
| log, "Setting TCP recv_buf size to 1 MB"; | ||
| ); | ||
| Ipadm::set_tcp_recv_buf().context("failed to set TCP recv_buf")?; | ||
|
|
||
| info!( | ||
| log, "Setting TCP congestion control algorithm to cubic"; | ||
| ); | ||
| Ipadm::set_tcp_congestion_control() | ||
| .context("failed to set TCP congestion_control")?; | ||
|
|
There was a problem hiding this comment.
Why is this being set to all zones and not just nexus?
There was a problem hiding this comment.
Because it's easier and Josh said it was "probably fine." However, it would probably be pretty easy to make it part of a command or flag that is only used by the Nexus zone setter upper.
There was a problem hiding this comment.
I think it's basically fine everywhere, FWIW. We should test to make sure that everything still works before and after obviously but I don't expect there will be any issues
There was a problem hiding this comment.
Sadly, it wouldn't be as straightforward as it seems to implement only to Nexus. Currently there's no nexus-specific SMF service that's required before the nexus service starts. We'd have to create a new service and do something like #5440.
I'm a bit wary of doing a blanket change to system defaults of all zones as we don't really have a comprehensive integration testing suite for this.
If everyone is OK with it, perhaps we can go the safer way? Not sure how urgent this is.
There was a problem hiding this comment.
I don't think you would want to create a new service. Rather, I would have the existing service accept an SMF property for configuration, which would be absent by default, and which you could override in the SMF profile we pass to the Nexus zone specifically
There was a problem hiding this comment.
But also I think it's low risk and I would probably just do it
There was a problem hiding this comment.
I trust the duke of illumos on this. From what I read elsewhere, it sounds like the default value is probably low for modern hardware anyway!
There was a problem hiding this comment.
This service also runs in the switch zone, are we sure this wouldn't have unintended consequences?
There was a problem hiding this comment.
I'll leave it up to all of you! If you're all ok with it then I'm ok with it. Can we just add a comment on the code with an explanation of why we're setting those new values?
There was a problem hiding this comment.
I've added a comment explaining why we're increasing recv_buf.
Fix comment type on `set_interface_mtu`.
Image uploads performed via the web console are 3-4x slower than uploads performed via the Oxide CLI[0]. We found that the CLI creates 8 separate TCP connections to upload the image chunks, while the console uses HTTP/2 to multiplex a single TCP connection six ways. The default TCP `recv_buf` size on Helios is 128 KB, which limits window size and therefore the number of packets that can be sent in parallel. By increasing this value to 1 MB, we can increase single-connection throughput by ~3x, bringing console performance to rough parity with the CLI. This does increase the amount of memory a potential DoS attack could consume, but 1 MB is still quite small relative to the total resources available on a compute sled. While we're at it, also update the TCP congestion control algorithm to `cubic` from its default value of `sunreno`, which may also help improve throughput. Closes #6601 [0] oxidecomputer/console#2096 Assisted-by: David Crespo <[email protected]>
wfchandler
force-pushed
the
wc/set-recv-buf
branch
from
1642d48 to
552d1c5
Compare
|
Rebasing to resolve CI failures. |
Image uploads performed via the [web console are 3-4x slower than uploads performed via the Oxide CLI](oxidecomputer/console#2096). We found that the CLI creates 8 separate TCP connections to upload the image chunks, while the console uses HTTP/2 to multiplex a single TCP connection six ways. The default TCP `recv_buf` size on Helios is 128 KB, which limits window size and therefore the number of packets that can be sent in parallel. By increasing this value to 1 MB, we can increase single-connection throughput by ~3x, bringing console performance to rough parity with the CLI. This does increase the amount of memory a potential DoS attack could consume, but 1 MB is still quite small relative to the total resources available on a compute sled. While we're at it, also update the TCP congestion control algorithm to `cubic` from its default value of `sunreno`, which may also help improve throughput. Closes #6601
Image uploads performed via the web console are 3-4x slower than uploads performed via the Oxide CLI. We found that the CLI creates 8 separate TCP connections to upload the image chunks, while the console uses HTTP/2 to multiplex a single TCP connection six ways. The default TCP
recv_bufsize on Helios is 128 KB, which limits window size and therefore the number of packets that can be sent in parallel. By increasing this value to 1 MB, we can increase single-connectionthroughput by ~3x, bringing console performance to rough parity with the CLI.
This does increase the amount of memory a potential DoS attack could consume, but 1 MB is still quite small relative to the total resources available on a compute sled.
While we're at it, also update the TCP congestion control algorithm to
cubicfrom its default value ofsunreno, which may also help improve throughput.Closes #6601