TCP_INFO support

[dtaht]

TCP_INFO (struct tcp_info) contains metrics for how the tcp layer is performing, and lends insight into what is otherwise a black box for most folk. Notable are base rtt, rtt, loss, retries, ecn_seen from my perceptual worrying-about-bufferbloat perspective...

Windows impl:

https://learn.microsoft.com/en-us/windows/win32/api/mstcpip/ns-mstcpip-tcp_info_v1

Apache traffic server's implementation: https://docs.trafficserver.apache.org/en/9.0.x/admin-guide/plugins/tcpinfo.en.html

How google measurement labs uses it: https://www.measurementlab.net/tests/tcp-info/ (out of band in this case)

It would be really nice to be able to monitor this set of metrics for more applications.

[Thomasdezeeuw]

This would be a nice addition. I think we should use same strategy we do for Mio's event: https://github.com/tokio-rs/mio/blob/1f8cd972f7a575e097c29da1e5aa2d4b7f399d8d/src/event/event.rs. That is define a TcpInfo struct that wraps the tcp_info OS specific struct and define getters on TcpInfo to get the OS specific data.

[dtaht]

Thank you for thinking about my request. Merely being able to log TCP_INFO before closing a TCP stream would be an awesome improvement on what happens today. Being able to poll it frequently, would allow for a rust application to better rate limit itself. I look at these frameworks, and my mind crashes, but it is indeed, just one system call, or auxiliary data that can be pulled from a tcp request.

The way that google measurement labs does it though, is different, in that they just log the whole binary tcp_info struct as one record, and figure out how to parse it later - from the kernel, the order of the fields is garunteed to be stable, but every new linux kernel release has added more data to that struct. So despite that innate desire to provide accessor functions you appear to have, just getting that whole structure from point a to point b, somehow, is helpful.

Somewhat related, recently a rather large microservices user, realized that 80% of the data they were sending to the web server never made it to the users, they applied the TCP_NOTSENT_LOWAT option (I think, 32k) to a huge improvement in overall service time. This document might provide some insight as to who that was (it led to the apache traffic server patch)

https://www.bitag.org/documents/BITAG_latency_explained.pdf

[Thomasdezeeuw]

Thank you for thinking about my request. Merely being able to log TCP_INFO before closing a TCP stream would be an awesome improvement on what happens today. Being able to poll it frequently, would allow for a rust application to better rate limit itself. I look at these frameworks, and my mind crashes, but it is indeed, just one system call, or auxiliary data that can be pulled from a tcp request.

I'm not sure if you're implying that socket2 should log anything, but we're not going to do that. We can add a nice getter and setter function to the Socket type though.

[BobAnkh]

I am interested in implementing this interface, but I have a question for which I haven't come up with a good solution: the struct tcp_info will have different implementations for different kernel versions, and we need to generate bindings for the header file /usr/include/linux/tcp.h accordingly. So how can we define a TcpInfo struct that wraps the tcp_info OS specific struct and define getters on TcpInfo to get the OS specific data? (how do we handle non-presented fields in tcp_info in TcpInfo when in older versions?)

[Thomasdezeeuw]

I am interested in implementing this interface, but I have a question for which I haven't come up with a good solution: the struct tcp_info will have different implementations for different kernel versions, and we need to generate bindings for the header file /usr/include/linux/tcp.h accordingly.

I don't want to maintain a struct definition, this will have to live in the libc crate.

So how can we define a TcpInfo struct that wraps the tcp_info OS specific struct and define getters on TcpInfo to get the OS specific data? (how do we handle non-presented fields in tcp_info in TcpInfo when in older versions?)

For the type in socket2 we can define something like struct TcpInfo { inner: sys::tcp_info }, where sys::tcp_info is libc::tcp_info for Unix and something else for Windows (maybe windows_sys::Win32::Networking::WinSock::TCP_INFO_v1? I don't know). Similar to TcpKeepalive we'll have the appropriate getters and setters depending on the OS, see:

socket2/src/lib.rs

Lines 401 to 552 in b39fdc6

	pub struct TcpKeepalive {
	#[cfg_attr(target_os = "openbsd", allow(dead_code))]
	time: Option<Duration>,
	#[cfg(not(any(
	target_os = "openbsd",
	target_os = "redox",
	target_os = "solaris",
	target_os = "nto",
	)))]
	interval: Option<Duration>,
	#[cfg(not(any(
	target_os = "openbsd",
	target_os = "redox",
	target_os = "solaris",
	target_os = "windows",
	target_os = "nto",
	)))]
	retries: Option<u32>,
	}
	impl TcpKeepalive {
	/// Returns a new, empty set of TCP keepalive parameters.
	pub const fn new() -> TcpKeepalive {
	TcpKeepalive {
	time: None,
	#[cfg(not(any(
	target_os = "openbsd",
	target_os = "redox",
	target_os = "solaris",
	target_os = "nto",
	)))]
	interval: None,
	#[cfg(not(any(
	target_os = "openbsd",
	target_os = "redox",
	target_os = "solaris",
	target_os = "windows",
	target_os = "nto",
	)))]
	retries: None,
	}
	}
	/// Set the amount of time after which TCP keepalive probes will be sent on
	/// idle connections.
	///
	/// This will set `TCP_KEEPALIVE` on macOS and iOS, and
	/// `TCP_KEEPIDLE` on all other Unix operating systems, except
	/// OpenBSD and Haiku which don't support any way to set this
	/// option. On Windows, this sets the value of the `tcp_keepalive`
	/// struct's `keepalivetime` field.
	///
	/// Some platforms specify this value in seconds, so sub-second
	/// specifications may be omitted.
	pub const fn with_time(self, time: Duration) -> Self {
	Self {
	time: Some(time),
	..self
	}
	}
	/// Set the value of the `TCP_KEEPINTVL` option. On Windows, this sets the
	/// value of the `tcp_keepalive` struct's `keepaliveinterval` field.
	///
	/// Sets the time interval between TCP keepalive probes.
	///
	/// Some platforms specify this value in seconds, so sub-second
	/// specifications may be omitted.
	#[cfg(any(
	target_os = "android",
	target_os = "dragonfly",
	target_os = "freebsd",
	target_os = "fuchsia",
	target_os = "illumos",
	target_os = "ios",
	target_os = "linux",
	target_os = "macos",
	target_os = "netbsd",
	target_os = "tvos",
	target_os = "watchos",
	target_os = "windows",
	))]
	#[cfg_attr(
	docsrs,
	doc(cfg(any(
	target_os = "android",
	target_os = "dragonfly",
	target_os = "freebsd",
	target_os = "fuchsia",
	target_os = "illumos",
	target_os = "ios",
	target_os = "linux",
	target_os = "macos",
	target_os = "netbsd",
	target_os = "tvos",
	target_os = "watchos",
	target_os = "windows",
	)))
	)]
	pub const fn with_interval(self, interval: Duration) -> Self {
	Self {
	interval: Some(interval),
	..self
	}
	}
	/// Set the value of the `TCP_KEEPCNT` option.
	///
	/// Set the maximum number of TCP keepalive probes that will be sent before
	/// dropping a connection, if TCP keepalive is enabled on this socket.
	#[cfg(all(
	feature = "all",
	any(
	target_os = "android",
	target_os = "dragonfly",
	target_os = "freebsd",
	target_os = "fuchsia",
	target_os = "illumos",
	target_os = "ios",
	target_os = "linux",
	target_os = "macos",
	target_os = "netbsd",
	target_os = "tvos",
	target_os = "watchos",
	)
	))]
	#[cfg_attr(
	docsrs,
	doc(cfg(all(
	feature = "all",
	any(
	target_os = "android",
	target_os = "dragonfly",
	target_os = "freebsd",
	target_os = "fuchsia",
	target_os = "illumos",
	target_os = "ios",
	target_os = "linux",
	target_os = "macos",
	target_os = "netbsd",
	target_os = "tvos",
	target_os = "watchos",
	)
	)))
	)]
	pub const fn with_retries(self, retries: u32) -> Self {
	Self {
	retries: Some(retries),
	..self
	}
	}
	}

[BobAnkh]

I don't want to maintain a struct definition, this will have to live in the libc crate.

Ok, then I think this feature might be blocked since libc haven't implemented this struct libc/#1609 for linux. I will see if there is an opportunity to implement tcp_info for libc in the future.

[dtaht]

the windows v2 tcp_info header is better, as it has more of the variables I (at least) care about. Thx, all for looking into it.