Adjust default keepalive interval to 325s

The default interval for keepalives today is (mostly) 300s. This value
was chosen to be short enough to keep connections alive even if the
network drops idle connections after 600s, which is the most
unreasonable idle-connection timeout we have seen in practice.

This nice round number happens to coincide with the Linux kernel default
for the apparently-unrelated `nf_conntrack_tcp_timeout_unacknowledged`
timeout. We have observed that in rare cases if the keepalive timer
expires within a few milliseconds of the conntrack unacknowledged timer
then the connection may erroneously be dropped. Although the ultimate
fix for this lies elsewhere, to reduce the probability of spurious
connection drops in affected environments this commit extends the
default keepalive interval to a more unusual 325s, avoiding the
problematic coincidence.

Relates elastic#59278

docs/reference/modules/http.asciidoc

@@ -153,18 +153,19 @@ determines whether it sends TCP keepalive probes.
 Defaults to `network.tcp.keep_alive`.
 
 `http.tcp.keep_idle`::
-(<<static-cluster-setting,Static>>) Configures the `TCP_KEEPIDLE` option for this socket, which
-determines the time in seconds that a connection must be idle before
-starting to send TCP keepalive probes. Defaults to `network.tcp.keep_idle`, which
-uses the system default. This value cannot exceed `300` seconds. Only applicable on
-Linux and macOS, and requires Java 11 or newer.
+(<<static-cluster-setting,Static>>) Configures the `TCP_KEEPIDLE` option for
+this socket, which determines the time in seconds that a connection must be
+idle before starting to send TCP keepalive probes. Defaults to
+`network.tcp.keep_idle`, which uses the system default. This value cannot
+exceed `325` seconds. Only applicable on Linux and macOS, and requires Java 11
+or newer.
 
 `http.tcp.keep_interval`::
-(<<static-cluster-setting,Static>>) Configures the `TCP_KEEPINTVL` option for this socket,
-which determines the time in seconds between sending TCP keepalive probes.
-Defaults to `network.tcp.keep_interval`, which uses the system default.
-This value cannot exceed `300` seconds. Only applicable on Linux and macOS, and requires
-Java 11 or newer.
+(<<static-cluster-setting,Static>>) Configures the `TCP_KEEPINTVL` option for
+this socket, which determines the time in seconds between sending TCP keepalive
+probes. Defaults to `network.tcp.keep_interval`, which uses the system default.
+This value cannot exceed `325` seconds. Only applicable on Linux and macOS, and
+requires Java 11 or newer.
 
 `http.tcp.keep_count`::
 (<<static-cluster-setting,Static>>) Configures the `TCP_KEEPCNT` option for this socket, which

docs/reference/modules/network.asciidoc

@@ -205,18 +205,18 @@ determines whether it sends TCP keepalive probes.
 
 `network.tcp.keep_idle`::
 (<<static-cluster-setting,Static>>)
-Configures the `TCP_KEEPIDLE` option for this socket, which
-determines the time in seconds that a connection must be idle before
-starting to send TCP keepalive probes. Defaults to `-1`, which uses
-the system default. This value cannot exceed `300` seconds. Only applicable on Linux and macOS,
-and requires Java 11 or newer.
+Configures the `TCP_KEEPIDLE` option for this socket, which determines the time
+in seconds that a connection must be idle before starting to send TCP keepalive
+probes. Defaults to `-1`, which uses the system default. This value cannot
+exceed `325` seconds. Only applicable on Linux and macOS, and requires Java 11
+or newer.
 
 `network.tcp.keep_interval`::
 (<<static-cluster-setting,Static>>)
-Configures the `TCP_KEEPINTVL` option for this socket,
-which determines the time in seconds between sending TCP keepalive probes.
-Defaults to `-1`, which uses the system default. This value cannot exceed `300` seconds.
-Only applicable on Linux and macOS, and requires Java 11 or newer.
+Configures the `TCP_KEEPINTVL` option for this socket, which determines the
+time in seconds between sending TCP keepalive probes. Defaults to `-1`, which
+uses the system default. This value cannot exceed `325` seconds. Only
+applicable on Linux and macOS, and requires Java 11 or newer.
 
 `network.tcp.keep_count`::
 (<<static-cluster-setting,Static>>)

docs/reference/modules/transport.asciidoc

@@ -74,22 +74,21 @@ Defaults to `network.tcp.keep_alive`.
 
 `transport.tcp.keep_idle`::
 (<<static-cluster-setting,Static>>)
-Configures the `TCP_KEEPIDLE` option for this socket, which
-determines the time in seconds that a connection must be idle before
-starting to send TCP keepalive probes. Defaults to `network.tcp.keep_idle` if set,
-or the system default otherwise.
-This value cannot exceed `300` seconds. In cases where the system default
-is higher than `300`, the value is automatically lowered to `300`. Only applicable on
-Linux and macOS, and requires Java 11 or newer.
+Configures the `TCP_KEEPIDLE` option for this socket, which determines the time
+in seconds that a connection must be idle before starting to send TCP keepalive
+probes. Defaults to `network.tcp.keep_idle` if set, or the system default
+otherwise. This value cannot exceed `325` seconds. In cases where the system
+default is higher than `325`, the value is automatically lowered to `325`. Only
+applicable on Linux and macOS, and requires Java 11 or newer.
 
 `transport.tcp.keep_interval`::
 (<<static-cluster-setting,Static>>)
-Configures the `TCP_KEEPINTVL` option for this socket,
-which determines the time in seconds between sending TCP keepalive probes.
-Defaults to `network.tcp.keep_interval` if set, or the system default otherwise.
-This value cannot exceed `300` seconds. In cases where the system default is higher than `300`,
-the value is automatically lowered to `300`. Only applicable on Linux and macOS,
-and requires Java 11 or newer.
+Configures the `TCP_KEEPINTVL` option for this socket, which determines the
+time in seconds between sending TCP keepalive probes. Defaults to
+`network.tcp.keep_interval` if set, or the system default otherwise. This value
+cannot exceed `325` seconds. In cases where the system default is higher than
+`325`, the value is automatically lowered to `325`. Only applicable on Linux
+and macOS, and requires Java 11 or newer.
 
 `transport.tcp.keep_count`::
 (<<static-cluster-setting,Static>>)

libs/core/src/main/java/org/elasticsearch/core/internal/net/NetUtils.java

@@ -20,6 +20,8 @@
  */
 public class NetUtils {
 
+    public static int MAX_REASONABLE_KEEPALIVE_INTERVAL = 325;
+
     /**
      * Returns the extended TCP_KEEPIDLE socket option, if available on this JDK
      */
@@ -69,7 +71,7 @@ public static void tryEnsureReasonableKeepAliveConfig(NetworkChannel socketChann
                     for (SocketOption<Integer> option : Arrays.asList(
                         NetUtils.getTcpKeepIdleSocketOptionOrNull(),
                         NetUtils.getTcpKeepIntervalSocketOptionOrNull())) {
-                        setMinValueForSocketOption(socketChannel, option, 300);
+                        setMinValueForSocketOption(socketChannel, option, MAX_REASONABLE_KEEPALIVE_INTERVAL);
                     }
                 }
             }

modules/transport-netty4/src/test/java/org/elasticsearch/transport/netty4/SimpleNetty4TransportTests.java

@@ -41,6 +41,7 @@
 
 import static java.util.Collections.emptyMap;
 import static java.util.Collections.emptySet;
+import static org.elasticsearch.core.internal.net.NetUtils.MAX_REASONABLE_KEEPALIVE_INTERVAL;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.hasItem;
 import static org.hamcrest.Matchers.instanceOf;
@@ -117,10 +118,10 @@ private void checkDefaultKeepAliveOptions(TcpChannel channel) throws IOException
         assertThat(socketChannel.supportedOptions(), hasItem(NetUtils.getTcpKeepIdleSocketOptionOrNull()));
         Integer keepIdle = socketChannel.getOption(NetUtils.getTcpKeepIdleSocketOptionOrNull());
         assertNotNull(keepIdle);
-        assertThat(keepIdle, lessThanOrEqualTo(500));
+        assertThat(keepIdle, lessThanOrEqualTo(MAX_REASONABLE_KEEPALIVE_INTERVAL));
         assertThat(socketChannel.supportedOptions(), hasItem(NetUtils.getTcpKeepIntervalSocketOptionOrNull()));
         Integer keepInterval = socketChannel.getOption(NetUtils.getTcpKeepIntervalSocketOptionOrNull());
         assertNotNull(keepInterval);
-        assertThat(keepInterval, lessThanOrEqualTo(500));
+        assertThat(keepInterval, lessThanOrEqualTo(MAX_REASONABLE_KEEPALIVE_INTERVAL));
     }
 }

server/src/main/java/org/elasticsearch/common/network/NetworkService.java

@@ -22,6 +22,8 @@
 import java.util.Objects;
 import java.util.function.Function;
 
+import static org.elasticsearch.core.internal.net.NetUtils.MAX_REASONABLE_KEEPALIVE_INTERVAL;
+
 public final class NetworkService {
 
     /** By default, we bind to loopback interfaces */
@@ -40,9 +42,9 @@ public final class NetworkService {
     public static final Setting<Boolean> TCP_KEEP_ALIVE =
         Setting.boolSetting("network.tcp.keep_alive", true, Property.NodeScope);
     public static final Setting<Integer> TCP_KEEP_IDLE =
-        Setting.intSetting("network.tcp.keep_idle", -1, -1, 300, Property.NodeScope);
+        Setting.intSetting("network.tcp.keep_idle", -1, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL, Property.NodeScope);
     public static final Setting<Integer> TCP_KEEP_INTERVAL =
-        Setting.intSetting("network.tcp.keep_interval", -1, -1, 300, Property.NodeScope);
+        Setting.intSetting("network.tcp.keep_interval", -1, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL, Property.NodeScope);
     public static final Setting<Integer> TCP_KEEP_COUNT =
         Setting.intSetting("network.tcp.keep_count", -1, -1, Property.NodeScope);
     public static final Setting<Boolean> TCP_REUSE_ADDRESS =

server/src/main/java/org/elasticsearch/http/HttpTransportSettings.java

@@ -24,6 +24,7 @@
 import static org.elasticsearch.common.settings.Setting.boolSetting;
 import static org.elasticsearch.common.settings.Setting.intSetting;
 import static org.elasticsearch.common.settings.Setting.listSetting;
+import static org.elasticsearch.core.internal.net.NetUtils.MAX_REASONABLE_KEEPALIVE_INTERVAL;
 
 public final class HttpTransportSettings {
 
@@ -93,9 +94,11 @@ public final class HttpTransportSettings {
     public static final Setting<Boolean> SETTING_HTTP_TCP_KEEP_ALIVE =
         boolSetting("http.tcp.keep_alive", NetworkService.TCP_KEEP_ALIVE, Setting.Property.NodeScope);
     public static final Setting<Integer> SETTING_HTTP_TCP_KEEP_IDLE =
-        intSetting("http.tcp.keep_idle", NetworkService.TCP_KEEP_IDLE, -1, 300, Setting.Property.NodeScope);
+        intSetting("http.tcp.keep_idle", NetworkService.TCP_KEEP_IDLE, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL,
+                Setting.Property.NodeScope);
     public static final Setting<Integer> SETTING_HTTP_TCP_KEEP_INTERVAL =
-        intSetting("http.tcp.keep_interval", NetworkService.TCP_KEEP_INTERVAL, -1, 300, Setting.Property.NodeScope);
+        intSetting("http.tcp.keep_interval", NetworkService.TCP_KEEP_INTERVAL, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL,
+                Setting.Property.NodeScope);
     public static final Setting<Integer> SETTING_HTTP_TCP_KEEP_COUNT =
         intSetting("http.tcp.keep_count", NetworkService.TCP_KEEP_COUNT, -1, Setting.Property.NodeScope);
     public static final Setting<Boolean> SETTING_HTTP_TCP_REUSE_ADDRESS =

server/src/main/java/org/elasticsearch/transport/TransportSettings.java

@@ -24,6 +24,7 @@
 import static org.elasticsearch.common.settings.Setting.intSetting;
 import static org.elasticsearch.common.settings.Setting.listSetting;
 import static org.elasticsearch.common.settings.Setting.timeSetting;
+import static org.elasticsearch.core.internal.net.NetUtils.MAX_REASONABLE_KEEPALIVE_INTERVAL;
 
 public final class TransportSettings {
 
@@ -70,15 +71,17 @@ public final class TransportSettings {
         affixKeySetting("transport.profiles.", "tcp.keep_alive",
             key -> boolSetting(key, TCP_KEEP_ALIVE, Setting.Property.NodeScope));
     public static final Setting<Integer> TCP_KEEP_IDLE =
-        intSetting("transport.tcp.keep_idle", NetworkService.TCP_KEEP_IDLE, -1, 300, Setting.Property.NodeScope);
+        intSetting("transport.tcp.keep_idle", NetworkService.TCP_KEEP_IDLE, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL,
+                Setting.Property.NodeScope);
     public static final Setting.AffixSetting<Integer> TCP_KEEP_IDLE_PROFILE =
         affixKeySetting("transport.profiles.", "tcp.keep_idle",
-            key -> intSetting(key, TCP_KEEP_IDLE, -1, 300, Setting.Property.NodeScope));
+            key -> intSetting(key, TCP_KEEP_IDLE, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL, Setting.Property.NodeScope));
     public static final Setting<Integer> TCP_KEEP_INTERVAL =
-        intSetting("transport.tcp.keep_interval", NetworkService.TCP_KEEP_INTERVAL, -1, 300, Setting.Property.NodeScope);
+        intSetting("transport.tcp.keep_interval", NetworkService.TCP_KEEP_INTERVAL, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL,
+                Setting.Property.NodeScope);
     public static final Setting.AffixSetting<Integer> TCP_KEEP_INTERVAL_PROFILE =
         affixKeySetting("transport.profiles.", "tcp.keep_interval",
-            key -> intSetting(key, TCP_KEEP_INTERVAL, -1, 300, Setting.Property.NodeScope));
+            key -> intSetting(key, TCP_KEEP_INTERVAL, -1, MAX_REASONABLE_KEEPALIVE_INTERVAL, Setting.Property.NodeScope));
     public static final Setting<Integer> TCP_KEEP_COUNT =
         intSetting("transport.tcp.keep_count", NetworkService.TCP_KEEP_COUNT, -1, Setting.Property.NodeScope);
     public static final Setting.AffixSetting<Integer> TCP_KEEP_COUNT_PROFILE =

test/framework/src/main/java/org/elasticsearch/transport/AbstractSimpleTransportTestCase.java

@@ -85,6 +85,7 @@
 
 import static java.util.Collections.emptyMap;
 import static java.util.Collections.emptySet;
+import static org.elasticsearch.core.internal.net.NetUtils.MAX_REASONABLE_KEEPALIVE_INTERVAL;
 import static org.elasticsearch.transport.TransportService.NOOP_TRANSPORT_INTERCEPTOR;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.empty;
@@ -140,10 +141,10 @@ public void setUp() throws Exception {
 
         connectionSettingsBuilder.put(TransportSettings.TCP_KEEP_ALIVE.getKey(), randomBoolean());
         if (randomBoolean()) {
-            connectionSettingsBuilder.put(TransportSettings.TCP_KEEP_IDLE.getKey(), randomIntBetween(1, 300));
+            connectionSettingsBuilder.put(TransportSettings.TCP_KEEP_IDLE.getKey(), between(1, MAX_REASONABLE_KEEPALIVE_INTERVAL));
         }
         if (randomBoolean()) {
-            connectionSettingsBuilder.put(TransportSettings.TCP_KEEP_INTERVAL.getKey(), randomIntBetween(1, 300));
+            connectionSettingsBuilder.put(TransportSettings.TCP_KEEP_INTERVAL.getKey(), between(1, MAX_REASONABLE_KEEPALIVE_INTERVAL));
         }
         if (randomBoolean()) {
             connectionSettingsBuilder.put(TransportSettings.TCP_KEEP_COUNT.getKey(), randomIntBetween(1, 10));