From cf0cab614d690d22874e578b6257a924dbf1edb9 Mon Sep 17 00:00:00 2001 From: David Turner Date: Wed, 29 Jul 2020 13:01:17 +0100 Subject: [PATCH] Clarify remote clusters' use of transport layer (#60268) Today there are a few places in the transport layer docs where we talk about communication between nodes _within a cluster_. We also use the transport layer for remote cluster connections, and these statements also apply there, but this is not clear from today's docs. This commit generalises these statements to make it clear that they apply to remote cluster connections too. It also adds a link from the docs on configuring TCP retries to the (deeply-buried) docs on preserving long-lived connections. --- .../modules/remote-clusters.asciidoc | 8 ++-- docs/reference/modules/transport.asciidoc | 41 ++++++++----------- .../setup/sysconfig/tcpretries.asciidoc | 26 ++++++++---- 3 files changed, 40 insertions(+), 35 deletions(-) diff --git a/docs/reference/modules/remote-clusters.asciidoc b/docs/reference/modules/remote-clusters.asciidoc index eec5ae2cfe3f3..5fa89c0dce37b 100644 --- a/docs/reference/modules/remote-clusters.asciidoc +++ b/docs/reference/modules/remote-clusters.asciidoc @@ -10,10 +10,10 @@ to a limited number of nodes in that remote cluster. There are two modes for remote cluster connections: <> and <>. -All the communication required between different clusters -goes through the <>. Remote cluster -connections consist of uni-directional connections from the coordinating -node to the remote remote connections. +Communication with a remote cluster uses the <> to establishe a number of <> TCP +connections from the coordinating nodes of the local cluster to the chosen +nodes in the remote cluster. [discrete] [[sniff-mode]] diff --git a/docs/reference/modules/transport.asciidoc b/docs/reference/modules/transport.asciidoc index f821d5e567e82..2d8af86e76944 100644 --- a/docs/reference/modules/transport.asciidoc +++ b/docs/reference/modules/transport.asciidoc @@ -1,19 +1,13 @@ [[modules-transport]] === Transport -The transport networking layer is used for internal communication between nodes -within the cluster. Each call that goes from one node to the other uses -the transport layer (for example, when an HTTP GET request is processed -by one node, and should actually be processed by another node that holds -the data). The transport module is also used for the `TransportClient` in the -{es} Java API. - -The transport mechanism is completely asynchronous in nature, meaning -that there is no blocking thread waiting for a response. The benefit of -using asynchronous communication is first solving the -http://en.wikipedia.org/wiki/C10k_problem[C10k problem], as well as -being the ideal solution for scatter (broadcast) / gather operations such -as search in Elasticsearch. +REST clients send requests to your {es} cluster over <>, but +the node that receives a client request cannot always handle it alone and must +normally pass it on to other nodes for further processing. It does this using +the transport networking layer. The transport layer is used for all internal +communication between nodes within a cluster, all communication with the nodes +of a <>, and also by the +`TransportClient` in the {es} Java API. [[transport-settings]] ==== Transport settings @@ -138,17 +132,18 @@ configured, and defaults otherwise to `transport.tcp.reuse_address`. [[long-lived-connections]] ===== Long-lived idle connections -Elasticsearch opens a number of long-lived TCP connections between each pair of -nodes in the cluster, and some of these connections may be idle for an extended -period of time. Nonetheless, Elasticsearch requires these connections to remain -open, and it can disrupt the operation of the cluster if any inter-node -connections are closed by an external influence such as a firewall. It is -important to configure your network to preserve long-lived idle connections -between Elasticsearch nodes, for instance by leaving `tcp.keep_alive` enabled -and ensuring that the keepalive interval is shorter than any timeout that might +A transport connection between two nodes is made up of a number of long-lived +TCP connections, some of which may be idle for an extended period of time. +Nonetheless, Elasticsearch requires these connections to remain open, and it +can disrupt the operation of your cluster if any inter-node connections are +closed by an external influence such as a firewall. It is important to +configure your network to preserve long-lived idle connections between +Elasticsearch nodes, for instance by leaving `tcp.keep_alive` enabled and +ensuring that the keepalive interval is shorter than any timeout that might cause idle connections to be closed, or by setting `transport.ping_schedule` if -keepalives cannot be configured. - +keepalives cannot be configured. Devices which drop connections when they reach +a certain age are a common source of problems to Elasticsearch clusters, and +must not be used. [[request-compression]] ===== Request compression diff --git a/docs/reference/setup/sysconfig/tcpretries.asciidoc b/docs/reference/setup/sysconfig/tcpretries.asciidoc index 1b12bbe0e7f9b..e25c07c388a1a 100644 --- a/docs/reference/setup/sysconfig/tcpretries.asciidoc +++ b/docs/reference/setup/sysconfig/tcpretries.asciidoc @@ -2,8 +2,9 @@ === TCP retransmission timeout Each pair of nodes in a cluster communicates via a number of TCP connections -which remain open until one of the nodes shuts down or communication between -the nodes is disrupted by a failure in the underlying infrastructure. +which <> until one of the nodes shuts down +or communication between the nodes is disrupted by a failure in the underlying +infrastructure. TCP provides reliable communication over occasionally-unreliable networks by hiding temporary network disruptions from the communicating applications. Your @@ -36,14 +37,23 @@ To set this value permanently, update the `net.ipv4.tcp_retries2` setting in `/etc/sysctl.conf`. To verify after rebooting, run `sysctl net.ipv4.tcp_retries2`. -{es} also implements its own health checks with timeouts that are much shorter -than the default retransmission timeout on Linux. However these health checks -must allow for application-level effects such as garbage collection pauses. We -do not recommend reducing any timeouts related to these application-level -health checks. - IMPORTANT: This setting applies to all TCP connections and will affect the reliability of communication with systems outside your cluster too. If your cluster communicates with external systems over an unreliable network then you may need to select a higher value for `net.ipv4.tcp_retries2`. For this reason, {es} does not adjust this setting automatically. + +==== Related configuration + +{es} also implements its own internal health checks with timeouts that are much +shorter than the default retransmission timeout on Linux. Since these are +application-level health checks their timeouts must allow for application-level +effects such as garbage collection pauses. You should not reduce any timeouts +related to these application-level health checks. + +You must also ensure your network infrastructure does not interfere with the +long-lived connections between nodes, <>. Devices which drop connections when they reach +a certain age are a common source of problems to Elasticsearch clusters, and +must not be used. +