Move file-based discovery to core

Today we support a static list of seed hosts in core Elasticsearch, and allow a
dynamic list of seed hosts to be provided via a file using the `discovery-file`
plugin. In fact the ability to provide a dynamic list of seed hosts is
increasingly useful in containerized environments, so this change moves this
functionality to core Elasticsearch to avoid the need for a plugin.

For BWC purposes the plugin still exists, but does nothing more than issue a
warning when it is used.

Furthermore, in order to start up nodes in integration tests we currently
assign a known port to each node before startup, which unfortunately sometimes
fails if another process grabs the selected port in the meantime. By moving the
`discovery-file` functionality into the core product we can use it to avoid
this race.

Relates elastic#29244
Closes elastic#33030

plugins/discovery-file/config/discovery-file/unicast_hosts.txt → distribution/src/config/discovery-file/unicast_hosts.txt

@@ -3,18 +3,23 @@
 # mechanism.  This file should contain one entry per line, where an entry is a
 # host/port combination.  The host and port should be separated by a `:`.  If
 # the port is left off, a default port of 9300 is assumed.  For example, if the
-# cluster has three nodes that participate in the discovery process:
-# (1) 66.77.88.99 running on port 9300 (2) 66.77.88.100 running on port 9305
-# and (3) 66.77.88.99 running on port 10005, then this file should contain the
+# cluster has three nodes that participate in the discovery process: (1)
+# 66.77.88.99 running on port 9300 (2) 66.77.88.100 running on port 9305 and
+# (3) 66.77.88.99 running on port 10005, then this file should contain the
 # following text:
 #
 #10.10.10.5
 #10.10.10.6:9305
 #10.10.10.5:10005
 #
-# For IPv6 addresses, make sure to put a bracket around the host part of the address,
-# for example: [2001:cdba:0000:0000:0000:0000:3257:9652]:9301 (where 9301 is the port).
+# For IPv6 addresses, make sure to put a bracket around the host part of the
+# address, for example: [2001:cdba:0000:0000:0000:0000:3257:9652]:9301 (where
+# 9301 is the port).
+#
+# NOTE: all lines starting with a `#` are comments, and comments must exist on
+# lines of their own (i.e. comments cannot begin in the middle of a line)
+#
+# NOTE: Set `discovery.zen.hosts_provider: file` to enable file-based discovery
+# and use this file. If this setting is not set then Elasticsearch will not
+# read this file.
 #
-# NOTE: all lines starting with a `#` are comments, and comments must exist
-# on lines of their own (i.e. comments cannot begin in the middle of a line)
-#

docs/plugins/discovery-file.asciidoc

@@ -1,71 +1,10 @@
 [[discovery-file]]
 === File-Based Discovery Plugin
 
-The file-based discovery plugin uses a list of hosts/ports in a `unicast_hosts.txt` file
-in the `config/discovery-file` directory for unicast discovery.
+The functionality provided by the `discovery-file` plugin is now available in
+Elasticsearch without requiring a plugin. This plugin still exists for backwards
+compatibility, but it no longer does anything and will be removed in a future
+version.
 
 :plugin_name: discovery-file
 include::install_remove.asciidoc[]
-
-[[discovery-file-usage]]
-[float]
-==== Using the file-based discovery plugin
-
-The file-based discovery plugin provides the ability to specify the
-unicast hosts list through a simple `unicast_hosts.txt` file that can
-be dynamically updated at any time. To enable, add the following in `elasticsearch.yml`:
-
-[source,yaml]
-----
-discovery.zen.hosts_provider: file
-----
-
-This plugin simply provides a facility to supply the unicast hosts list for
-zen discovery through an external file that can be updated at any time by a side process.
-
-For example, this gives a convenient mechanism for an Elasticsearch instance
-that is  run in docker containers to be dynamically supplied a list of IP
-addresses to connect to for zen discovery when those IP addresses may not be
-known at node startup.
-
-Note that the file-based discovery plugin is meant to augment the unicast
-hosts list in `elasticsearch.yml` (if specified), not replace it.  Therefore,
-if there are valid unicast host entries in `discovery.zen.ping.unicast.hosts`,
-they will be used in addition to those supplied in `unicast_hosts.txt`.
-
-Anytime a change is made to the `unicast_hosts.txt` file, even as Elasticsearch
-continues to run, the new changes will be picked up by the plugin and the
-new hosts list will be used for the next pinging round for master election.
-
-Upon installation of the plugin, a default `unicast_hosts.txt` file will
-be found in the `$CONFIG_DIR/discovery-file` directory.  This default file
-will contain some comments about what the file should contain.  All comments
-for this file must appear on their lines starting with `#` (i.e. comments
-cannot start in the middle of a line).
-
-[[discovery-file-format]]
-[float]
-==== unicast_hosts.txt file format
-
-The format of the file is to specify one unicast host entry per line.
-Each unicast host entry consists of the host (host name or IP address) and
-an optional transport port number.  If the port number is specified, is must
-come immediately after the host (on the same line) separated by a `:`.
-If the port number is not specified, a default value of 9300 is used.
-
-For example, this is an example of `unicast_hosts.txt` for a cluster with
-four nodes that participate in unicast discovery, some of which are not
-running on the default port:
-
-[source,txt]
-----------------------------------------------------------------
-10.10.10.5
-10.10.10.6:9305
-10.10.10.5:10005
-# an IPv6 address
-[2001:0db8:85a3:0000:0000:8a2e:0370:7334]:9301
-----------------------------------------------------------------
-
-Host names are allowed instead of IP addresses (similar to
-`discovery.zen.ping.unicast.hosts`), and IPv6 addresses must be
-specified in brackets with the port coming after the brackets.