Merge remote-tracking branch 'elastic/master' into retention-lease-ex…

…piration

* elastic/master:
  Removing unused methods in Numbers (elastic#37186)
  Fix setting by time unit (elastic#37192)
  [DOCS] Cleans up xpackml attributes
  ML: fix delayed data annotations on secured cluster (elastic#37193)
  Update version in SearchRequest and related test
  [DOCS] Adds overview and API ref for cluster voting configurations (elastic#36954)
  ML: changing JobResultsProvider.getForecastRequestStats to support > 1 index (elastic#37157)
  Separate out validation of groups of settings (elastic#34184)

docs/reference/cluster.asciidoc

@@ -104,3 +104,5 @@ include::cluster/tasks.asciidoc[]
 include::cluster/nodes-hot-threads.asciidoc[]
 
 include::cluster/allocation-explain.asciidoc[]
+
+include::cluster/voting-exclusions.asciidoc[]

docs/reference/cluster/voting-exclusions.asciidoc

@@ -0,0 +1,76 @@
+[[voting-config-exclusions]]
+== Voting configuration exclusions API
+++++
+<titleabbrev>Voting Configuration Exclusions</titleabbrev>
+++++
+
+Adds or removes master-eligible nodes from the
+<<modules-discovery-voting,voting configuration exclusion list>>.
+
+[float]
+=== Request
+
+`POST _cluster/voting_config_exclusions/<node_name>` +
+
+`DELETE _cluster/voting_config_exclusions`
+
+[float]
+=== Path parameters
+
+`node_name`::
+  A <<cluster-nodes,node filter>> that identifies {es} nodes.
+
+[float]
+=== Description
+
+By default, if there are more than three master-eligible nodes in the cluster
+and you remove fewer than half of the master-eligible nodes in the cluster at
+once, the <<modules-discovery-voting,voting configuration>> automatically
+shrinks.
+
+If you want to shrink the voting configuration to contain fewer than three nodes
+or to remove half or more of the master-eligible nodes in the cluster at once,
+you must use this API to remove departed nodes from the voting configuration
+manually. It adds an entry for that node in the voting configuration exclusions
+list. The cluster then tries to reconfigure the voting configuration to remove
+that node and to prevent it from returning.
+
+If the API fails, you can safely retry it.  Only a successful response
+guarantees that the node has been removed from the voting configuration and will
+not be reinstated.
+
+NOTE: Voting exclusions are required only when you remove at least half of the
+master-eligible nodes from a cluster in a short time period. They are not
+required when removing master-ineligible nodes or fewer than half of the
+master-eligible nodes.
+
+The <<modules-discovery-settings,`cluster.max_voting_config_exclusions`
+setting>> limits the size of the voting configuration exclusion list. The
+default value is `10`. Since voting configuration exclusions are persistent and
+limited in number, you must clear the voting config exclusions list once the
+exclusions are no longer required.
+
+There is also a
+<<modules-discovery-settings,`cluster.auto_shrink_voting_configuration` setting>>,
+which is set to true by default. If it is set to false, you must use this API to
+maintain the voting configuration.
+
+For more information, see <<modules-discovery-removing-nodes>>.
+
+[float]
+=== Examples
+
+Add `nodeId1` to the voting configuration exclusions list:
+[source,js]
+-------------------------------------------------- 
+POST /_cluster/voting_config_exclusions/nodeId1
+--------------------------------------------------
+// CONSOLE
+// TEST[catch:bad_request]
+
+Remove all exclusions from the list:
+[source,js]
+--------------------------------------------------
+DELETE /_cluster/voting_config_exclusions
+--------------------------------------------------
+// CONSOLE

docs/reference/ml/aggregations.asciidoc

@@ -8,7 +8,7 @@ and to configure your jobs to analyze aggregated data.
 
 One of the benefits of aggregating data this way is that {es} automatically
 distributes these calculations across your cluster. You can then feed this
-aggregated data into {xpackml} instead of raw results, which
+aggregated data into the {ml-features} instead of raw results, which
 reduces the volume of data that must be considered while detecting anomalies.
 
 There are some limitations to using aggregations in {dfeeds}, however.

docs/reference/ml/apis/resultsresource.asciidoc

@@ -269,7 +269,7 @@ probability of this occurrence.
 
 There can be many anomaly records depending on the characteristics and size of
 the input data. In practice, there are often too many to be able to manually
-process them. The {xpackml} features therefore perform a sophisticated
+process them. The {ml-features} therefore perform a sophisticated
 aggregation of the anomaly records into buckets.
 
 The number of record results depends on the number of anomalies found in each

docs/reference/ml/configuring.asciidoc

@@ -2,12 +2,12 @@
 [[ml-configuring]]
 == Configuring machine learning
 
-If you want to use {xpackml} features, there must be at least one {ml} node in
+If you want to use {ml-features}, there must be at least one {ml} node in
 your cluster and all master-eligible nodes must have {ml} enabled. By default,
 all nodes are {ml} nodes. For more information about these settings, see 
 {ref}/modules-node.html#modules-node-xpack[{ml} nodes].
 
-To use the {xpackml} features to analyze your data, you must create a job and
+To use the {ml-features} to analyze your data, you must create a job and
 send your data to that job.
 
 * If your data is stored in {es}:

docs/reference/ml/functions.asciidoc

@@ -2,7 +2,7 @@
 [[ml-functions]]
 == Function reference
 
-The {xpackml} features include analysis functions that provide a wide variety of
+The {ml-features} include analysis functions that provide a wide variety of
 flexible ways to analyze data for anomalies.
 
 When you create jobs, you specify one or more detectors, which define the type of

docs/reference/ml/functions/count.asciidoc

@@ -14,7 +14,7 @@ in one field is unusual, as opposed to the total count.
 Use high-sided functions if you want to monitor unusually high event rates.
 Use low-sided functions if you want to look at drops in event rate.
 
-The {xpackml} features include the following count functions:
+The {ml-features} include the following count functions:
 
 * xref:ml-count[`count`, `high_count`, `low_count`]
 * xref:ml-nonzero-count[`non_zero_count`, `high_non_zero_count`, `low_non_zero_count`]

docs/reference/ml/functions/geo.asciidoc

@@ -5,7 +5,7 @@
 The geographic functions detect anomalies in the geographic location of the
 input data.
 
-The {xpackml} features include the following geographic function: `lat_long`.
+The {ml-features} include the following geographic function: `lat_long`.
 
 NOTE: You cannot create forecasts for jobs that contain geographic functions. 
 You also cannot add rules with conditions to detectors that use geographic 
@@ -72,7 +72,7 @@ For example, JSON data might contain the following transaction coordinates:
 
 In {es}, location data is likely to be stored in `geo_point` fields. For more
 information, see {ref}/geo-point.html[Geo-point datatype]. This data type is not
-supported natively in {xpackml} features. You can, however, use Painless scripts
+supported natively in {ml-features}. You can, however, use Painless scripts
 in `script_fields` in your {dfeed} to transform the data into an appropriate
 format. For example, the following Painless script transforms
 `"coords": {"lat" : 41.44, "lon":90.5}` into `"lat-lon": "41.44,90.5"`:

docs/reference/ml/functions/info.asciidoc

@@ -6,7 +6,7 @@ that is contained in strings within a bucket. These functions can be used as
 a more sophisticated method to identify incidences of data exfiltration or
 C2C activity, when analyzing the size in bytes of the data might not be sufficient.
 
-The {xpackml} features include the following information content functions:
+The {ml-features} include the following information content functions:
 
 * `info_content`, `high_info_content`, `low_info_content`
 

docs/reference/ml/functions/metric.asciidoc

@@ -6,7 +6,7 @@ The metric functions include functions such as mean, min and max. These values
 are calculated for each bucket. Field values that cannot be converted to
 double precision floating point numbers are ignored.
 
-The {xpackml} features include the following metric functions:
+The {ml-features} include the following metric functions:
 
 * <<ml-metric-min,`min`>>
 * <<ml-metric-max,`max`>>

docs/reference/ml/functions/rare.asciidoc

@@ -27,7 +27,7 @@ with shorter bucket spans typically being measured in minutes, not hours.
 for typical data.
 ====
 
-The {xpackml} features include the following rare functions:
+The {ml-features} include the following rare functions:
 
 * <<ml-rare,`rare`>>
 * <<ml-freq-rare,`freq_rare`>>
@@ -85,7 +85,7 @@ different rare status codes compared to the population is regarded as highly
 anomalous. This analysis is based on the number of different status code values,
 not the count of occurrences.
 
-NOTE: To define a status code as rare the {xpackml} features look at the number
+NOTE: To define a status code as rare the {ml-features} look at the number
 of distinct status codes that occur, not the number of times the status code
 occurs. If a single client IP experiences a single unique status code, this
 is rare, even if it occurs for that client IP in every bucket.

docs/reference/ml/functions/sum.asciidoc

@@ -11,7 +11,7 @@ If want to look at drops in totals, use low-sided functions.
 If your data is sparse, use `non_null_sum` functions. Buckets without values are
 ignored; buckets with a zero value are analyzed.
 
-The {xpackml} features include the following sum functions:
+The {ml-features} include the following sum functions:
 
 * xref:ml-sum[`sum`, `high_sum`, `low_sum`]
 * xref:ml-nonnull-sum[`non_null_sum`, `high_non_null_sum`, `low_non_null_sum`]

docs/reference/ml/functions/time.asciidoc

@@ -6,7 +6,7 @@ The time functions detect events that happen at unusual times, either of the day
 or of the week. These functions can be used to find unusual patterns of behavior,
 typically associated with suspicious user activity.
 
-The {xpackml} features include the following time functions:
+The {ml-features} include the following time functions:
 
 * <<ml-time-of-day,`time_of_day`>>
 * <<ml-time-of-week,`time_of_week`>>

docs/reference/ml/transforms.asciidoc

@@ -569,7 +569,7 @@ GET _ml/datafeeds/datafeed-test4/_preview
 // TEST[skip:needs-licence]
 
 In {es}, location data can be stored in `geo_point` fields but this data type is
-not supported natively in {xpackml} analytics. This example of a script field
+not supported natively in {ml} analytics. This example of a script field
 transforms the data into an appropriate format. For more information,
 see <<ml-geo-functions>>.
 

docs/reference/modules/discovery.asciidoc

@@ -13,6 +13,16 @@ module. This module is divided into the following sections:
     unknown, such as when a node has just started up or when the previous
     master has failed.
 
+<<modules-discovery-quorums>>::
+
+    This section describes how {es} uses a quorum-based voting mechanism to
+    make decisions even if some nodes are unavailable.
+
+<<modules-discovery-voting>>::
+
+    This section describes the concept of voting configurations, which {es}
+    automatically updates as nodes leave and join the cluster.
+
 <<modules-discovery-bootstrap-cluster>>::
 
     Bootstrapping a cluster is required when an Elasticsearch cluster starts up
@@ -40,26 +50,27 @@ module. This module is divided into the following sections:
     Cluster state publishing is the process by which the elected master node
     updates the cluster state on all the other nodes in the cluster.
 
-<<modules-discovery-quorums>>::
+<<cluster-fault-detection>>::
+
+    {es} performs health checks to detect and remove faulty nodes.
 
-    This section describes the detailed design behind the master election and
-    auto-reconfiguration logic.
-
 <<modules-discovery-settings,Settings>>::
 
     There are settings that enable users to influence the discovery, cluster
     formation, master election and fault detection processes.    
 
 include::discovery/discovery.asciidoc[]
 
+include::discovery/quorums.asciidoc[]
+
+include::discovery/voting.asciidoc[]
+
 include::discovery/bootstrapping.asciidoc[]
 
 include::discovery/adding-removing-nodes.asciidoc[]
 
 include::discovery/publishing.asciidoc[]
 
-include::discovery/quorums.asciidoc[]
-
 include::discovery/fault-detection.asciidoc[]
 
-include::discovery/discovery-settings.asciidoc[]
+include::discovery/discovery-settings.asciidoc[]

docs/reference/modules/discovery/adding-removing-nodes.asciidoc

@@ -12,6 +12,7 @@ cluster, and to scale the cluster up and down by adding and removing
 master-ineligible nodes only. However there are situations in which it may be
 desirable to add or remove some master-eligible nodes to or from a cluster.
 
+[[modules-discovery-adding-nodes]]
 ==== Adding master-eligible nodes
 
 If you wish to add some nodes to your cluster, simply configure the new nodes
@@ -24,6 +25,7 @@ cluster. You can use the `cluster.join.timeout` setting to configure how long a
 node waits after sending a request to join a cluster. Its default value is `30s`.
 See <<modules-discovery-settings>>.
 
+[[modules-discovery-removing-nodes]]
 ==== Removing master-eligible nodes
 
 When removing master-eligible nodes, it is important not to remove too many all
@@ -50,7 +52,7 @@ will never automatically move a node on the voting exclusions list back into the
 voting configuration. Once an excluded node has been successfully
 auto-reconfigured out of the voting configuration, it is safe to shut it down
 without affecting the cluster's master-level availability. A node can be added
-to the voting configuration exclusion list using the following API:
+to the voting configuration exclusion list using the <<voting-config-exclusions>> API. For example:
 
 [source,js]
 --------------------------------------------------

docs/reference/modules/discovery/discovery-settings.asciidoc

@@ -3,6 +3,15 @@
 
 Discovery and cluster formation are affected by the following settings:
 
+`cluster.auto_shrink_voting_configuration`::
+
+    Controls whether the <<modules-discovery-voting,voting configuration>>
+    sheds departed nodes automatically, as long as it still contains at least 3
+    nodes. The default value is `true`. If set to `false`, the voting
+    configuration never shrinks automatically and you must remove departed
+    nodes manually with the <<voting-config-exclusions,voting configuration
+    exclusions API>>.
+
 [[master-election-settings]]`cluster.election.back_off_time`::
 
     Sets the amount to increase the upper bound on the wait before an election
@@ -152,9 +161,11 @@ APIs are not be blocked and can run on any available node.
 
     Provides a list of master-eligible nodes in the cluster. The list contains
     either an array of hosts or a comma-delimited string. Each value has the
-    format `host:port` or `host`, where `port` defaults to the setting `transport.profiles.default.port`. Note that IPv6 hosts must be bracketed.
+    format `host:port` or `host`, where `port` defaults to the setting
+    `transport.profiles.default.port`. Note that IPv6 hosts must be bracketed.
     The default value is `127.0.0.1, [::1]`. See <<unicast.hosts>>.
 
 `discovery.zen.ping.unicast.hosts.resolve_timeout`::
 
-    Sets the amount of time to wait for DNS lookups on each round of discovery. This is specified as a <<time-units, time unit>> and defaults to `5s`.
+    Sets the amount of time to wait for DNS lookups on each round of discovery.
+    This is specified as a <<time-units, time unit>> and defaults to `5s`.

docs/reference/modules/discovery/fault-detection.asciidoc

@@ -2,8 +2,9 @@
 === Cluster fault detection
 
 The elected master periodically checks each of the nodes in the cluster to
-ensure that they are still connected and healthy. Each node in the cluster also periodically checks the health of the elected master. These checks
-are known respectively as _follower checks_ and _leader checks_.
+ensure that they are still connected and healthy. Each node in the cluster also
+periodically checks the health of the elected master. These checks are known
+respectively as _follower checks_ and _leader checks_.
 
 Elasticsearch allows these checks to occasionally fail or timeout without
 taking any action. It considers a node to be faulty only after a number of
@@ -16,4 +17,4 @@ and retry setting values and attempts to remove the node from the cluster.
 Similarly, if a node detects that the elected master has disconnected, this
 situation is treated as an immediate failure. The node bypasses the timeout and
 retry settings and restarts its discovery phase to try and find or elect a new
-master.
+master.