Merge branch 'master' into feature/aggregate-metrics

buildSrc/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 8.0.0
-lucene            = 8.5.1
+lucene            = 8.6.0-snapshot-6c9024f7735
 
 bundled_jdk_vendor = adoptopenjdk
 bundled_jdk = 14+36

docs/Versions.asciidoc

@@ -1,8 +1,8 @@
 
 include::{asciidoc-dir}/../../shared/versions/stack/{source_branch}.asciidoc[]
 
-:lucene_version:        8.5.1
-:lucene_version_path:   8_5_1
+:lucene_version:        8.6.0
+:lucene_version_path:   8_6_0
 :jdk:                   11.0.2
 :jdk_major:             11
 :build_flavor:          default

docs/plugins/discovery-ec2.asciidoc

@@ -236,7 +236,8 @@ The `discovery-ec2` plugin can automatically set the `aws_availability_zone`
 node attribute to the availability zone of each node. This node attribute
 allows you to ensure that each shard has copies allocated redundantly across
 multiple availability zones by using the
-{ref}/allocation-awareness.html[Allocation Awareness] feature.
+{ref}/modules-cluster.html#shard-allocation-awareness[Allocation Awareness]
+feature.
 
 In order to enable the automatic definition of the `aws_availability_zone`
 attribute, set `cloud.node.auto_attributes` to `true`. For example:
@@ -327,8 +328,9 @@ labelled as `Moderate` or `Low`.
 
 * It is a good idea to distribute your nodes across multiple
 http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[availability
-zones] and use {ref}/allocation-awareness.html[shard allocation awareness] to
-ensure that each shard has copies in more than one availability zone.
+zones] and use {ref}/modules-cluster.html#shard-allocation-awareness[shard
+allocation awareness] to ensure that each shard has copies in more than one
+availability zone.
 
 * Do not span a cluster across regions. {es} expects that node-to-node
 connections within a cluster are reasonably reliable and offer high bandwidth

docs/reference/aggregations/bucket/datehistogram-aggregation.asciidoc

@@ -183,7 +183,6 @@ Defined as 1000 milliseconds each
 
 minutes (m) ::
 All minutes begin at 00 seconds.
-
 Defined as 60 seconds each (60,000 milliseconds)
 
 hours (h) ::
@@ -193,7 +192,6 @@ Defined as 60 minutes each (3,600,000 milliseconds)
 days (d) ::
 All days begin at the earliest possible time, which is usually 00:00:00
 (midnight).
-
 Defined as 24 hours (86,400,000 milliseconds)
 
 [[fixed_interval_examples]]

docs/reference/api-conventions.asciidoc

@@ -8,6 +8,7 @@ API, unless otherwise specified.
 
 * <<multi-index>>
 * <<date-math-index-names>>
+* <<cron-expressions>>
 * <<common-options>>
 * <<url-access-control>>
 
@@ -143,6 +144,8 @@ GET /%3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogs
 // TEST[s/^/PUT logstash-2016.09.20\nPUT logstash-2016.09.19\nPUT logstash-2016.09.18\n/]
 // TEST[s/now/2016.09.20%7C%7C/]
 
+include::rest-api/cron-expressions.asciidoc[]
+
 [[common-options]]
 === Common options
 

docs/reference/ccr/apis/follow/get-follow-info.asciidoc

@@ -23,13 +23,6 @@ PUT /follower_index/_ccr/follow?wait_for_active_shards=1
 --------------------------------------------------
 // TESTSETUP
 // TEST[setup:remote_cluster_and_leader_index]
-
-[source,console]
---------------------------------------------------
-POST /follower_index/_ccr/pause_follow
---------------------------------------------------
-// TEARDOWN
-
 //////////////////////////
 
 [source,console]
@@ -76,7 +69,8 @@ replication options and whether the follower indices are active or paused.
 
 //Begin parameters
 `parameters`::
-(object) An object that encapsulates {ccr} parameters.
+(object) An object that encapsulates {ccr} parameters. If the follower index's `status` is `paused`,
+this object is omitted.
 +
 .Properties of `parameters`
 [%collapsible%open]
@@ -132,7 +126,7 @@ to read from the leader again.
 leader index.
 
 `status`::
-(string) Whether index following is `active` or `paused`. 
+(string) Whether index following is `active` or `paused`.
 ====
 //End follower_indices
 
@@ -146,7 +140,7 @@ This example retrieves follower info:
 GET /follower_index/_ccr/info
 --------------------------------------------------
 
-The API returns the following results:
+If the follower index is `active`, the API returns the following results:
 
 [source,console-result]
 --------------------------------------------------
@@ -173,3 +167,33 @@ The API returns the following results:
     ]
 }
 --------------------------------------------------
+
+////
+[source,console]
+--------------------------------------------------
+POST /follower_index/_ccr/pause_follow
+--------------------------------------------------
+// TEST[continued]
+
+[source,console]
+--------------------------------------------------
+GET /follower_index/_ccr/info
+--------------------------------------------------
+// TEST[continued]
+////
+
+If the follower index is `paused`, the API returns the following results:
+
+[source,console-result]
+--------------------------------------------------
+{
+    "follower_indices" : [
+        {
+            "follower_index" : "follower_index",
+            "remote_cluster" : "remote_cluster",
+            "leader_index" : "leader_index",
+            "status" : "paused"
+        }
+    ]
+}
+--------------------------------------------------

docs/reference/commands/croneval.asciidoc

@@ -3,7 +3,7 @@
 [[elasticsearch-croneval]]
 == elasticsearch-croneval
 
-Validates and evaluates a cron expression. 
+Validates and evaluates a <<cron-expressions,cron expression>>. 
 
 [discrete]
 === Synopsis
@@ -19,12 +19,11 @@ bin/elasticsearch-croneval <expression>
 === Description
 
 This command enables you to verify that your
-https://en.wikipedia.org/wiki/Cron[cron] expressions are valid for use with the
-{es} {alert-features} and produce the expected results.
+cron expressions are valid for use with
+{es} and produce the expected results.
 
 This command is provided in the `$ES_HOME/bin` directory.
 
-[discrete]
 === Parameters
 
 `-c, --count` <Integer>::
@@ -45,7 +44,7 @@ This command is provided in the `$ES_HOME/bin` directory.
   Shows verbose output.
 
 [discrete]
-=== Examples
+=== Example
 
 If the cron expression is valid, the following command displays the next
 20 times that the schedule will be triggered:

docs/reference/eql/eql-search-api.asciidoc

@@ -84,13 +84,6 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
 [[eql-search-api-request-body]]
 ==== {api-request-body-title}
 
-`query`::
-(Required, string)
-<<eql-syntax,EQL>> query you wish to run.
-+
-IMPORTANT: This parameter supports a subset of EQL syntax. See
-<<eql-unsupported-syntax>>.
-
 `event_category_field`::
 (Required*, string)
 Field containing the event classification, such as `process`, `file`, or
@@ -100,6 +93,31 @@ Defaults to `event.category`, as defined in the {ecs-ref}/ecs-event.html[Elastic
 Common Schema (ECS)]. If an index does not contain the `event.category` field,
 this value is required.
 
+`filter`::
+(Optional, <<query-dsl,query DSL object>>)
+Query, written in query DSL, used to filter the events on which the EQL query
+runs.
+
+`implicit_join_key_field`::
+(Optional, string)
+Reserved for future use.
+
+`query`::
+(Required, string)
+<<eql-syntax,EQL>> query you wish to run.
++
+IMPORTANT: This parameter supports a subset of EQL syntax. See
+<<eql-unsupported-syntax>>.
+
+`search_after`::
+(Optional, string)
+Reserved for future use.
+
+`size`::
+(Optional, integer or float)
+Maximum number of matching events to return. Defaults to `50`. Values must be
+greater than `0`.
+
 [[eql-search-api-timestamp-field]]
 `timestamp_field`::
 +
@@ -116,24 +134,6 @@ milliseconds since the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in
 ascending order.
 --
 
-`implicit_join_key_field`::
-(Optional, string)
-Reserved for future use.
-
-`filter`::
-(Optional, <<query-dsl,query DSL object>>)
-Query, written in query DSL, used to filter the events on which the EQL query
-runs.
-
-`search_after`::
-(Optional, string)
-Reserved for future use.
-
-`size`::
-(Optional, integer or float)
-Maximum number of matching events to return. Defaults to `50`. Values must be
-greater than `0`.
-
 [role="child_attributes"]
 [[eql-search-api-response-body]]
 ==== {api-response-body-title}

docs/reference/ilm/actions/ilm-rollover.asciidoc

@@ -43,19 +43,24 @@ PUT my_index-000001
 You must specify at least one rollover condition. 
 An empty rollover action is invalid.
 
-`max_size`::
-(Optional, <<byte-units, byte units>>)
-Triggers roll over after the specified maximum primary shard index storage size is reached.  
+`max_age`::
+(Optional,  <<time-units, time units>>)  
+Triggers roll over after the maximum elapsed time from index creation is reached. 
 
 `max_docs`::
 (Optional, integer)
 Triggers roll over after the specified maximum number of documents is reached.
 Documents added since the last refresh are not included in the document count.
 The document count does *not* include documents in replica shards.
 
-`max_age`::
-(Optional,  <<time-units, time units>>)  
-Triggers roll over after the maximum elapsed time from index creation is reached. 
+`max_size`::
+(Optional, <<byte-units, byte units>>)
+Triggers roll over when the index reaches a certain size.
+This is the total size of all primary shards in the index. 
+Replicas are not counted toward the maximum index size.
++
+TIP: To see the current index size, use the <<cat-indices, _cat indices>> API. 
+The `pri.store.size` value shows the combined size of all primary shards.
 
 [[ilm-rollover-ex]]
 ==== Example

docs/reference/ilm/apis/move-to-step.asciidoc

@@ -47,6 +47,58 @@ an unexpected step into the next step.
 
 include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
 
+[role="child_attributes"]
+[[ilm-move-to-step-request-body]]
+==== {api-request-body-title}
+
+`current_step`::
+(Required, object)
++
+.Properties of `current_step`
+[%collapsible%open]
+====
+`phase`::
+(Required, string)
+The name of the current phase.
+Must match the phase as returned by the <<ilm-explain-lifecycle, explain>> API.
+
+`action`::
+(Required, string)
+The name of the current action. 
+Must match the action as returned by the <<ilm-explain-lifecycle, explain>> API.
+
+`name`::
+(Required, string)
+The name of the current step. 
+Must match the step as returned by the <<ilm-explain-lifecycle, explain>> API.
+If {ilm-init} encounters a problem while performing an action, 
+it halts execution of the policy and transitions to the `ERROR` step.
+If you are trying to advance a policy after troubleshooting a failure, 
+you specify this `ERROR` step as the current step. 
+For more information, see <<index-lifecycle-error-handling, {ilm-init} error handling>>.
+
+====
+
+`next_step`::
+(Required, object)
++
+.Properties of `next_step`
+[%collapsible%open]
+====
+`phase`::
+(Required, string)
+The name of the phase that contains the action you want to perform or resume.
+
+`action`::
+(Required, string)
+The name action you want to perform or resume. 
+
+`name`::
+(Required, string)
+The name of the step to move to and execute. 
+
+====
+
 [[ilm-move-to-step-example]]
 ==== {api-examples-title}
 

docs/reference/ilm/using-policies-rollover.asciidoc

@@ -4,10 +4,17 @@
 == Configure rollover
 [[using-policies-rollover]]
 You control when the rollover action is triggered by specifying one or more
-rollover parameters. The rollover is performed once any of the criteria are
-met. Because the criteria are checked periodically, the index might grow
-slightly beyond the specified threshold. To control how often the criteria are
-checked, specify the `indices.lifecycle.poll_interval` cluster setting.
+rollover criteria:
+
+* Maximum size (the combined size of all primary shards in the index)
+* Maximum document count
+* Maximum age
+
+The rollover is performed once any of the criteria are met. 
+Because the criteria are checked periodically, the index might grow
+slightly beyond the specified threshold. 
+To control how often the criteria are checked, 
+specify the `indices.lifecycle.poll_interval` cluster setting.
 
 IMPORTANT: New indices created via rollover will not automatically inherit the
 policy used by the old index, and will not use any policy by default. Therefore,
@@ -16,24 +23,6 @@ it is highly recommended to apply the policy via
 setting, for your indices which specifies the policy you wish to use for each
 new index.
 
-The rollover action takes the following parameters:
-
-[[rollover-action-params]]
-.`rollover` Action Parameters
-[options="header"]
-|===
-|Name |Description
-|max_size |The maximum estimated size the primary shard of the index is allowed
-to grow to. Defaults to `null`. Optional.
-|max_docs |The maximum number of document the index should
-contain. Defaults to `null`. Optional.
-|max_age |The maximum age of the index. Defaults to `null`. Optional.
-|===
-
-These parameters are used to determine when the index is considered "full" and
-a rollover should be performed. Where multiple criteria are defined the
-rollover operation will be performed once any of the criteria are met.
-
 The following request defines a policy with a rollover action that triggers
 when the index size reaches 25GB. The old index is subsequently deleted after
 30 days.
@@ -127,7 +116,7 @@ the new index, enabling indexing to continue uninterrupted.
 [[skipping-rollover]]
 === Skipping Rollover
 
-The `index.lifecycle.indexing_complete` setting indicates to {ilm} whether this
+The `index.lifecycle.indexing_complete` setting indicates to {ilm-init} whether this
 index has already been rolled over. If it is set to `true`, that indicates that
 this index has already been rolled over and does not need to be rolled over
 again. Therefore, {ilm} will skip any Rollover Action configured in the