Merge branch 'master' into Read_ingest_pipeline

BigPandaToo · Dec 10, 2020 · 90cef0b · 90cef0b
2 parents 1d38ed9 + 27ca279
commit 90cef0b
Show file tree

Hide file tree

Showing 530 changed files with 4,551 additions and 1,040 deletions.
diff --git a/.ci/bwcVersions b/.ci/bwcVersions
@@ -27,5 +27,6 @@ BWC_VERSION:
   - "7.9.3"
   - "7.10.0"
   - "7.10.1"
+  - "7.10.2"
   - "7.11.0"
   - "8.0.0"
diff --git a/docs/reference/docs/delete-by-query.asciidoc b/docs/reference/docs/delete-by-query.asciidoc
@@ -213,7 +213,10 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll_size]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_timeout]
+`search_timeout`::
+(Optional, <<time-units, time units>>)
+Explicit timeout for each search request.
+Defaults to no timeout.
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=slices]
 

diff --git a/docs/reference/docs/update-by-query.asciidoc b/docs/reference/docs/update-by-query.asciidoc
@@ -213,7 +213,10 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll_size]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_timeout]
+`search_timeout`::
+(Optional, <<time-units, time units>>)
+Explicit timeout for each search request.
+Defaults to no timeout.
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=slices]
 

diff --git a/docs/reference/eql/delete-async-eql-search-api.asciidoc b/docs/reference/eql/delete-async-eql-search-api.asciidoc
@@ -7,8 +7,6 @@
 <titleabbrev>Delete async EQL search</titleabbrev>
 ++++
 
-beta::[]
-
 Deletes an <<eql-search-async,async EQL search>> or a
 <<eql-search-store-sync-eql-search,stored synchronous EQL search>>. The API also
 deletes results for the search.

diff --git a/docs/reference/eql/detect-threats-with-eql.asciidoc b/docs/reference/eql/detect-threats-with-eql.asciidoc
@@ -3,8 +3,6 @@
 [[eql-ex-threat-detection]]
 == Example: Detect threats with EQL
 
-beta::[]
-
 This example tutorial shows how you can use EQL to detect security threats and
 other suspicious behavior. In the scenario, you're tasked with detecting
 https://attack.mitre.org/techniques/T1218/010/[regsvr32 misuse] in Windows event

diff --git a/docs/reference/eql/eql-search-api.asciidoc b/docs/reference/eql/eql-search-api.asciidoc
@@ -7,8 +7,6 @@
 <titleabbrev>EQL search</titleabbrev>
 ++++
 
-beta::[]
-
 Returns search results for an <<eql,Event Query Language (EQL)>> query.
 
 EQL assumes each document in a data stream or index corresponds to an

diff --git a/docs/reference/eql/eql.asciidoc b/docs/reference/eql/eql.asciidoc
@@ -6,8 +6,6 @@
 <titleabbrev>EQL</titleabbrev>
 ++++
 
-beta::[]
-
 Event Query Language (EQL) is a query language for event-based time series
 data, such as logs, metrics, and traces.
 

diff --git a/docs/reference/eql/functions.asciidoc b/docs/reference/eql/functions.asciidoc
@@ -6,8 +6,6 @@
 <titleabbrev>Function reference</titleabbrev>
 ++++
 
-beta::[]
-
 {es} supports the following <<eql-functions,EQL functions>>. Most EQL functions
 are case-sensitive by default.
 

diff --git a/docs/reference/eql/get-async-eql-search-api.asciidoc b/docs/reference/eql/get-async-eql-search-api.asciidoc
@@ -7,8 +7,6 @@
 <titleabbrev>Get async EQL search</titleabbrev>
 ++++
 
-beta::[]
-
 Returns the current status and available results for an <<eql-search-async,async
 EQL search>> or a <<eql-search-store-sync-eql-search,stored synchronous EQL
 search>>.

diff --git a/docs/reference/eql/pipes.asciidoc b/docs/reference/eql/pipes.asciidoc
@@ -6,8 +6,6 @@
 <titleabbrev>Pipe reference</titleabbrev>
 ++++
 
-beta::[]
-
 {es} supports the following <<eql-pipes,EQL pipes>>.
 
 [discrete]

diff --git a/docs/reference/eql/syntax.asciidoc b/docs/reference/eql/syntax.asciidoc
@@ -6,8 +6,6 @@
 <titleabbrev>Syntax reference</titleabbrev>
 ++++
 
-beta::[]
-
 [discrete]
 [[eql-basic-syntax]]
 === Basic syntax

diff --git a/docs/reference/ingest/processors/uri-parts.asciidoc b/docs/reference/ingest/processors/uri-parts.asciidoc
@@ -3,7 +3,7 @@
 [[uri-parts-processor]]
 === URI parts processor
 ++++
-<titleabbrev>URI Parts</titleabbrev>
+<titleabbrev>URI parts</titleabbrev>
 ++++
 
 Parses a Uniform Resource Identifier (URI) string and extracts its components as

diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc
@@ -13,7 +13,7 @@ are stored and indexed. For instance, use mappings to define:
 * custom rules to control the mapping for
   <<dynamic-mapping,dynamically added fields>>.
 
-A mapping definition has:
+A mapping definition includes metadata fields and fields:
 
 <<mapping-fields,Metadata fields>>::
 
@@ -30,68 +30,34 @@ document. Each field has its own <<mapping-types, data type>>.
 NOTE: Before 7.0.0, the 'mappings' definition used to include a type name.
 For more details, please see <<removal-of-types>>.
 
-[[mapping-limit-settings]]
 [discrete]
-=== Settings to prevent mappings explosion
-
-Defining too many fields in an index can lead to a
-mapping explosion, which can cause out of memory errors and difficult
-situations to recover from.
+[[mapping-limit-settings]]
+== Settings to prevent mapping explosion
+Defining too many fields in an index can lead to a mapping explosion, which can
+cause out of memory errors and difficult situations to recover from.
 
 Consider a situation where every new document inserted
 introduces new fields, such as with <<dynamic-mapping,dynamic mapping>>.
 Each new field is added to the index mapping, which can become a
 problem as the mapping grows.
 
-Use the following settings to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion:
-
-`index.mapping.total_fields.limit`::
-    The maximum number of fields in an index. Field and object mappings, as well as
-    field aliases count towards this limit. The default value is `1000`.
-+
-[IMPORTANT]
-====
-The limit is in place to prevent mappings and searches from becoming too
-large. Higher values can lead to performance degradations and memory issues,
-especially in clusters with a high load or few resources.
-
-If you increase this setting, we recommend you also increase the
-<<search-settings,`indices.query.bool.max_clause_count`>> setting, which
-limits the maximum number of <<query-dsl-bool-query,boolean clauses>> in a query.
-====
-+
-[TIP]
-====
-If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type.
-====
-
-`index.mapping.depth.limit`::
-    The maximum depth for a field, which is measured as the number of inner
-    objects. For instance, if all fields are defined at the root object level,
-    then the depth is `1`. If there is one object mapping, then the depth is
-    `2`, etc. Default is `20`.
-
-// tag::nested-fields-limit[]
-`index.mapping.nested_fields.limit`::
-    The maximum number of distinct `nested` mappings in an index. The `nested` type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting
-    limits the number of unique `nested` types per index. Default is `50`.
-// end::nested-fields-limit[]
-
-// tag::nested-objects-limit[]
-`index.mapping.nested_objects.limit`::
-    The maximum number of nested JSON objects that a single document can contain across all
-    `nested` types. This limit helps to prevent out of memory errors when a document contains too many nested
-    objects. Default is `10000`.
-// end::nested-objects-limit[]
-
-`index.mapping.field_name_length.limit`::
-    Setting for the maximum length of a field name. This setting isn't really something that addresses
-    mappings explosion but might still be useful if you want to limit the field length.
-    It usually shouldn't be necessary to set this setting. The default is okay
-    unless a user starts to add a huge number of fields with really long names. Default is
-    `Long.MAX_VALUE` (no limit).
+Use the <<mapping-settings-limit,mapping limit settings>> to limit the number
+of field mappings (created manually or dynamically) and prevent documents from
+causing a mapping explosion.
+
+[discrete]
+[[runtime-fields]]
+== Runtime fields
+Typically, you index data into {es} to promote faster search. However, indexing
+can be slow and requires more disk space, and you have to reindex your data to
+add fields to existing documents.
+
+<<runtime,Runtime fields>> are not indexed, which saves disk space and makes
+data ingest faster. You can add runtime fields to existing documents without
+reindexing your data and calculate field values dynamically at search time.
 
 [discrete]
+[[dynamic-mapping-intro]]
 == Dynamic mapping
 
 Fields and mapping types do not need to be defined before being used. Thanks
@@ -114,7 +80,7 @@ You can create field mappings when you <<create-mapping,create an index>> and
 
 [discrete]
 [[create-mapping]]
-== Create an index with an explicit mapping
+=== Create an index with an explicit mapping
 
 You can use the <<indices-create-index,create index>> API to create a new index
 with an explicit mapping.
@@ -255,8 +221,12 @@ The API returns the following response:
 
 include::mapping/removal_of_types.asciidoc[]
 
+include::mapping/mapping-settings-limit.asciidoc[]
+
 include::mapping/types.asciidoc[]
 
+include::mapping/runtime.asciidoc[]
+
 include::mapping/fields.asciidoc[]
 
 include::mapping/params.asciidoc[]

diff --git a/docs/reference/mapping/mapping-settings-limit.asciidoc b/docs/reference/mapping/mapping-settings-limit.asciidoc
@@ -0,0 +1,49 @@
+[[mapping-settings-limit]]
+== Mapping limit settings
+Use the following settings to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion:
+
+`index.mapping.total_fields.limit`::
+    The maximum number of fields in an index. Field and object mappings, as well as
+    field aliases count towards this limit. The default value is `1000`.
++
+[IMPORTANT]
+====
+The limit is in place to prevent mappings and searches from becoming too
+large. Higher values can lead to performance degradations and memory issues,
+especially in clusters with a high load or few resources.
+
+If you increase this setting, we recommend you also increase the
+<<search-settings,`indices.query.bool.max_clause_count`>> setting, which
+limits the maximum number of <<query-dsl-bool-query,boolean clauses>> in a query.
+====
++
+[TIP]
+====
+If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type.
+====
+
+`index.mapping.depth.limit`::
+    The maximum depth for a field, which is measured as the number of inner
+    objects. For instance, if all fields are defined at the root object level,
+    then the depth is `1`. If there is one object mapping, then the depth is
+    `2`, etc. Default is `20`.
+
+// tag::nested-fields-limit[]
+`index.mapping.nested_fields.limit`::
+    The maximum number of distinct `nested` mappings in an index. The `nested` type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting
+    limits the number of unique `nested` types per index. Default is `50`.
+// end::nested-fields-limit[]
+
+// tag::nested-objects-limit[]
+`index.mapping.nested_objects.limit`::
+    The maximum number of nested JSON objects that a single document can contain across all
+    `nested` types. This limit helps to prevent out of memory errors when a document contains too many nested
+    objects. Default is `10000`.
+// end::nested-objects-limit[]
+
+`index.mapping.field_name_length.limit`::
+    Setting for the maximum length of a field name. This setting isn't really something that addresses
+    mappings explosion but might still be useful if you want to limit the field length.
+    It usually shouldn't be necessary to set this setting. The default is okay
+    unless a user starts to add a huge number of fields with really long names. Default is
+    `Long.MAX_VALUE` (no limit).