From 64cc39061affb9c577ec0c32a56cb2f234aab5b9 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Thu, 10 Dec 2020 17:50:01 -0500 Subject: [PATCH 1/4] [DOCS] Add dynamic runtime fields to docs. --- .../mapping/dynamic/field-mapping.asciidoc | 38 +++++++++---------- .../reference/mapping/params/dynamic.asciidoc | 18 +++++---- docs/reference/mapping/runtime.asciidoc | 9 ++++- 3 files changed, 37 insertions(+), 28 deletions(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index c1652baf2d2eb..bf066d22096aa 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -1,35 +1,36 @@ [[dynamic-field-mapping]] === Dynamic field mapping -By default, when a previously unseen field is found in a document, -Elasticsearch will add the new field to the type mapping. This behaviour can -be disabled, both at the document and at the <> level, by -setting the <> parameter to `false` (to ignore new fields) or to `strict` (to throw -an exception if an unknown field is encountered). +When {es} detects a new field in a document, it automatically adds the field to +the type mapping. The <> parameter controls whether new +fields are added dynamically. -Assuming `dynamic` field mapping is enabled, some simple rules are used to -determine which data type the field should have: +You can disable this behavior, both at the document and at the +<> level. Setting the `dynamic` parameter to +`false` ignores new fields, and `strict` throws an exception if {es} encounters +an unknown field. + +When dynamic field mapping is enabled, {es} uses the following rules to +determine how to map data types for each field. [horizontal] -*JSON data type*:: *Elasticsearch data type* +*JSON data type*:: *Elasticsearch data type* `null`:: No field is added. `true` or `false`:: <> field -floating{nbsp}point{nbsp}number:: <> field +floating{nbsp}point{nbsp}number:: <> indexed field or <> runtime field integer:: <> field object:: <> field ++ +Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. array:: Depends on the first non-`null` value in the array. -string:: Either a <> field - (if the value passes <>), - a <> or <> field - (if the value passes <>) - or a <> field, with a <> sub-field. +string:: Either a <> field (if the value passes <>), a <> or <> field (if the value passes <>), or a <> field with a <> sub-field. -These are the only <> that are dynamically -detected. All other data types must be mapped explicitly. +NOTE: These are the only <> that {es} detects +dynamically. All other data types must be mapped explicitly. -Besides the options listed below, dynamic field mapping rules can be further -customised with <>. +Besides the following options, you can customize dynamic field mapping rules +with <>. [[date-detection]] ==== Date detection @@ -129,4 +130,3 @@ PUT my-index-000001/_doc/1 <1> The `my_float` field is added as a <> field. <2> The `my_integer` field is added as a <> field. - diff --git a/docs/reference/mapping/params/dynamic.asciidoc b/docs/reference/mapping/params/dynamic.asciidoc index 2eddb010d20cd..26f672079f090 100644 --- a/docs/reference/mapping/params/dynamic.asciidoc +++ b/docs/reference/mapping/params/dynamic.asciidoc @@ -41,16 +41,18 @@ GET my-index-000001/_mapping <4> The details of how new fields are detected and added to the mapping is explained in <>. -The `dynamic` setting controls whether new fields can be added dynamically or -not. It accepts three settings: +The `dynamic` parameter controls whether new fields are added dynamically, and +accepts the following settings: [horizontal] -`true`:: Newly detected fields are added to the mapping. (default) -`false`:: Newly detected fields are ignored. These fields will not be indexed so will not be searchable - but will still appear in the `_source` field of returned hits. These fields will not be added - to the mapping, new fields must be added explicitly. -`strict`:: If new fields are detected, an exception is thrown and the document is rejected. New fields - must be explicitly added to the mapping. +`true`:: New fields are added to the mapping (default). +`runtime`:: New fields are added as <>. These fields + are not indexed, and are loaded from `_source` at query time. +`false`:: New fields are ignored. These fields will not be indexed + or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added + to the mapping, and new fields must be added explicitly. +`strict`:: If new fields are detected, an exception is thrown and the document + is rejected. New fields must be explicitly added to the mapping. The `dynamic` setting may be set at the mapping type level, and on each <>. Inner objects inherit the setting from their parent diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index f53ebb9e3edc4..f1dd3920ec942 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -6,7 +6,10 @@ add fields to existing documents. With _runtime fields_, you can add fields to documents already indexed to {es} without reindexing your data. You access runtime fields from the search API like any other field, and {es} -sees runtime fields no differently. +sees runtime fields no differently. You can define runtime fields in the +<> or in the +<>. Your choice, which is part of the +inherent flexibility of runtime fields. [discrete] [[runtime-benefits]] @@ -83,6 +86,10 @@ script. If you define a runtime field without a script, {es} evaluates the field at search time, looks at each document containing that field, retrieves the `_source`, and returns a value if one exists. +If <> is enabled where the +`dynamic` parameter is set to `runtime`, new fields are automatically added to +the index mapping as runtime fields. + Runtime fields are similar to the <> parameter of the `_search` request, but also make the script results available anywhere in a search request. From 00d9880582f353d17fd96fe6cb2049981e720cfe Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Mon, 14 Dec 2020 09:05:11 -0500 Subject: [PATCH 2/4] Clarifying edits and example changes. --- .../mapping/dynamic/field-mapping.asciidoc | 10 +-- .../reference/mapping/params/dynamic.asciidoc | 90 ++++++++++--------- 2 files changed, 55 insertions(+), 45 deletions(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index bf066d22096aa..5514853b5261c 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -1,11 +1,10 @@ [[dynamic-field-mapping]] === Dynamic field mapping -When {es} detects a new field in a document, it automatically adds the field to -the type mapping. The <> parameter controls whether new -fields are added dynamically. +When {es} detects a new field in a document, it _dynamically_ adds the field to +the type mapping. The <> parameter controls this behavior. -You can disable this behavior, both at the document and at the +You can disable dynamic mapping, both at the document and at the <> level. Setting the `dynamic` parameter to `false` ignores new fields, and `strict` throws an exception if {es} encounters an unknown field. @@ -22,7 +21,8 @@ floating{nbsp}point{nbsp}number:: <> indexed field or <> field object:: <> field + -Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. +NOTE: Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. + array:: Depends on the first non-`null` value in the array. string:: Either a <> field (if the value passes <>), a <> or <> field (if the value passes <>), or a <> field with a <> sub-field. diff --git a/docs/reference/mapping/params/dynamic.asciidoc b/docs/reference/mapping/params/dynamic.asciidoc index 26f672079f090..468dd5620c3ab 100644 --- a/docs/reference/mapping/params/dynamic.asciidoc +++ b/docs/reference/mapping/params/dynamic.asciidoc @@ -1,24 +1,32 @@ [[dynamic]] === `dynamic` -By default, fields can be added _dynamically_ to a document, or to -<> within a document, just by indexing a document -containing the new field. For instance: +When you index a document containing a new field, {es} adds the +field _dynamically_ to a document or to inner objects within a document. The +following document adds the string field `username`, the object field +`name`, and two string fields under the `name` object: [source,console] --------------------------------------------------- -PUT my-index-000001/_doc/1 <1> +---- +PUT my-index-000001/_doc/1 { "username": "johnsmith", - "name": { + "name": { <1> "first": "John", "last": "Smith" } } GET my-index-000001/_mapping <2> +---- +<1> Refer to fields under the `name` object as `name.first` and `name.last`. +<2> Check the mapping to view changes. + +The following document adds two string fields: `email` and `name.middle`: -PUT my-index-000001/_doc/2 <3> +[source,console] +---- +PUT my-index-000001/_doc/2 { "username": "marywhite", "email": "mary@white.com", @@ -29,37 +37,24 @@ PUT my-index-000001/_doc/2 <3> } } -GET my-index-000001/_mapping <4> --------------------------------------------------- +GET my-index-000001/_mapping +---- -<1> This document introduces the string field `username`, the object field - `name`, and two string fields under the `name` object which can be - referred to as `name.first` and `name.last`. -<2> Check the mapping to verify the above. -<3> This document adds two string fields: `email` and `name.middle`. -<4> Check the mapping to verify the changes. +TIP: Use the <> to update the `dynamic` +setting on existing fields. -The details of how new fields are detected and added to the mapping is explained in <>. +[[dynamic-inner-objects]] +==== Setting `dynamic` on inner objects +<> inherit the `dynamic` setting from their parent +object or from the mapping type. In the following example, dynamic mapping is +disabled at the type level, so no new top-level fields will be added +dynamically. -The `dynamic` parameter controls whether new fields are added dynamically, and -accepts the following settings: - -[horizontal] -`true`:: New fields are added to the mapping (default). -`runtime`:: New fields are added as <>. These fields - are not indexed, and are loaded from `_source` at query time. -`false`:: New fields are ignored. These fields will not be indexed - or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added - to the mapping, and new fields must be added explicitly. -`strict`:: If new fields are detected, an exception is thrown and the document - is rejected. New fields must be explicitly added to the mapping. - -The `dynamic` setting may be set at the mapping type level, and on each -<>. Inner objects inherit the setting from their parent -object or from the mapping type. For instance: +However, the `user.social_networks` object enables dynamic mapping, so you can +add fields to this inner object. [source,console] --------------------------------------------------- +---- PUT my-index-000001 { "mappings": { @@ -70,8 +65,8 @@ PUT my-index-000001 "name": { "type": "text" }, - "social_networks": { <3> - "dynamic": true, + "social_networks": { + "dynamic": true, <3> "properties": {} } } @@ -79,11 +74,26 @@ PUT my-index-000001 } } } --------------------------------------------------- +---- -<1> Dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically. +<1> Disables dynamic mapping at the type level. <2> The `user` object inherits the type-level setting. -<3> The `user.social_networks` object enables dynamic mapping, so new fields may be added to this inner object. +<3> Enables dynamic mapping for this inner object. + +[[dynamic-parameters]] +==== Parameters for `dynamic` +The `dynamic` parameter controls whether new fields are added dynamically, and +accepts the following parameters: + +[horizontal] +`true`:: New fields are added to the mapping (default). +`runtime`:: New fields are added to the mapping as <>. + These fields are not indexed, and are loaded from `_source` at query time. +`false`:: New fields are ignored. These fields will not be indexed + or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added + to the mapping, and new fields must be added explicitly. +`strict`:: If new fields are detected, an exception is thrown and the document + is rejected. New fields must be explicitly added to the mapping. -TIP: The `dynamic` setting can be updated on existing fields -using the <>. +See <> for details of how new fields are detected and added to +the mapping. From 217aaee1c46ac08d078aec0db6b68062b74e10ca Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Mon, 14 Dec 2020 15:25:18 -0500 Subject: [PATCH 3/4] Creating better table and incorporating review comments. --- .../mapping/dynamic/field-mapping.asciidoc | 33 ++++++++++--------- .../reference/mapping/params/dynamic.asciidoc | 6 +--- 2 files changed, 18 insertions(+), 21 deletions(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index 5514853b5261c..0e8cc6664734e 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -6,29 +6,30 @@ the type mapping. The <> parameter controls this behavior. You can disable dynamic mapping, both at the document and at the <> level. Setting the `dynamic` parameter to -`false` ignores new fields, and `strict` throws an exception if {es} encounters -an unknown field. +`false` ignores new fields, and `strict` rejects the document if {es} +encounters an unknown field. When dynamic field mapping is enabled, {es} uses the following rules to determine how to map data types for each field. -[horizontal] -*JSON data type*:: *Elasticsearch data type* - -`null`:: No field is added. -`true` or `false`:: <> field -floating{nbsp}point{nbsp}number:: <> indexed field or <> runtime field -integer:: <> field -object:: <> field -+ -NOTE: Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. - -array:: Depends on the first non-`null` value in the array. -string:: Either a <> field (if the value passes <>), a <> or <> field (if the value passes <>), or a <> field with a <> sub-field. - NOTE: These are the only <> that {es} detects dynamically. All other data types must be mapped explicitly. +[cols="3"] +|=== +h| JSON data type h| `dynamic:true` h| `dynamic:runtime` + |`null` 2*| No field added + |`true` or `false` 2*| `boolean` + |floating point number | `float` | `double` + |`integer` 2*| `long` + |`object`<> 2*| `object` + |`array` 2*| Depends on the first non-`null` value in the array + |`string` that passes <> 2*| `date` + |`string` that passes <> | `double` or `long` | `double` + |`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword` +3+| [[dynamic-object-footnote]]1. Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | +|=== + Besides the following options, you can customize dynamic field mapping rules with <>. diff --git a/docs/reference/mapping/params/dynamic.asciidoc b/docs/reference/mapping/params/dynamic.asciidoc index 468dd5620c3ab..6ded8e4f7809c 100644 --- a/docs/reference/mapping/params/dynamic.asciidoc +++ b/docs/reference/mapping/params/dynamic.asciidoc @@ -1,8 +1,7 @@ [[dynamic]] === `dynamic` -When you index a document containing a new field, {es} adds the -field _dynamically_ to a document or to inner objects within a document. The +When you index a document containing a new field, {es} <> to a document or to inner objects within a document. The following document adds the string field `username`, the object field `name`, and two string fields under the `name` object: @@ -94,6 +93,3 @@ accepts the following parameters: to the mapping, and new fields must be added explicitly. `strict`:: If new fields are detected, an exception is thrown and the document is rejected. New fields must be explicitly added to the mapping. - -See <> for details of how new fields are detected and added to -the mapping. From 1fa1d8fd052aa00e7196deca0d7279954d9bc6ae Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Mon, 14 Dec 2020 16:06:08 -0500 Subject: [PATCH 4/4] Change numeral to superscript. --- docs/reference/mapping/dynamic/field-mapping.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index 0e8cc6664734e..bf5a9165d6f28 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -27,7 +27,7 @@ h| JSON data type h| `dynamic:true` h| `dynamic:runtime` |`string` that passes <> 2*| `date` |`string` that passes <> | `double` or `long` | `double` |`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword` -3+| [[dynamic-object-footnote]]1. Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | +3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | |=== Besides the following options, you can customize dynamic field mapping rules