Skip to content

Commit

Permalink
[DOCS] Add dynamic runtime fields to docs (#66194)
Browse files Browse the repository at this point in the history
* [DOCS] Add dynamic runtime fields to docs.

* Clarifying edits and example changes.

* Creating better table and incorporating review comments.

* Change numeral to superscript.
  • Loading branch information
Adam Locke authored Dec 14, 2020
1 parent 9bb6a3a commit fe54c2f
Show file tree
Hide file tree
Showing 3 changed files with 85 additions and 69 deletions.
61 changes: 31 additions & 30 deletions docs/reference/mapping/dynamic/field-mapping.asciidoc
Original file line number Diff line number Diff line change
@@ -1,35 +1,37 @@
[[dynamic-field-mapping]]
=== Dynamic field mapping

By default, when a previously unseen field is found in a document,
Elasticsearch will add the new field to the type mapping. This behaviour can
be disabled, both at the document and at the <<object,`object`>> level, by
setting the <<dynamic,`dynamic`>> parameter to `false` (to ignore new fields) or to `strict` (to throw
an exception if an unknown field is encountered).

Assuming `dynamic` field mapping is enabled, some simple rules are used to
determine which data type the field should have:

[horizontal]
*JSON data type*:: *Elasticsearch data type*

`null`:: No field is added.
`true` or `false`:: <<boolean,`boolean`>> field
floating{nbsp}point{nbsp}number:: <<number,`float`>> field
integer:: <<number,`long`>> field
object:: <<object,`object`>> field
array:: Depends on the first non-`null` value in the array.
string:: Either a <<date,`date`>> field
(if the value passes <<date-detection,date detection>>),
a <<number,`double`>> or <<number,`long`>> field
(if the value passes <<numeric-detection,numeric detection>>)
or a <<text,`text`>> field, with a <<keyword,`keyword`>> sub-field.

These are the only <<mapping-types,field data types>> that are dynamically
detected. All other data types must be mapped explicitly.

Besides the options listed below, dynamic field mapping rules can be further
customised with <<dynamic-templates,`dynamic_templates`>>.
When {es} detects a new field in a document, it _dynamically_ adds the field to
the type mapping. The <<dynamic,`dynamic`>> parameter controls this behavior.

You can disable dynamic mapping, both at the document and at the
<<object,`object`>> level. Setting the `dynamic` parameter to
`false` ignores new fields, and `strict` rejects the document if {es}
encounters an unknown field.

When dynamic field mapping is enabled, {es} uses the following rules to
determine how to map data types for each field.

NOTE: These are the only <<mapping-types,field data types>> that {es} detects
dynamically. All other data types must be mapped explicitly.

[cols="3"]
|===
h| JSON data type h| `dynamic:true` h| `dynamic:runtime`
|`null` 2*| No field added
|`true` or `false` 2*| `boolean`
|floating point number | `float` | `double`
|`integer` 2*| `long`
|`object`<<dynamic-object-footnote,^1^>> 2*| `object`
|`array` 2*| Depends on the first non-`null` value in the array
|`string` that passes <<date-detection,date detection>> 2*| `date`
|`string` that passes <<numeric-detection,numeric detection>> | `double` or `long` | `double`
|`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword`
3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | |
|===

Besides the following options, you can customize dynamic field mapping rules
with <<dynamic-templates,`dynamic_templates`>>.

[[date-detection]]
==== Date detection
Expand Down Expand Up @@ -129,4 +131,3 @@ PUT my-index-000001/_doc/1

<1> The `my_float` field is added as a <<number,`float`>> field.
<2> The `my_integer` field is added as a <<number,`long`>> field.

84 changes: 46 additions & 38 deletions docs/reference/mapping/params/dynamic.asciidoc
Original file line number Diff line number Diff line change
@@ -1,24 +1,31 @@
[[dynamic]]
=== `dynamic`

By default, fields can be added _dynamically_ to a document, or to
<<object,inner objects>> within a document, just by indexing a document
containing the new field. For instance:
When you index a document containing a new field, {es} <<dynamic-mapping,adds the field dynamically>> to a document or to inner objects within a document. The
following document adds the string field `username`, the object field
`name`, and two string fields under the `name` object:

[source,console]
--------------------------------------------------
PUT my-index-000001/_doc/1 <1>
----
PUT my-index-000001/_doc/1
{
"username": "johnsmith",
"name": {
"name": { <1>
"first": "John",
"last": "Smith"
}
}
GET my-index-000001/_mapping <2>
----
<1> Refer to fields under the `name` object as `name.first` and `name.last`.
<2> Check the mapping to view changes.

The following document adds two string fields: `email` and `name.middle`:

PUT my-index-000001/_doc/2 <3>
[source,console]
----
PUT my-index-000001/_doc/2
{
"username": "marywhite",
"email": "[email protected]",
Expand All @@ -29,35 +36,24 @@ PUT my-index-000001/_doc/2 <3>
}
}
GET my-index-000001/_mapping <4>
--------------------------------------------------

<1> This document introduces the string field `username`, the object field
`name`, and two string fields under the `name` object which can be
referred to as `name.first` and `name.last`.
<2> Check the mapping to verify the above.
<3> This document adds two string fields: `email` and `name.middle`.
<4> Check the mapping to verify the changes.
GET my-index-000001/_mapping
----

The details of how new fields are detected and added to the mapping is explained in <<dynamic-mapping>>.
TIP: Use the <<indices-put-mapping,PUT mapping API>> to update the `dynamic`
setting on existing fields.

The `dynamic` setting controls whether new fields can be added dynamically or
not. It accepts three settings:
[[dynamic-inner-objects]]
==== Setting `dynamic` on inner objects
<<object,Inner objects>> inherit the `dynamic` setting from their parent
object or from the mapping type. In the following example, dynamic mapping is
disabled at the type level, so no new top-level fields will be added
dynamically.

[horizontal]
`true`:: Newly detected fields are added to the mapping. (default)
`false`:: Newly detected fields are ignored. These fields will not be indexed so will not be searchable
but will still appear in the `_source` field of returned hits. These fields will not be added
to the mapping, new fields must be added explicitly.
`strict`:: If new fields are detected, an exception is thrown and the document is rejected. New fields
must be explicitly added to the mapping.

The `dynamic` setting may be set at the mapping type level, and on each
<<object,inner object>>. Inner objects inherit the setting from their parent
object or from the mapping type. For instance:
However, the `user.social_networks` object enables dynamic mapping, so you can
add fields to this inner object.

[source,console]
--------------------------------------------------
----
PUT my-index-000001
{
"mappings": {
Expand All @@ -68,20 +64,32 @@ PUT my-index-000001
"name": {
"type": "text"
},
"social_networks": { <3>
"dynamic": true,
"social_networks": {
"dynamic": true, <3>
"properties": {}
}
}
}
}
}
}
--------------------------------------------------
----

<1> Dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
<1> Disables dynamic mapping at the type level.
<2> The `user` object inherits the type-level setting.
<3> The `user.social_networks` object enables dynamic mapping, so new fields may be added to this inner object.
<3> Enables dynamic mapping for this inner object.

TIP: The `dynamic` setting can be updated on existing fields
using the <<indices-put-mapping,PUT mapping API>>.
[[dynamic-parameters]]
==== Parameters for `dynamic`
The `dynamic` parameter controls whether new fields are added dynamically, and
accepts the following parameters:

[horizontal]
`true`:: New fields are added to the mapping (default).
`runtime`:: New fields are added to the mapping as <<runtime,runtime fields>>.
These fields are not indexed, and are loaded from `_source` at query time.
`false`:: New fields are ignored. These fields will not be indexed
or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added
to the mapping, and new fields must be added explicitly.
`strict`:: If new fields are detected, an exception is thrown and the document
is rejected. New fields must be explicitly added to the mapping.
9 changes: 8 additions & 1 deletion docs/reference/mapping/runtime.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ add fields to existing documents. With _runtime fields_, you can add
fields to documents already indexed to {es} without reindexing your data.

You access runtime fields from the search API like any other field, and {es}
sees runtime fields no differently.
sees runtime fields no differently. You can define runtime fields in the
<<runtime-mapping-fields,index mapping>> or in the
<<runtime-search-request,search request>>. Your choice, which is part of the
inherent flexibility of runtime fields.

[discrete]
[[runtime-benefits]]
Expand Down Expand Up @@ -83,6 +86,10 @@ script. If you define a runtime field without a script, {es} evaluates the
field at search time, looks at each document containing that field, retrieves
the `_source`, and returns a value if one exists.

If <<dynamic-field-mapping,dynamic field mapping>> is enabled where the
`dynamic` parameter is set to `runtime`, new fields are automatically added to
the index mapping as runtime fields.

Runtime fields are similar to the <<script-fields,`script_fields`>> parameter
of the `_search` request, but also make the script results available anywhere
in a search request.
Expand Down

0 comments on commit fe54c2f

Please sign in to comment.