Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Add dynamic runtime fields to docs (#66194) #66304

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 31 additions & 30 deletions docs/reference/mapping/dynamic/field-mapping.asciidoc
Original file line number Diff line number Diff line change
@@ -1,35 +1,37 @@
[[dynamic-field-mapping]]
=== Dynamic field mapping

By default, when a previously unseen field is found in a document,
Elasticsearch will add the new field to the type mapping. This behaviour can
be disabled, both at the document and at the <<object,`object`>> level, by
setting the <<dynamic,`dynamic`>> parameter to `false` (to ignore new fields) or to `strict` (to throw
an exception if an unknown field is encountered).

Assuming `dynamic` field mapping is enabled, some simple rules are used to
determine which data type the field should have:

[horizontal]
*JSON data type*:: *Elasticsearch data type*

`null`:: No field is added.
`true` or `false`:: <<boolean,`boolean`>> field
floating{nbsp}point{nbsp}number:: <<number,`float`>> field
integer:: <<number,`long`>> field
object:: <<object,`object`>> field
array:: Depends on the first non-`null` value in the array.
string:: Either a <<date,`date`>> field
(if the value passes <<date-detection,date detection>>),
a <<number,`double`>> or <<number,`long`>> field
(if the value passes <<numeric-detection,numeric detection>>)
or a <<text,`text`>> field, with a <<keyword,`keyword`>> sub-field.

These are the only <<mapping-types,field data types>> that are dynamically
detected. All other data types must be mapped explicitly.

Besides the options listed below, dynamic field mapping rules can be further
customised with <<dynamic-templates,`dynamic_templates`>>.
When {es} detects a new field in a document, it _dynamically_ adds the field to
the type mapping. The <<dynamic,`dynamic`>> parameter controls this behavior.

You can disable dynamic mapping, both at the document and at the
<<object,`object`>> level. Setting the `dynamic` parameter to
`false` ignores new fields, and `strict` rejects the document if {es}
encounters an unknown field.

When dynamic field mapping is enabled, {es} uses the following rules to
determine how to map data types for each field.

NOTE: These are the only <<mapping-types,field data types>> that {es} detects
dynamically. All other data types must be mapped explicitly.

[cols="3"]
|===
h| JSON data type h| `dynamic:true` h| `dynamic:runtime`
|`null` 2*| No field added
|`true` or `false` 2*| `boolean`
|floating point number | `float` | `double`
|`integer` 2*| `long`
|`object`<<dynamic-object-footnote,^1^>> 2*| `object`
|`array` 2*| Depends on the first non-`null` value in the array
|`string` that passes <<date-detection,date detection>> 2*| `date`
|`string` that passes <<numeric-detection,numeric detection>> | `double` or `long` | `double`
|`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword`
3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | |
|===

Besides the following options, you can customize dynamic field mapping rules
with <<dynamic-templates,`dynamic_templates`>>.

[[date-detection]]
==== Date detection
Expand Down Expand Up @@ -129,4 +131,3 @@ PUT my-index-000001/_doc/1

<1> The `my_float` field is added as a <<number,`float`>> field.
<2> The `my_integer` field is added as a <<number,`long`>> field.

84 changes: 46 additions & 38 deletions docs/reference/mapping/params/dynamic.asciidoc
Original file line number Diff line number Diff line change
@@ -1,24 +1,31 @@
[[dynamic]]
=== `dynamic`

By default, fields can be added _dynamically_ to a document, or to
<<object,inner objects>> within a document, just by indexing a document
containing the new field. For instance:
When you index a document containing a new field, {es} <<dynamic-mapping,adds the field dynamically>> to a document or to inner objects within a document. The
following document adds the string field `username`, the object field
`name`, and two string fields under the `name` object:

[source,console]
--------------------------------------------------
PUT my-index-000001/_doc/1 <1>
----
PUT my-index-000001/_doc/1
{
"username": "johnsmith",
"name": {
"name": { <1>
"first": "John",
"last": "Smith"
}
}

GET my-index-000001/_mapping <2>
----
<1> Refer to fields under the `name` object as `name.first` and `name.last`.
<2> Check the mapping to view changes.

The following document adds two string fields: `email` and `name.middle`:

PUT my-index-000001/_doc/2 <3>
[source,console]
----
PUT my-index-000001/_doc/2
{
"username": "marywhite",
"email": "[email protected]",
Expand All @@ -29,35 +36,24 @@ PUT my-index-000001/_doc/2 <3>
}
}

GET my-index-000001/_mapping <4>
--------------------------------------------------

<1> This document introduces the string field `username`, the object field
`name`, and two string fields under the `name` object which can be
referred to as `name.first` and `name.last`.
<2> Check the mapping to verify the above.
<3> This document adds two string fields: `email` and `name.middle`.
<4> Check the mapping to verify the changes.
GET my-index-000001/_mapping
----

The details of how new fields are detected and added to the mapping is explained in <<dynamic-mapping>>.
TIP: Use the <<indices-put-mapping,PUT mapping API>> to update the `dynamic`
setting on existing fields.

The `dynamic` setting controls whether new fields can be added dynamically or
not. It accepts three settings:
[[dynamic-inner-objects]]
==== Setting `dynamic` on inner objects
<<object,Inner objects>> inherit the `dynamic` setting from their parent
object or from the mapping type. In the following example, dynamic mapping is
disabled at the type level, so no new top-level fields will be added
dynamically.

[horizontal]
`true`:: Newly detected fields are added to the mapping. (default)
`false`:: Newly detected fields are ignored. These fields will not be indexed so will not be searchable
but will still appear in the `_source` field of returned hits. These fields will not be added
to the mapping, new fields must be added explicitly.
`strict`:: If new fields are detected, an exception is thrown and the document is rejected. New fields
must be explicitly added to the mapping.

The `dynamic` setting may be set at the mapping type level, and on each
<<object,inner object>>. Inner objects inherit the setting from their parent
object or from the mapping type. For instance:
However, the `user.social_networks` object enables dynamic mapping, so you can
add fields to this inner object.

[source,console]
--------------------------------------------------
----
PUT my-index-000001
{
"mappings": {
Expand All @@ -68,20 +64,32 @@ PUT my-index-000001
"name": {
"type": "text"
},
"social_networks": { <3>
"dynamic": true,
"social_networks": {
"dynamic": true, <3>
"properties": {}
}
}
}
}
}
}
--------------------------------------------------
----

<1> Dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
<1> Disables dynamic mapping at the type level.
<2> The `user` object inherits the type-level setting.
<3> The `user.social_networks` object enables dynamic mapping, so new fields may be added to this inner object.
<3> Enables dynamic mapping for this inner object.

TIP: The `dynamic` setting can be updated on existing fields
using the <<indices-put-mapping,PUT mapping API>>.
[[dynamic-parameters]]
==== Parameters for `dynamic`
The `dynamic` parameter controls whether new fields are added dynamically, and
accepts the following parameters:

[horizontal]
`true`:: New fields are added to the mapping (default).
`runtime`:: New fields are added to the mapping as <<runtime,runtime fields>>.
These fields are not indexed, and are loaded from `_source` at query time.
`false`:: New fields are ignored. These fields will not be indexed
or searchable, but will still appear in the `_source` field of returned hits. These fields will not be added
to the mapping, and new fields must be added explicitly.
`strict`:: If new fields are detected, an exception is thrown and the document
is rejected. New fields must be explicitly added to the mapping.
9 changes: 8 additions & 1 deletion docs/reference/mapping/runtime.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ add fields to existing documents. With _runtime fields_, you can add
fields to documents already indexed to {es} without reindexing your data.

You access runtime fields from the search API like any other field, and {es}
sees runtime fields no differently.
sees runtime fields no differently. You can define runtime fields in the
<<runtime-mapping-fields,index mapping>> or in the
<<runtime-search-request,search request>>. Your choice, which is part of the
inherent flexibility of runtime fields.

[discrete]
[[runtime-benefits]]
Expand Down Expand Up @@ -83,6 +86,10 @@ script. If you define a runtime field without a script, {es} evaluates the
field at search time, looks at each document containing that field, retrieves
the `_source`, and returns a value if one exists.

If <<dynamic-field-mapping,dynamic field mapping>> is enabled where the
`dynamic` parameter is set to `runtime`, new fields are automatically added to
the index mapping as runtime fields.

Runtime fields are similar to the <<script-fields,`script_fields`>> parameter
of the `_search` request, but also make the script results available anywhere
in a search request.
Expand Down