Skip to content

Commit

Permalink
ingest: document fields that support templating (#34536)
Browse files Browse the repository at this point in the history 
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
  • Loading branch information
@jakelandis
jakelandis committed Oct 24, 2018
1 parent e0ee802 commit e9a380c
Showing 1 changed file with 32 additions and 31 deletions.
63 changes: 32 additions & 31 deletions docs/reference/ingest/ingest-node.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -758,16 +758,16 @@ Accepts a single value or an array of values.
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be appended to
| `value` | yes | - | The value to be appended
| `field` | yes | - | The field to be appended to. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be appended. Supports <<accessing-template-fields,template snippets>>.
|======

[source,js]
--------------------------------------------------
{
"append": {
"field": "field1",
"value": ["item2", "item3", "item4"]
"field": "tags",
"value": ["production", "{{app}}", "{{owner}}"]
}
}
--------------------------------------------------
Expand All @@ -794,7 +794,7 @@ the field is not a supported format or resultant value exceeds 2^63.
--------------------------------------------------
{
"bytes": {
"field": "foo"
"field": "file.size"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -832,7 +832,7 @@ still be updated with the unconverted field value.
--------------------------------------------------
{
"convert": {
"field" : "foo",
"field" : "url.port",
"type": "integer"
}
}
Expand All @@ -856,8 +856,8 @@ in the same order they were defined as part of the processor definition.
| `field` | yes | - | The field to get the date from.
| `target_field` | no | @timestamp | The field that will hold the parsed date.
| `formats` | yes | - | An array of the expected date formats. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days.
| `timezone` | no | UTC | The timezone to use when parsing the date. Supports <<accessing-template-fields,template snippets>>.
| `locale` | no | ENGLISH | The locale to use when parsing the date, relevant when parsing month names or week days. Supports <<accessing-template-fields,template snippets>>.
|======

Here is an example that adds the parsed date to the `timestamp` field based on the `initial_date` field:
Expand Down Expand Up @@ -895,8 +895,8 @@ the timezone and locale values.
"field" : "initial_date",
"target_field" : "timestamp",
"formats" : ["ISO8601"],
"timezone" : "{{ my_timezone }}",
"locale" : "{{ my_locale }}"
"timezone" : "{{my_timezone}}",
"locale" : "{{my_locale}}"
}
}
]
Expand Down Expand Up @@ -1041,12 +1041,12 @@ understands this to mean `2016-04-01` as is explained in the <<date-math-index-n
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to get the date or timestamp from.
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second).
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date. Supports <<accessing-template-fields,template snippets>>.
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). Supports <<accessing-template-fields,template snippets>>.
| `date_formats` | no | yyyy-MM-dd'T'HH:mm:ss.SSSZ | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a Joda pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
| `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
| `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here.
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. An valid Joda pattern is expected here. Supports <<accessing-template-fields,template snippets>>.
|======

[[dissect-processor]]
Expand Down Expand Up @@ -1372,14 +1372,15 @@ to the requester.
[options="header"]
|======
| Name | Required | Default | Description
| `message` | yes | - | The error message of the `FailException` thrown by the processor
| `message` | yes | - | The error message thrown by the processor. Supports <<accessing-template-fields,template snippets>>.
|======

[source,js]
--------------------------------------------------
{
"fail": {
"message": "an error message"
"if" : "ctx.tags.contains('production') != true",
"message": "The production tag is not present, found tags: {{tags}}"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2084,7 +2085,7 @@ Removes existing fields. If one field doesn't exist, an exception will be thrown
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | Fields to be removed
| `field` | yes | - | Fields to be removed. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist or is `null`, the processor quietly exits without modifying the document
|======

Expand All @@ -2094,7 +2095,7 @@ Here is an example to remove a single field:
--------------------------------------------------
{
"remove": {
"field": "foo"
"field": "user_agent"
}
}
--------------------------------------------------
Expand All @@ -2106,7 +2107,7 @@ To remove multiple fields, you can use the following query:
--------------------------------------------------
{
"remove": {
"field": ["foo", "bar"]
"field": ["user_agent", "url"]
}
}
--------------------------------------------------
Expand All @@ -2120,18 +2121,18 @@ Renames an existing field. If the field doesn't exist or the new name is already
.Rename Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed
| `target_field` | yes | - | The new name of the field
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
| Name | Required | Default | Description
| `field` | yes | - | The field to be renamed. Supports <<accessing-template-fields,template snippets>>.
| `target_field` | yes | - | The new name of the field. Supports <<accessing-template-fields,template snippets>>.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
|======

[source,js]
--------------------------------------------------
{
"rename": {
"field": "foo",
"target_field": "foobar"
"field": "provider",
"target_field": "cloud.provider"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2249,18 +2250,18 @@ its value will be replaced with the provided one.
.Set Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update
| `value` | yes | - | The value to be set for the field
| `override`| no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
| Name | Required | Default | Description
| `field` | yes | - | The field to insert, upsert, or update. Supports <<accessing-template-fields,template snippets>>.
| `value` | yes | - | The value to be set for the field. Supports <<accessing-template-fields,template snippets>>.
| `override` | no | true | If processor will update fields with pre-existing non-null-valued field. When set to `false`, such fields will not be touched.
|======

[source,js]
--------------------------------------------------
{
"set": {
"field": "field1",
"value": 582.1
"field": "host.os.name",
"value": "{{os}}"
}
}
--------------------------------------------------
Expand Down Expand Up @@ -2313,7 +2314,7 @@ Throws an error when the field is not an array.
--------------------------------------------------
{
"sort": {
"field": "field_to_sort",
"field": "array_field_to_sort",
"order": "desc"
}
}
Expand Down

0 comments on commit e9a380c

Please sign in to comment.