Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit multiple fields from a runtime field script (#75108) #76287

Merged
merged 1 commit into from
Aug 10, 2021

Conversation

romseygeek
Copy link
Contributor

We have recently introduced support for grok and dissect to the runtime fields
Painless context that allows to split a field into multiple fields. However, each runtime
field can only emit values for a single field. This commit introduces support for emitting
multiple fields from the same script.

The API call to define a runtime field that emits multiple fields is the following:

PUT localhost:9200/logs/_mappings
{
    "runtime" : {
      "log" : {
        "type" : "composite",
        "script" : "emit(grok(\"%{COMMONAPACHELOG}\").extract(doc[\"message.keyword\"].value))",
        "fields" : {
            "clientip" : {
                "type" : "ip"
            },
            "response" : {
                "type" : "long"
            }
        }
      }
    }
}

The script context for this new field type accepts two emit signatures:

  • emit(String, Object)
  • emit(Map)

Sub-fields need to be declared under fields in order to be discoverable through
the field_caps API and accessible through the search API.

The way that it emits multiple fields is by returning multiple MappedFieldTypes
from RuntimeField#asMappedFieldTypes. The sub-fields are instances of the
runtime fields that are already supported, with a little tweak to adapt the script
defined by their parent to an artificial script factory for each of the sub-fields
that makes its corresponding sub-field accessible. This approach allows to reuse
all of the existing runtime fields code for the sub-fields.

The runtime section has been flat so far as it has not supported objects until now.
That stays the same, meaning that runtime fields can have dots in their names.
Because there are though two ways to create the same field with the introduction
of the ability to emit multiple fields, we have to make sure that a runtime field with
a certain name cannot be defined twice, which is why the following mappings are
rejected with the error Found two runtime fields with same name [log.response]:

PUT localhost:9200/logs/_mappings
{
    "runtime" : {
        "log.response" : {
            "type" : "keyword"
        },
        "log" : {
            "type" : "composite",
            "script" : "emit(\"response\", grok(\"%{COMMONAPACHELOG}\").extract(doc[\"message.keyword\"].value)?.response)",
            "fields" : {
                "response" : {
                    "type" : "long"
                }
            }
        }
    }
}

Closes #68203

We have recently introduced support for grok and dissect to the runtime fields
Painless context that allows to split a field into multiple fields. However, each runtime
field can only emit values for a single field. This commit introduces support for emitting
multiple fields from the same script.

The API call to define a runtime field that emits multiple fields is the following:

```
PUT localhost:9200/logs/_mappings
{
    "runtime" : {
      "log" : {
        "type" : "composite",
        "script" : "emit(grok(\"%{COMMONAPACHELOG}\").extract(doc[\"message.keyword\"].value))",
        "fields" : {
            "clientip" : {
                "type" : "ip"
            },
            "response" : {
                "type" : "long"
            }
        }
      }
    }
}
```

The script context for this new field type accepts two emit signatures:

* `emit(String, Object)`
* `emit(Map)`

Sub-fields need to be declared under fields in order to be discoverable through
the field_caps API and accessible through the search API.

The way that it emits multiple fields is by returning multiple MappedFieldTypes
from RuntimeField#asMappedFieldTypes. The sub-fields are instances of the
runtime fields that are already supported, with a little tweak to adapt the script
defined by their parent to an artificial script factory for each of the sub-fields
that makes its corresponding sub-field accessible. This approach allows to reuse
all of the existing runtime fields code for the sub-fields.

The runtime section has been flat so far as it has not supported objects until now.
That stays the same, meaning that runtime fields can have dots in their names.
Because there are though two ways to create the same field with the introduction
of the ability to emit multiple fields, we have to make sure that a runtime field with
a certain name cannot be defined twice, which is why the following mappings are
rejected with the error `Found two runtime fields with same name [log.response]`:

```
PUT localhost:9200/logs/_mappings
{
    "runtime" : {
        "log.response" : {
            "type" : "keyword"
        },
        "log" : {
            "type" : "composite",
            "script" : "emit(\"response\", grok(\"%{COMMONAPACHELOG}\").extract(doc[\"message.keyword\"].value)?.response)",
            "fields" : {
                "response" : {
                    "type" : "long"
                }
            }
        }
    }
}
```

Closes elastic#68203
@romseygeek romseygeek self-assigned this Aug 10, 2021
@romseygeek romseygeek merged commit 94407ed into elastic:7.x Aug 10, 2021
@romseygeek romseygeek deleted the runtime/composite-7x branch August 10, 2021 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants