Add support for emitting multiple fields values from a script

[javanna]

We recently added support for runtime fields, that are computed at search time based on a painless script. As of today, a runtime field script can emit values for a single field, the one that the script is declared under.

We would like to add the ability for a script to emit values for multiple fields. This will be achieved by introducing support for a new field type (name to be defined) as part of the runtime section. Its script emits fields that belong to such object. This is particularly useful given that scripts support grok and dissect (#68088):

PUT localhost:9200/logs/_mappings
{
  "runtime" : {
    "log" : {
      "type" : "tbd",
      "script": '''
        emit(grok('%{COMMONAPACHELOG}').extract(doc["message"].value)));
      ''',
      "fields" : {
        "clientip" : {
          "type" : "ip"
        },
        "verb" : {
          "type" : "keyword"
        },
        "request" : {
          "type" : "keyword"
        },
        "response" : {
          "type" : "long"
        }
      }
    }
  },
  "properties" : {
    "message" : {
      "type" : "keyword"
    }
  }
}

In the example above, the grok function splits the message field into sub-fields based on the provided grok pattern, and each of the resulting fields is emitted in the following loop. The emitted fields need to be listed under the sub-fields in order to specify their type and make them searchable (and discoverable through field_caps) like any other field:

POST /logs*/_search
{
  "aggs": {
    "response_codes": {
      "range": {
        "field": "log.response",
        "ranges": [
          { "to": 300 },
          { "from": 300, "to": 400 },
          { "from": 500 }
        ]
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jtibshirani]

An alternative would be to model this using multi-fields:

"message" : {
      "type" : "keyword",
      "script": '''
        Map fields = grok('%{COMMONAPACHELOG}').extract(doc["message"].value);
        for (Map.Entry field : fields) {
          emit(field.getKey(), field.getValue());
        }
      ''',
      "fields" : {
        "clientip" : {
          "type" : "ip"
        }
      }
    }
  },

I think it fits okay into the multi-fields concept: the parent field holds some value, and each subfield consults the same value but exposes it differently. A benefit of this approach is that there's only one way to define sub-fields, so the conflict in the example isn't possible. A more theoretical point, but I also like that it maintains an invariant: any field that stores + parses a document value is a leaf field type (not an object mapper). This includes fields that expose virtual subfields like flattened in addition to non-virtual ones like aggregate_metric_double.

[javanna]

Thanks for the feedback!

One point of concern is how the runtime field will be shaped when it is made indexed. We'd love to be able to paste it under the properties section. That sounds harder if we follow the multi-fields approach, I'm afraid. Or maybe not, as long as scripts behave the same under both the runtime and the properties section.

I wonder if it would be harder to follow that the script for a keyword field can emit multiple fields or not depending on whether it has sub-fields defined. The intention so far would be to expose that key-value emit (or something along those lines) only to the object variant of a runtime field.

On the naming conflict, I believe that you could still define a message.clientip field with the dot in its name, outside of the message field of type keyword, and we would have to possibly forbid having both. It is still up for discussion what the right behaviour should be, my intention was only to point out the potential problem in the description above.

I was also under the impression that the invariant you mentioned is valid with both examples, because you still need to define the leaf fields. The script is meta: it tells what to emit and where to take it from, but the object is still a collection of other fields.

Does this make sense to you?

[javanna]

One additional point that came to mind is that the message field itself may be defined under properties, and doc['message'] would refer to it. If the script to split the field is defined under a message field of type keyword, that forces the mssage field to be runtime too? That field, to my mind, is really just a container of other fields, but on its own, it has little meaning when referred to, for instance in query.

[javanna]

Heads up: I have updated the description of the issue according to recent discussions and the draft PR I opened. The API has slightly changed, and we need to come up with a name for the new field type if we decide to not use object.