[Index pattern field editor] Support "object" runtime fields

[sebelga]

We need to add support in the index pattern field editor for the upcoming runtime object field.

For that we will need to rethink the UI of the flyout as we will need to take into consideration if the script returns an object with multiple fields or a single field value.

The script will emit a one level depth object.

{
  field1: 'foo',
  field2: 'bar',
  field3: 'baz',
  field4: 'boo'
}

The user will then need to

1. specify which one of those 4 fields he wants to include in the object
2. specify their type

"runtime" : {
  "myRuntimeField" : {
    "type" : "object",
    "script": '''
      Map fields = grok('%{COMMONAPACHELOG}').extract(doc["message"].value);
      for (Map.Entry field : fields) {
        emit(field.getKey(), field.getValue());
      }
    ''',
    "properties" : {
      // Only include those 2 fields in the runtime object
      "field1" : {
        "type" : "ip"
      },
      "field3" : {
        "type" : "keyword"
      },
    }
  }
}

Tasks

• Design the UX to handle multiple fields
  • what do we do with the format and priority sections with object fields?
  • how do we display the preview for single and object fields?
  • how do we let the user provide documents to execute the script against?
  • where do we display the debug callout if the script fails?
• Add support for object runtime field "subFields" on the IndexPattern ([Index pattern] Add support for object runtime field "subFields" #99177)
• Implement a preview handler (React hook) that will execute the runtime script and return the value that is emitted.
  • This handler will accept a runtime type, a runtime painless script and an array of documents. Optionally we could also provide to this handler an index pattern and an array of document IDs).
  • The handler will return a single value or an object with properties
  • This handler will need to be exposed by the index pattern field editor plugin so other plugins (e.g. data parsing) can consume it.
• Build a reusable <FieldSelector /> component
  • If the above preview handler returns an object, this object will be fed to a <FieldSelector /> component
  • This component will call ES and ask for the recommended type for each of the leaf objects.
  • Once we know the recommended type we will list the all fields emitted by the script. Each field will have the recommended type prepopulated in a dropdown select. The user will be able to change the type if he wants to.
  • The user can select (through checkboxes or toggles) the fields he wants to include in his runtime object field.
  • This reusable component will need to be exposed by the index pattern field editor plugin so other plugins can consume it.

Other consideration

• When saving an object runtime field on the index pattern we should have a flag to indicate if this field has been generated from the data parsing UI or not. This flag will allow us, when clicking "edit field", to redirect to the enhanced data parsing UI instead of the index pattern field editor.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[timroes]

@sebelga I fear this might become more complicated then most of the other fields so far. For those "child fields" to properly function across Kibana (i.e. like real object fields), they'll need to have individual field instances in the index pattern (which would also allow individual formatting). We actually currently throw away the root nested or object field since we cannot do anything on the root in Kibana. I guess with the object runtime fields we'll need to keep those around, since I assume this is what the user actually edits? We might need to validate whether different places of Kibana will work if we have those inside.

Also since Kibana (e.g. in Discover or Maps or Visualize) will always ooperate on the leaf field, we'll show those in lists and such and so we'll have an edit button for them. I assume we still want to edit the overall script when editing those fields? Though not sure how that integrates with that you might still want to change the field formatter specifically for that child field.

Does this answer your question or do you want us to clarify in more detail for any specific area?

[sebelga]

Thanks @timroes, I've created #99177 with a proposal to support leaf object for object runtime fields. Let me know any feedback/concerns on that issue. My goal is that the consuming app should not know and worry about object runtime "sub fields". They will appear just like any other fields.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.