[Logs UI] Support mappings-based runtime fields in the Logs UI

[weltenwort]

🔭 Summary

With elastic/elasticsearch#60100 a new high-level API for retrieving fields from source and stored values was introduced. The goal is to simplify the API even though new features like runtime fields are about to be added (elastic/elasticsearch#59332).

The following issues suggest that the implementation is incomplete at the moment:

• Follow-up improvements to field retrieval. elasticsearch#60985

• "fields" API return a text instead of a number for a numeric sub-field elasticsearch#61033

• Highlighting should leverage the new field retrieval API elasticsearch#59931

• Add fetch fields support for runtime fields elasticsearch#60775

These might be mitigated in short order, though, so we should be prepared to switch to the new field retrieval API from the current _source API along with the core Kibana apps.

🎟️ Task breakdown

⚠️ This is potentially incomplete list.

The following areas will need to be switched from _source access to the new fields API:

• the log entry retrieval in the stream ([Logs UI] Retrieve log entry field values via the unified fields API #76506)
• log entry highlighting in the stream ([Logs UI] Retrieve log entry field values via the unified fields API #76506)
• log entry retrieval in the details flyout ([Logs UI] Retrieve log entry field values via the unified fields API #76506)
• log analysis result document retrieval ([Logs UI] Retrieve ml result field values via the unified fields API #76509)

Other aspects that might require changes:

• ensure expensive suggestions in the KQL bar are timeouted and cancelled appropriately (related to [KQL]: Autocomplete performance issues with runtime fields #75647)

Open questions

• The KQL query bar might not play well with runtime fields. Will we automatically inherit any fixes by the App Arch team or do we need to take action ourselves? ([KQL]: Autocomplete performance issues with runtime fields #75647)

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[stacey-gammon]

The following areas will need to be switched from _source access to the new fields API:

There are some differences in the structure of the field values returned from these APIs:

• Nested fields are flattened
• Arrays are flattened
• Subfields are returned

We'll be syncing with ES team to determine what this means for Discover.

[weltenwort]

There are some differences in the structure of the field values returned from these APIs

Good point, thank you. We'll have to adapt the way we format the values.

[mukeshelastic]

@weltenwort is there more to do to support runtime fields in logs UI? If not, we can close the ticket?

[weltenwort]

Since the description of the issue scopes this to reading and displaying runtime fields in the Logs UI, I think we should be good.

The creation of runtime fields will be handled in a different workstream (e.g. analyst-created runtime fields, index patterns everywhere).

[tbragin]

@mukeshelastic @weltenwort I misunderstood this effort to represent creation of runtime fields from the Logs UI. If this ticket is not it, where do we have a product brief and ticket for it?

[mukeshelastic]

@tbragin For Logs UI, the initial planned user interaction in the context of runtime fields creation, is enabling 'data prep UI' experience of easy log parsing. The product brief for data prep UI is here To build that experience, the foundational pieces are being built - supporting index patterns in our UI, incorporating runtime fields editor to enable the data parsing experience that supports grok. Both of these activities are under design and discussion stage and @weltenwort is representing our team in them. Once we figure out the specifics, we will create tickets for them.

@weltenwort @jasonrhodes Please feel free to correct or add more color as you see fit.

[weltenwort]

Yes, the specific steps towards realizing the value have changed with the arrival of runtime fields. IMHO our previous ingest-pipeline-based user workflow doesn't make the best use of the new technology. I hope we'll get around to sketching a workflow that harmonizes better with the "analyst-created runtime fields" and "index patterns everywhere" themes pursued in discover and visualize.