[Logs Explorer] Explain meaning of field key

[ruflin]

Logs and metrics document stored in Elasticsearch can contain many fields. Some of the field keys are self descriptive, other need more details to explain it especially if the user shipping the data is different from the user consuming the data. For example in the screenshot below, it would be nice to have an explanation on what orchestrator.cluster.name as key exactly means in this context (hint: it is the k8s cluster name). Some info can also be found in the ECS fields. For metrics, it becomes more complex. For system.memory.actual.free as description can be found here in our docs but many metrics are not documented.

User looking at fields should have an option to get more information about the field, be it in the field list (please do not add one more icon to the list) shown below, a visualisation or many other places in Kibana.

There are several sources from where this information could come from:

• Integration docs
• ECS fields
• Otel docs
• Google -> prometheus exporter docs
• GenAI
• Heuristics

We could start simple with bringing in the fields we know about which are ECS and integrations and then expand further. Key is to have a place to show the information.

[tommyers-elastic]

@agithomas has been doing some very interesting work in a side-project which aims to help integration developers better understand fields by analysing their names. the original intention of the tool was the help categorise metrics and determine their importance when designing dashboards, but in the short term, i think the explanations of field names generated by the tool would exceptionally useful to embed in kibana.

the tool uses langchain and is currently written in python. we could write a small wrapper around the tool with some persistent caching and host it in MKI.

[agithomas]

Thanks @tommyers-elastic for introducing my side-project work here. I call the tool as 'Obs-Guidance'

Apart from the potential LLM hallucination problem, we must be watchful about how much information can be embedded in the markdown files, especially when the markdown file exposes details of 100+ number of metrics, navigation becomes challenging. Too few information or No information is presently the case. Obs-Guidance is an attempt to solve this problem.

Obs-Guidance presently uses Elastic Integration docs, Otel docs, Google , Bing to do LLM-RAG and it is certainly extensible to any data sources.

[tommyers-elastic]

my thinking is that the information would be 'on-demand' - so that we can utilize the tool for any custom fields as well. that avoids the issue of having to embed thousands of lines of descriptions in markdown, and offers us the ability to do things like allow users to flag descriptions as inaccurrate (e.g. hallicinations), and invalidate the upstream cache.

[ruflin]

Having something like a service would also allows us to heavily cache the result information assuming the context of the request is identical. It would be nice if we can start implementing something like this in phases / layers. First get the simple info in and start playing around with the LLM to see how it works and only later enable it by default.