[Lens] meta fields are not supported in visualizations

[rajvi-patel-22]

Description

Below two SpringBoot dashboards contains Audit Events [Logs Spring Boot] panel which uses _id field:

1. [Spring Boot] Audit Events
2. [Spring Boot] Overview Dashboard

These visualizaions are working as expected in 8.4.0 and 8.5.0. But in Kibana 8.6.0 and later versions it shows below error.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[stratoula]

@rajvi-elastic how was these visualizations been created? _id is not aggregateble so it doesn't appear on Lens field list 🤔

This is from 8.5

As seen on the Es documentation

The _id field is restricted from use in aggregations, sorting, and scripting.  In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

[rajvi-patel-22]

@stratoula, In cloud instance it is showing _id field is not aggregateble. But when I checked in on-prem ES instance, _id field is aggregateble.

[stratoula]

Hmm, I dont know why, from the Lens side if the field is aggregatable it will be shown in the list and can be used in aggregations.

@elastic/kibana-data-discovery any thoughts here?

[lukasolson]

Hmm, this is definitely odd. Is there anything different in your mappings for the on-prem vs. cloud cluster?

[rajvi-patel-22]

Hmm, this is definitely odd. Is there anything different in your mappings for the on-prem vs. cloud cluster?

No @lukasolson , we are not providing any other mappings. But we are using elastic-package to develop integrations. It seems like in elastic-package (till Kibana v8.4.0) _id field was aggregatable. When I configured stand alone instance using rpm installation, _id field is not agggatable.

As per our analysis,

	Is _id field aggregatable till Kibana v8.4?	Is _id field aggregatable in Kibana v8.6+?
Cloud instance	No	No
Stand alone (RPM installation)	No	No
Stand alone (elastic-package)	Yes	No

[rajvi-patel-22]

@stratoula Can you suggest the alternate solution for the _id field? How can we use _id field in the visualization?

[stratoula]

@rajvi-elastic if it is not aggregateble you cant use it I am afraid. I amn not sure if you create a runtime field from the _id and use this instead.

[rajvi-patel-22]

@rajvi-elastic if it is not aggregateble you cant use it I am afraid. I amn not sure if you create a runtime field from the _id and use this instead.

@stratoula I believe that is only possible if we have used fingerprint processor to generate _id. Because according to documentation _id field is generated only after document has been parsed from ingest pipeline. [Reference]

The _id can either be assigned at indexing time, or a unique _id can be generated by Elasticsearch. This field is not configurable in the mappings.

[dej611]

In general is recommended to make a copy of the _id content into another aggregatable field:

In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

From: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-id-field.html

I amn not sure if you create a runtime field from the _id and use this instead.

It is possible to do it, but it is not recommended and requires changing a dynamic setting in the cluster.

[rajvi-patel-22]

In general is recommended to make a copy of the _id content into another aggregatable field:

@dej611 I think that is only possible if we are generating _id field in ingest pipeline. Otherwise _id field is generated only after document has been parsed from ingest pipeline.

[lukasolson]

This seems to be a configuration issue rather than a bug with Discover. I'm removing our team label, but feel free to add back if I'm missing something.

[stratoula]

Same for viz team I am afraid, aggregations work for aggregatable fields, so if _id is not aggregatable then ES ags are not going to wotk

[rajvi-patel-22]

@stratoula @lukasolson, I am not sure why this is happening in elastic-package instance only.

However, as per the documentation, we can create a new field and copy the value of _id into that field. So that we can use _id field in dashboards.

Do you have any other better solution to incorporate meta fields into the dashboards?

[stratoula]

@rajvi-elastic no I am afraid that the best solution is the one that is mentioned in the ES documentation

[rajvi-patel-22]

@rajvi-elastic no I am afraid that the best solution is the one that is mentioned in the ES documentation

@stratoula, Understood. Thank you!

[davismcphee]

Since there seems to be an agreement that this is configuration related, I'm closing this issue as part of a backlog grooming effort. If you believe this should still be considered, please reopen it with a comment.