Retrieve field data density via API

[timroes]

In the now removed field stats API we had the information about the density of data within a field. That's a useful measurement, we would like to have in Kibana for prioritizing field suggestions when creating visualizations.

Since it seems that information is no longer in the field caps API, I would like to check why it got removed, if there are some replacement APIs we can use instead and if not currently, tracking some enhancements of making that data available via API.

cc @flash1293 @wylieconlon @chrisdavies

[elasticmachine]

Pinging @elastic/es-core-features

[elasticmachine]

Pinging @elastic/es-search

[jasonrhodes]

Some related conversations:

• [Infra UI] Limit "group by" choices for Metrics Explorer UI based on metric selection kibana#39613
• [Infra UI] Only display available fields in Metric Explorer kibana#36843

[jpountz]

The general motivation was to move logic from Kibana to Elasticsearch. We replaced field stats with field caps at the same time as we removed the need for Kibana to select the relevant indices based on the time range by introducing the new can_match phase.

The field caps API is more lightweight than the field stats API since it only needs to look at the mappings of an index. In practice it means it only needs to go to one shard for each of the queried indices. On the other hand, field stats exposed information about the data, so it had to go to every queried shard and request some metadata such as the min/max values and document counts for every field. Nothing that can't be changed, but this is worth consideration.

If you are mostly interested in getting information about fields that actually exist in indices, then we have been discussing ways to fix this information at index-time, e.g. by giving Beats one index per module instead of having mappings that include fields for every possible module including the disabled ones. This would be a better fix in my opinion.

Regarding ranking field suggestions, maybe one metric we could expose somehow would be the number of indices that have this field.

[javanna]

No activity in 3 years, closing for now, feel free to reopen / comment if this enhancement request is still important, so that we can re-open the discussion around it.