Upgrade the Dataview API request to filter out "Empty fields"

[ninoslavmiskovic]

We're soon introducing an enhancement to the field_caps API that will allow the exclusion of fields that have always been empty, on an index level, from the field list. These fields will be categorized under the section titled "Empty fields."

This idea is to refine the Dataview API request. This upgraded request will retrieve the field list but omit the empty fields.

This update is primarily aimed at enhancing the user experience (UX) by streamlining the field list presented during searches, particularly in the unified search interface. Users will benefit from a more relevant and concise list, displaying only fields that contain values. Example of a long field list that could be shorter:

Skaermoptagelse.2024-02-01.kl.17.41.18.mov

Additionally, this change is expected to reduce network traffic and lower associated costs.

This improvement will be reflected across all functionalities that utilize the data-view request for obtaining the field list.

Important Note:

We are also considering the provision for users to access the "Empty fields" when needed. This functionality will be integrated into the design of the field list and unified search interfaces, among others, allowing users to retrieve these fields on demand

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Setting this to blocked, because filtering out empty fields as a default would cause performance issues on Elasticsearch side, when there are lots of shards queried for just returning fields that have/had values

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.