Plan to remove index pattern field mapping cache

[mattkime]

The desire to remove the mapping cache has been around for a while and will solve numerous problems - #6498

The main potential bottleneck is the elasticsearch field_caps api. The field mapping cache is a simple transform of its results.

@jtibshirani has assured me that the field_caps api is performant. We should investigate to ensure this is the case.

The problem in removing it is that we need to make sure our code remains performant in all circumstances. An elasticsearch performance test of the field_caps api would help instill confidence. I see two potential results -

a) We're confident that the field_caps api will always be performant. Just remove the mapping cache already and be done with it.

b) We think the field_caps api will usually be performant. In this case we could remove the field mapping in some cases but not others. In this way we could transition toward removing the field mapping cache, perhaps providing an escape hatch if users have poor experience or defining the cases where a cache is necessary.

Either path should allow us to remove the field mapping cache in a considerable number of cases.

--

Request for help determining performance limitations - elastic/elasticsearch#59581

---

Reasons to do this

• Metricbeat / filebeat running into field list refresh problems - [Metricbeat|Filebeat] Number of fields will prevent Kibana index pattern creation beats#20432
• Eliminate need for Metricbeat / filebeat to generate field lists
• Eliminate need to manually refresh field list when index fields change
  • request to provide api to refresh - Add API call to refresh index fields #57699
• Respect field level security concerns
• Eliminate auto refresh cases - Index pattern automatic refresh - move to migration step #71589

Bugs it may resolve

• Index pattern management refresh issue - Index pattern management screen is stale (non-updating) after refreshing #60359

---

Plan for removal -
0) Move 'count' out of field list.

1. Provide server side support for loading field lists in index patterns. Its currently read only. rollups has its own endpoint for creating its field lists. I'd prefer to add this to the OSS code instead of creating a registry. Rollups v2 should get rid of the need for this separate endpoint.
2. Remove code and UI related to refreshing index patterns.
3. Verify index patterns are loading properly and remain performant.

[ppisljar]

with removing the cache, would that mean that indexPatterns.get(id) always queries elasticsearch ?
what about indexPattern.getNonScriptedFields() ?

or are we talking only about index pattern creation ?

[mattkime]

with removing the cache, would that mean that indexPatterns.get(id) always queries elasticsearch ?

If not loading from client side cache, yes.

what about indexPattern.getNonScriptedFields()?

Field list references and getNonScriptedFields are currently synchronous. I think it would be good to move toward async access but we'd likely need to approach that later with a new index pattern api.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[XavierM]

@mattkime I wanted to share with you that the security solution is using the field_caps API from elastic search and asking for at least seven different alias at least to get all the fields (almost 6500 fields) information from these indices alias.

I would like also to tell you that we are also adding some other data into it like description and indexes to know where the field belongs to.

We are starting to refactor our code to use the index patterns service from the kibana platform, but we realized our field API is faster than the refreshFields function. Therefore we will use both in conjunction, so we can get the best of the two APIs.

I also know that infra and logs from observability are doing the same thing for the index metric-* alias and using field_caps API. I really do think that using the field_caps API will be a benefit for every solution using your API.

[mattkime]

@XavierM Do you know why hitting field caps directly was more performant than loading from a saved object field? Its quite surprising.

[mattkime]

Resolved via #82223