Facilitate adding new fields to existing indices

[jpountz]

One use-case we would like to better support at the stack level is the ability to introduce new fields in time-based indices. The way it is done today consists of adding fields to the index template and maybe an ingest processor so that future indices would pick it up. One issue with this approach is that the field only becomes really available after some time, when all indices in common time ranges have this new field indexed. These couple hours/days during which the field is not yet available on enough historical data are sometimes problematic.

Could we support adding field views to existing indices to better support this use-case, e.g. by supporting scripted fields? Performance would obviously be bad for some time, but this is sometimes a better trade-off than having to wait for a long time before being effectively able to use the new field in Kibana.

Probably depends on #48058

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[colings86]

The mapping of this field will probably need to contain the type of result expected by the script so we can apply it correctly. For example this could be used to determine which aggregations are applicable to the field and could be returned by field caps API as the field type so client applications can treat the field as if it were an indexed field of the relevant type

[jpountz]

@colings86 Agreed. I think it's also important so that these fields can work in the case that some indices have fields mapped as double (for instance) and other indices have the field mapped as a script. One option is to have an option in the mapping like you suggested, another one would be to have one field type for each return type.

[javanna]

Now that we have support for runtime fields as well as data streams, I would lean towards closing this issue in favor of ##72142 . Feel free to reopen if you think otherwise.