Synthetic _source automatically sets store to true on text fields

[martijnvg]

There are at least some metric related integrations that have templates that use the text field type.
When these integrations are migrated to tsdb, then often this fails initially complaining with the error that text field either needs a subfield of type keyword or should be stored (by setting store=true, enabling stored fields for this field specifically).
This is a bit confusing sometimes, because the error doesn't come from ES directly and takes time to figure out what needs to be changed.

There are two options here, remove the text field from template when upgrading to tsdb or make suggested changes to the field of type text. What often happens is that store is set to true. However if this is always the intended approach for integrations, should Elasticsearch change the default of store to true for tsdb?

Relates to #96254

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[agithomas]

haproxy integration has datastream stats which has field having type text. Since there exist a text type field, on doing TSDB enablement (stack : 8.8.2), error mentioned below appears. This will be a blocker for TSDB enablement of HAProxy.

Install the package Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"illegal_argument_exception\n\tRoot causes:\n\t\tillegal_argument_exception: field [haproxy.stat.source.address] of type [text] doesn't support synthetic source unless it is stored or has a sub-field of type [keyword] with doc values or stored and without a normalizer"}

[martijnvg]

We discussed this and we should default the store option to true in case synthetic source is used and there are no sub fields of type keyword. This way synthetic source can synthesise the field from the stored field. If there is a keyword sub field then synthetic source can synthesise from the sub field.

This better than failing with an error. If store is specifically configured to false on text fields then creating a template or index with synthetic source will return the ... doesn't support synthetic source unless it is stored or has a sub-field of type [keyword] with doc values or stored and without a normalizer error.

[agithomas]

We discussed this and we should default the store option to true

I assume, based on the discussion here, what is stopping from having this option used is not having the support of package-spec. Correct?

Please let me know from which version of Elasticsearch, the support of store : (true | false) is present? 8.8.0 and above ?

[martijnvg]

Please let me know from which version of Elasticsearch, the support of store : (true | false) is present? 8.8.0 and above ?

Setting store to true has been option since tsdb was GA-ed iirc. So I think from 8.7.0 and later.

I assume, based on the discussion #96254 (comment),

Note that the `error: building package failed: invalid content found in built zip package: found 1 validation error:

1. file "fields.yml" is invalid: field 0.fields.3.fields.7: Additional property store is not allowed` error isn't an elasticsearch error.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)