Ensure that Beats index template loading works with v2 index templates

[ycombinator]

Elasticsearch is introducing v2 of index templates. To understand the motivation for this change and it's details, see: elastic/elasticsearch#53101.

We need to ensure that when Beats creates index templates in a v2 index templates universe, the templates continue to work as before. If not, we need to figure out a migration path from v1 to v2 templates.

---

My strong suspicion is that the v1 index templates created by Beats should continue to function as-is as v2 index templates. This is mainly because the index pattern in a Beat's index template contain the Beat version, e.g. metricbeat-8.0.0-* or filebeat-7.5.3-*. So template inheritance doesn't come into play. And without template inheritance, there is not difference between v1 and v2 templates.

However, we still should confirm the above by testing against an actual ES cluster that has v2 templates implemented. That's what this issue is tracking.

Related: #17809

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

Note that there are separate APIs for v1 vs. v2 index templates:

V1:

GET/PUT/DELETE /_template/<name>

V2:

GET/PUT/DELETE /_component_template/<name>
GET/PUT/DELETE /_index_template/<name>

So while the actual template contents might not need to change, Beats code will still need to be adjusted to use the correct API depending on the ES version.

[simitt]

So template inheritance doesn't come into play.

APM Server actually uses template inheritance for ILM templates per index. We are investigating implications and how we can switch to v2 templates, and whether or not that requires changes in beats. We'll keep track of it in elastic/apm-server#3692.

[vbohata]

For large deployment like our the template inheritance is very common and needed. For example we use beats templates together with our templates. Multiple templates match same indices. The beats should reflect this. Maybe it could just allow the beats template to be generated as a component so other templates could include it as a component template.