Rename ES templates and pipelines according to the naming conventions

[felixbarny]

See #96267

Also add docs for the new naming convention.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[eyalkoren]

The current plan is to rename only "stack" templates, meaning - such that are registered through StackTemplateRegistry. Other templates will follow after this PR is merged.
We plan to add new index templates that are registered according to the new naming convention and within them refer to the renamed component templates.
Component templates will be registered with both the existing name (through StackTemplateRegistry) and with new names according to the naming convention (through a new registry). They will use the same template file resources.
We should be able to either hide the old component template names or mark them as deprecated in the stack management UI.

Breaking this into implementation steps:

• add new version of stack index templates that
  • are composed of component templates that adhere to the new convention
  • are registered with names that adhere to the new convention
  • have priority of 100
• make sure to do the same for index templates that are duplicated for serverless, if there are such
• change priority to 99 in all index templates that are registered through StackTemplateRegistry
• add the ability to hide or mark legacy component template names as deprecated
• have a plan on removal of legacy component template names
• add a new stack template registry that
  • registers with the new component names
  • registers the new index templates
  • applies the stack.templates.enabled flag (defaults to true) only for index templates- if set to false, only index templates are not registered, component templates are still registered
• add proper documentation about the naming convention and especially about the @ restriction
• document deprecation of the old component template names
• open issues for rename followups in Fleet, other template registries etc.

[dakrone]

have priority of 100
...
change priority to 99 in all index templates that are registered through StackTemplateRegistry

I would prefer that we not change the existing priority, and instead use a different priority for the new templates (250? 201? Either that or decrease the 'old' ones to 150 and use 200 for the new ones). The reason is that we document the existing priority (200) on the index template page — https://www.elastic.co/guide/en/elasticsearch/reference/8.9/index-templates.html under the "Avoid index pattern collisions" section. I think changing the priority verges on the line of a breaking change. We'll need to update this documentation as well, as part of this work.

[eyalkoren]

@dakrone the scope of this issue covers only index templates that are installed by StackTemplateRegistry, which have priority set to 100.

I would prefer that we not change the existing priority

This is the reason not to keep current index templates with same priority and install the new ones with a different (higher) priority.
With the proposal above we are not changing the priority of the builtin index templates, we are installing two instead of one. The logs-*-*, metrics-*-* and synthetics-*-* patterns will still be covered out-of-the-box with the same priority.

[dakrone]

@eyalkoren yep you're right, I misread. Thanks for the correction! In that case it sounds like the docs wouldn't need to change, sorry about my mistake!

[felixbarny]

@andreidan in the context of this issue, we'd want to rename some stack component templates. How can we go about this since some of these are shadowed for serverless? Would we need to merge the changes at the same time in the serverless repo?

[felixbarny]

change priority to 99 in all index templates that are registered through StackTemplateRegistry

We can't rename component templates as the current names may be used in user's index templates. So storing component templates both under the old and the new name makes sense. However, I think we can just rename index templates instead of storing them under the old name with a lower precedence and under the new name with the current precedence. That's because there's no other entity that would reference an index template with its name. Let me know if I'm missing something but I think we can just move them out of the LecacyStackTemplateRegistry.

[andreidan]

@andreidan in the context of this issue, we'd want to rename some stack component templates. How can we go about this since some of these are shadowed for serverless? Would we need to merge the changes at the same time in the serverless repo?

@felixbarny The serverless override is done at the physical json file level based on path, which sounds like won't change according to > "They will use the same template file resources" ?

If the template resource files remain named the same there's nothing to do as the serverless overrides exist already. If the actual files get renamed, or new files get added, serverless be updated such that overrides with the new names exist.

[felixbarny]

The file name will change. I suppose we'll need to merge the serverless PR that renames accordingly at the same time as the PR in the main Elasticsearch repo.