Adding the ability to mark templates and pipelines as deprecated

[felixbarny]

In #96267, we've aligned on a new naming convention for components that are installed via template registries.

In #99975, we've started to rename component templates and ingest pipelines. To avoid backwards compatibility issues, we're adding the components under both the new and the old name.

However, that duplication is probably confusing to users who are looking at the stack management UI. Which of these component templates should they use? This is where a deprecated flag could come in handy. We could hide deprecated component templates by default and issue a deprecation warning when using a deprecated component template.

It would also allow us to remove the old component templates after a deprecation period. The deprecation warnings can give us insights on on usage of the deprecated component templates and whether it's safe to remove them.

The question is whether working on this is time well spent or whether we're fine with just continuing to provision the templates under both the old and the new name. We'd only document the new names. But the old name will still be visible under stack management.

@eyalkoren @dakrone @jbaiera what's your opinion on that?

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[eyalkoren]

I think if we find a way to do this without being blocked by UI changes, this is time very well spent. Returning duplicates is very confusing and messy.
A proposal:

• add a deprecation marker to templates' metadata
• filter based on this marker through the internal APIs (e.g. in TransportGetComponentTemplateAction) to return only non-deprecated ones

The benefits:

• seems easy...
• no UI changes required
• this filters both for UI and external API queries

I am not deeply familiar with these areas, so I hope this makes sense.

[felixbarny]

I'm not sure if we can just filter out deprecated component templates from the response. At least when directly accessing a deprecated component template by name, I don't think ES should return a 404. I suppose we'd need something similar to the expand_wildcards option Elasticsearch to either include or exclude deprecated component templates when a wildcard is used, such as GET /_component_template/*.

[felixbarny]

I've created a quick POC for this: #101148

When fully implemented this does touch a lot of areas so before taking on the full work, I'd like to get consensus on whether we all think it's a good idea to do that.

I don't really have a strong opinion either way. I think it would be a nice to have but not entirely sure if it's worth the effort.

@jbaiera and @dakrone I'd really like to get your opinion on this and whether or not this is a blocker for

• Rename component templates and pipelines according to the new naming conventions #99975