[Discuss] User experience for pre-installed index templates

[jakelandis]

Currently Elasticsearch install quite a few index templates by default. Below is a screen shot from 7.12 for all of the pre-installed composable index templates. There are additional legacy templates installed too, but this issue is intended to only focus only on composable index templates since the legacy templates will be upgraded in the near future.

Types of index templates

In general there are 4 types of index templates:

1. Installed by default we don't users changing (UI shows these as "System", they also have the "managed" flag)
2. Installed by default that we are OK with the user changing (UI shows these as "Managed", they also have the "managed" flag)
3. Installed by some application/solution for a specific purpose. (UI can display these as either System or Managed, they should have the "managed" flag)
4. Installed by the user.

If a user never touches index templates that have the "managed" flag then are no concerns in the context of this issue. This issue is to discuss the ideal user experience for when they want to change a "managed" index template.

Changing a managed template

Current state API:

The API only makes the distinction between "Managed" or not via a boolean in the metadata of the index template. Any template may be changed by the user with any warning.

Current state UI:

The UI makes a distinction between "Managed" and "System" where "System" is also a managed index template, but visual cues show that it should not be changed by a user. The UI will provide some (correct) scary warnings when trying to change a "System" index template, but does show any warnings when changing a "Managed" template.

Issues changing managed templates

All templates have the ability to express a version, and for "managed" templates this has some additional meaning. (Assuming Elasticsearch is managing the template i.e. 1 & 2 types above) Elasticsearch looks at that version currently in the cluster and the version it wishes to install. If the version it wishes to install is greater then the currently installed version it will install it (overwriting the prior version).

If a user had customized the "managed" template, all of those customization's are now lost. These customization's can be mappings, settings, and associated ILM policies. Additionally since these are composable templates, a user may also update the component template with the same issue of overwriting customizations.

Possible UX changes

• Provide better warnings (both UI and API) for changing managed templates
• Disallow updating managed templates and introduce official concepts such as hidden/system resources See System-owned resources (policies, templates, jobs, watches, etc) #66413
• Bump the version number up by large values (ie. + 100) when using the API/UI to update managed templates so any user updates take precedence during the version check.
• Automatically fork/clone the the managed template to a new namespace, but with a higher priority to help avoid conflicts.
• others ??

Any change here should account for both the UI and the API.

cc: @jethr0null @cjcenizal @dakrone

[elasticmachine]

Pinging @elastic/es-ui (Team:UI)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[cjcenizal]

Great, succinct description of the problem @jakelandis! Thanks for writing this up and tagging the UI team. The core of the issue seems to be:

• ES has a mechanism for updating managed templates. Why does ES update managed templates?
• Users have a mechanism for customizing managed templates. Why do users customize managed templates?

For us to choose the best way to reconcile these two conflicting mechanism, I think we need to answer these "Why" questions first.

If we can get some clear answers, then maybe we can form some opinions about how the two mechanisms work together. For example, if ES needs to update managed templates in order to deliver on certain guarantees about our product (e.g. Fleet policies will behave in a specific way), then we might decide that it's more important to ensure ES can update templates to a predictable configuration than it is to preserve user customizations. In this case, we might want to look into alternative ways to help users reapply customizations after their templates get updated, even if it's a manual process. Or more likely we'll end up with the option you mentioned:

Disallow updating managed templates and introduce official concepts such as hidden/system resources

[dakrone]

ES has a mechanism for updating managed templates. Why does ES update managed templates?

A bugfix, enhancement, or feature that we want to take advantage of related to our default shipped templates. e.g. #64978 (comment)

Users have a mechanism for customizing managed templates. Why do users customize managed templates?

In my opinion, users change managed templates to enhance the definition because it's perceived to be easier than creating their own custom template. From a UX perspective, I think we should dissuade a user from editing these templates in place, instead making it easy to "copy" a template into one the user can customize. But, this is only a hunch from my end, and I don't have any telemetry to back it up. I'd love to hear how other people view customizing managed templates.

[jpountz]

Disallow updating managed templates and introduce official concepts such as hidden/system resources

This would be my preference, I'd also suggest that we prevent users from using system component templates in their own templates, to make sure that we can freely change our logs/metrics/synthetics templates without risk of breaking users.