Alias templates

[martijnvg]

The meta issue will track the development of a new experimental feature named alias templates.

Tasks:

• TBD

Background

Many usages of Elasticsearch write data into a write alias (an alias instance pointing to an index as write an index) and use rollover to create a new backing index if needed, which also updates the write alias. The creation of the write alias is crucial for these kinds of setups, otherwise data is being indexed into a concrete index and then rollover fails to do its job. Creating the write alias is a bit strange, because it also requires creating the first backing index as part of setting up a solution that requires a write alias. On top of this the write alias can easily be removed if the backing indices are removed via the delete index api.

Alias templates

Alias templates create a write alias and a backing index when indexing into a namespace that is neither an index or an alias (unused namespace). Also index settings, mappings and other aliases that are associated with the alias template are applied to the index that get created.

When indexing into a namespace that is neither an index nor an alias and matches with an alias template then a write alias is created using the provided index name and a concrete index is created using the provided index name as a prefix for the name of the index. The write alias points to this index. The settings and mappings in the alias template will be applied when creating the concrete index. The name of the new index will be ‘[prefix]-000000’. This will make the concrete index rollover compatible.

There are exceptions to the above described behaviour:

• If the unused namespace name ends with a dash and 6 digits and has a prefix for which a write alias does exist then only a regular index is created and the settings, mappings and aliases are applied that match with the alias template. This is important in order for rollover to work correctly, otherwise a new write alias would be created, whereas instead the rollover api should update the existing write alias.
• If concrete indices already exist for a write alias for an unused namespace then only a write alias is created. The write alias will refer to the concrete index with highest digits in its suffix.

A non existing namespace can only match with a single alias template. Only the best matching template is applied. Best matching is defined by the template with the highest priority. If multiple alias templates match that have the same priority then that results into an error. Either at alias template creation time or when alias templates are evaluated as part of index creation. Preferably the put alias template api should fail in this case, but it can be tricky to figure whether multiple templates are going to match that have the same priority. We can also think about not allowing alias templates with the same priority regardless whether the patterns are overlapping. This is something that can easily be enforced.

Additional constraints:

• A template does need to be valid on its own. Concretely this means that creating a new index with settings and mappings from a single alias template should be successful and not result in validation errors.
• The write alias is always created last, this is in order to avoid that the write alias gets overwritten by a custom alias defined in the alias template.
• Ingest should check alias templates in order determine whether a default or final pipeline needs to be executed. This should be similar to what ingest does today with index templates.
• If an alias template matches then all index templates will be ignored. Even if index templates would match with the alias or index name. If alias templates are not used then setups using index templates remain to function as before. This does allow systems to experiment with alias templates or gradually move from index templates to alias templates.

Component templates

Multiple alias templates should be able to reuse config. Despite the fact that only a single alias template can match with an unused namespace. This is what the purpose is of component templates. A component template only contains settings, mappings and aliases. Alias templates can be composed of one or more component templates.

Component templates never match at alias/index creation time directly and are only included via matching alias templates. Component templates can’t be composed of other component templates. Also component templates can only contain settings and mappings that are valid on their own.

The order in which the names of component templates are specified in an alias template determines the order in which the component templates are applied. The setting, mapping and aliases specified inside an alias template are applied last.

Config and APIs

Both alias and component templates are stored in the cluster state as separate resources.

Alias Templates Config

Config:

{
    "alias_patterns": [...],
    "target_index_pattern": "...",
    "priority": 0-∞,
    "composed_of": [...],
    "version": 0-∞,
    "template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    }
}

Explanation of the configuration fields:

1. alias_patterns (required), a string array. Each string contains a wildcard pattern.
2. target_index_pattern (optional), a json string. Allows to customize the name of the backing index. Defaults to ‘{name}-000000’. The name variable is the write alias being indexed into. This for example allows the use of date math: ‘{name}-{now/d}-000000’. The backing index name suffix should always be -000000 and if that is forgotten this will be appended.
3. priority (optional), a numeric value indicating precedence over other alias templates. When indexing into a non existing index/alias, the name may match with patterns of multiple alias templates, the alias template with the highest order value will be selected. Other alias templates will be ignored. Defaults to 0.
4. composed_of (optional), a string array with the names of components templates that will be applied on a new index together with the current template. The order in the string array determines the order in which the component templates are applied. If composed_of is specified then the alias template is applied last. This is to ensure that the settings/mappings for the alias template are never overwritten by component templates it references to.
5. version (optional), a numeric value. External systems can use this version number to simplify template management. The version parameter is completely optional and not automatically generated by Elasticsearch. Defaults to null.
6. template (required), a json object containing at least settings, mappings or aliases.
7. settings (optional), a json object representing index settings.
8. mappings (optional), a json object representing mappings.
9. aliases (optional), a json object representing aliases.

Alias Template APIs

Put alias template api:

PUT /_alias_template/[name]
{
   ...
}

Get alias template api:

GET /_alias_template/[name]

Delete alias template api:

DELETE /_alias_template/[name]

Simulate alias template api:

GET /_alias_template/[index_alias_name]/_simulate

The simulate api provides insight which alias template would match with an index alias name that doesn’t exist yet and from which templates would be inherited.

Component Template Config

Config:

{
    "version": 0-∞,
    "template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    }
}

Explanation of the configuration fields:

1. version (optional), a numeric value. External systems can use this version number to simplify template management. The version parameter is completely optional and not automatically generated by Elasticsearch. Defaults to null.
2. template (required), a json object containing at least settings, mappings or aliases.
3. settings (optional), a json object representing index settings.
4. mappings (optional), a json object representing mappings.
5. aliases (optional), a json object representing aliases.

Component Template APIs

Put component template api:

PUT /_component_template/[name]
{
   ...
}

Get component template api:

GET /_component_template/[name]

Delete component template api:

DELETE /_component_template/[name]

The alias_required parameter

The alias_required flag ensures that when indexing into a namespace that is not an alias that a failure is returned instead of just indexing into a concrete index. This should prevent situations where it is expected that data is being indexed via an alias, but the alias hasn’t been created, because the alias template wasn’t set up. The bulk api should support the alias_required flag.

Example

Creating an alias template for mysql error logs:

PUT /_alias_template/mysql_error_log
{
    "alias_patterns": ["logs-*-mysql_error"],
    "mapping": { … },
    "settings": { … }
}

Indexing data:

POST /logs-prod-mysql_error/_doc
{ … }

The ‘logs-mysql_error’ namespace matches. This results in the creation of a ‘logs-prod-mysql_error-000000’ index and ‘logs-prod-mysql_error’ alias that points to ‘logs-prod-mysql_error-000000’ as a write index. Settings and mappings from the ‘mysql_error_log’ alias template will be applied.

Rolling over:

POST /logs-prod-mysql_error/_rollover/
{ … }

This results in the creation of ‘logs-mysql_access-000001’ index and the alias ‘logs-mysql_access’ pointing to this index as a write index. The settings and mappings from the ‘mysql_error_log’ template are applied.

This originates from the create index request that the rollover api triggers. The reason why not a new write alias is created is because the index name ends with a dash and a six digit number. The rest of the name matches with ‘logs-mysql_error’ write alias. This makes the alias template logic skip the creation of a new alias. Only settings and mappings of the matching alias template are applied.

All the logs mysql indices gets removed:

DELETE /logs-prod-mysql_error-*

Note that this also deletes the ‘logs-prod-mysql_error’ alias.
Data gets continuously indexed:

POST /logs-prod-mysql_error/_doc
{ … }

Since ‘logs-mysql_error’ is neither an index nor a write alias, the index creation logic kicks in. The alias template ‘mysql_access_log’ matches again. However this time only a write alias is created with the name ‘logs-mysql_access’ that points to the ‘logs-mysql_access-000000’ index.

Relates to logstash-plugins/logstash-output-elasticsearch#858

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

[cjcenizal]

CC @elastic/es-ui

[matschaffer]

Would love to see some more concrete examples like the POST /logs-prod-mysql_error/_doc one.

Especially curious about the use case for having these fields on both alias template and component template

"template": {
       "settings": {...},
       "mappings": {...},
       "aliases": {...}
    }

And do I understand correctly that when using alias template and component template, we wouldn't use index templates at all (for the same scope of indices)?

[martijnvg]

The alias template feature is superseded by data streams (#53100) and index templates v2 (#53101).

The main reason behind is that aliases are not a first class concept in Elasticsearch. Alias templates requires a lot of knowledge about ES in order to get started with ingesting time-based data (aliases, alias templates, and write aliases). Data streams will be a first class way to interact with time based in Elasticsearch. The data stream design as stand currently will not include any configuration. The ability to explicitly compose templates will be implemented via index templates v2.