introduce new `indices` key with multi ilm and template handling #10002

simitt · 2019-01-10T17:21:57Z

As discussed offline, this WIP PR is a start to address configuration changes related to index, template, ilm handling. Since this is a larger topic I created a feature branch feature-multi-ilm against which we can create PRs.

related to ##9919

Points to cover:

Introduce new config options with root key indices. Deprecate config options setup.template, output.elasticsearch.index, output.elasticsearch.indices. Remove beta config option ilm. See new config options example.
Allow multiple Templates for one beat, by defining a template per index.
Allow multiple rollover policies, one per index/rollover_alias.
Ensure output config in pipeline uses the new indices configuration if available. Currently an index is always set, even when indices are configured, as a fallback when no condition applies. In the future one index that doesn't have a condition must be set in indices, which is used as the default index. If ILM is enabled, the rollover_alias must be used instead of the name.
Check for dependencies: Can/should the template.name and template.pattern be derived from the rollover_alias when ILM is enabled? Which fields are required to be set per indices entry?
Should ILM policy be overwritten when trying to load multiple times, or should it be configurable?
Add tests and fix system tests

@urso, @ph, @elastic/apm-server this PR is not ready for a detailed review, but I'd appreciate high-level feedback.

UPDATE:

cherry-pick policy handling from @urso 's implementation for 6.x
fix test for default yml config without indices, output.elasticsearch.index and setup.template section
@urso I introduced an index.DefaultConfig setting so every beat can set its default index from outside. This is specifically helpful for apm-server (see Add indices defaults in code apm-server#1480). I am not exactly clear on what the default configurations for all the beats should be (especially with ECS changes). I'd appreciate if someone else could update the configs and overwrite the index.DefaultConfig if necessary. I tried to update the config for metricbeat myself.

libbeat/template/config.go

libbeat/index/config.go

libbeat/ilm/load.go

libbeat/ilm/config.go

libbeat/_meta/config.reference.yml

libbeat/cmd/instance/beat.go

libbeat/ilm/config.go

libbeat/index/config.go

libbeat/cmd/export/ilm_policy.go

urso · 2019-01-16T23:23:48Z

libbeat/cmd/export/template.go

 				os.Exit(1)
 			}
+			logp.Info("Loaded Elasticsearch templates.")
 		},


We will have to clean this up as well, so to not print all templates to stdout, right?

We print the templates, we do not load anything. Maybe we shouldn't print Loaded Elasticsearch templates. once we are done.

We will have to clean this up as well, so to not print all templates to stdout, right?

We print exactly the templates that would be loaded - I think this behaves as expected.

We print the templates, we do not load anything. Maybe we shouldn't print Loaded Elasticsearch templates. once we are done.

agreed

libbeat/cmd/instance/beat.go

urso · 2019-01-16T23:25:45Z

libbeat/ilm/conditions.go

+		return false
+	}
+
+	code, body, err := client.Request("GET", "/_xpack", "", nil, nil)


hm... how do we get the info once _xpack is removed from the API?

Good question, but this check is not introduced newly here. Can you link to the according issue where _xpack should be removed, so I can follow up.

libbeat/ilm/config.go

simitt · 2019-01-18T10:38:42Z

@urso I am closing this for now to avoid unnecessary noise, will reopen when it is final.

ph · 2019-01-17T20:44:05Z

libbeat/ilm/config.go

+		*m = ModeAuto
+	case "true", "True", "yes", "y", "Yes":
+		*m = ModeEnabled
+	case "false", "False", "no", "n", "No":


This is inconstant with the rest of beats we do not support all the boolean possibilities.

lets use an explicit "auto", "false", "true"

only lowercase or also uppercase?

libbeat/ilm/config_test.go

libbeat/ilm/load.go

libbeat/ilm/policy.go

libbeat/index/config.go

libbeat/index/loader.go

ph · 2019-01-18T13:31:53Z

@simitt I have still added a few notes to this PR, I will do a final review when its reopen.

simitt · 2019-01-18T17:25:34Z

great, thanks for the feedback @ph !

Further changes and adaptions. Add metricbeat.yml as reference example.

…mplates

simitt · 2019-01-22T11:03:14Z

@ph, @urso this is ready for review. I updated the description above with some open questions/points.

cc @elastic/apm-server

simitt · 2019-01-22T17:25:46Z

Closing this PR as discussions around how to organise configuration settings properly in the future are still going on.

cassandracomar · 2019-08-15T17:38:03Z

Hi @simitt , is this work likely to be merged at any point? Not having this prevents us from being able to use ILM altogether as sending all of our cluster logs to the same index is introducing too many differing fields depending on the source of the log, leaving us with a Kibana that can't determine what fields are in our index patterns. We're disabling ILM for now in favor of using curator to do the same work for us but it looks like curator is being obsoleted over time in favor of ILM.

simitt · 2019-08-19T09:28:19Z

Handing off this question to @urso, as the ILM related index handling for APM and beats developed differently.

simitt requested a review from a team as a code owner January 10, 2019 17:21

houndci-bot reviewed Jan 10, 2019

View reviewed changes

simitt mentioned this pull request Jan 10, 2019

[WIP] ILM with configuration per index #9616

Closed

simitt force-pushed the multi-ilm-multi-templates branch from 02b3c3d to e4a76b1 Compare January 16, 2019 15:53

simitt requested review from a team as code owners January 16, 2019 15:53

simitt changed the base branch from feature-multi-ilm to master January 16, 2019 15:55

urso reviewed Jan 16, 2019

View reviewed changes

houndci-bot reviewed Jan 17, 2019

View reviewed changes

libbeat/ilm/config.go Show resolved Hide resolved

simitt closed this Jan 18, 2019

ph reviewed Jan 18, 2019

View reviewed changes

simitt added 7 commits January 19, 2019 15:24

introduce new indices key with multi ilm and template handling

3e34317

Integrate handling to write to proper index or rollover_alias,

891bf53

Further changes and adaptions. Add metricbeat.yml as reference example.

Adressing PR review suggestions.

9a3e10b

Adding tests for indices handling.

04b42b0

Merge remote-tracking branch 'elastic/master' into multi-ilm-multi-te…

b0e28fc

…mplates

more tests and fixes

547207e

Merge remote-tracking branch 'elastic/master' into multi-ilm-multi-te…

ca7a100

…mplates

simitt reopened this Jan 22, 2019

simitt force-pushed the multi-ilm-multi-templates branch from 662d763 to f157a98 Compare January 22, 2019 10:56

simitt changed the title ~~[WIP] [ILM] introduce new indices key with multi ilm and template handling~~ [ILM] introduce new indices key with multi ilm and template handling Jan 22, 2019

simitt changed the title ~~[ILM] introduce new indices key with multi ilm and template handling~~ introduce new indices key with multi ilm and template handling Jan 22, 2019

simitt added the review label Jan 22, 2019

fixup! more tests and fixes

81ebf66

simitt force-pushed the multi-ilm-multi-templates branch from f157a98 to 81ebf66 Compare January 22, 2019 11:11

Update metricbeat

a51b38c

simitt mentioned this pull request Jan 22, 2019

Add indices defaults in code elastic/apm-server#1480

Closed

simitt closed this Jan 22, 2019

simitt mentioned this pull request Feb 1, 2019

Support for ILM in APM Server elastic/apm-server#1290

Closed

simitt deleted the multi-ilm-multi-templates branch August 4, 2020 14:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce new `indices` key with multi ilm and template handling #10002

introduce new `indices` key with multi ilm and template handling #10002

simitt commented Jan 10, 2019 •

edited

Loading

urso Jan 16, 2019

simitt Jan 17, 2019

urso Jan 16, 2019

simitt Jan 17, 2019

simitt commented Jan 18, 2019

ph Jan 17, 2019

simitt Jan 21, 2019

ph commented Jan 18, 2019

simitt commented Jan 18, 2019

simitt commented Jan 22, 2019

simitt commented Jan 22, 2019

cassandracomar commented Aug 15, 2019

simitt commented Aug 19, 2019

introduce new indices key with multi ilm and template handling #10002

introduce new indices key with multi ilm and template handling #10002

Conversation

simitt commented Jan 10, 2019 • edited Loading

urso Jan 16, 2019

Choose a reason for hiding this comment

simitt Jan 17, 2019

Choose a reason for hiding this comment

urso Jan 16, 2019

Choose a reason for hiding this comment

simitt Jan 17, 2019

Choose a reason for hiding this comment

simitt commented Jan 18, 2019

ph Jan 17, 2019

Choose a reason for hiding this comment

simitt Jan 21, 2019

Choose a reason for hiding this comment

ph commented Jan 18, 2019

simitt commented Jan 18, 2019

simitt commented Jan 22, 2019

simitt commented Jan 22, 2019

cassandracomar commented Aug 15, 2019

simitt commented Aug 19, 2019

introduce new `indices` key with multi ilm and template handling #10002

introduce new `indices` key with multi ilm and template handling #10002

simitt commented Jan 10, 2019 •

edited

Loading