[Fleet] Configuration of ILM and index templates

[jordansissel]

Talking with @mattapperson about Fleet, I've learned that most things will not be configurable on the beats side of things, including templates/ilm policy/etc.

Is this true?

I'm thinking about how this might work in production, and I'm seeing it like this:

1. Fleet is upgraded along with Elastic stack (kibana/elasticsearch)
2. Fleet installs the newest template/ilm policy/etc similar to beats setup
3. Fleet begins shipping upgraded versions of beats/etc to the fleet agents around the world.

The default beats templates and ILM policy do not work for our use case (default is 1 shard per index, 50 gb per index). We use ~5-10 shards per index and rollover around 150gb. among some other settings (node_left.timeout, total_shards_per_node, etc)

This seems to present a race condition: We must find a way to install our template and ILM policy changes before any new beats begin sending data, otherwise new indices will be created with an incorrect index settings and ILM policy?

[elasticmachine]

Pinging @elastic/epm (Feature:EPM)

[mattapperson]

@jordansissel Just to clarify. These things can be overwritten to edit. And I am not sure about roadmap of supporting this use-case in some way.

[roncohen]

hey @jordansissel. Sounds like there's a couple of misconceptions.

We have packages, mapping templates are defined in packages. If you have special requirements to mapping templates, you can create your own package based off an existing one or from scratch. You can control which packages get updated at what time. If you change an ILM policy, fleet will not revert it just because you upgrade a package.

If you create a new cluster, then you must install your package into the new cluster before you have beats sending data. Happy to jump on a call to talk it through

[mattapperson]

@roncohen a couple of questions:

1. “creating your own custom package” is, if my understanding is correct, a roadmap item, not on initial release?
2. “ fleet will not revert it just because you upgrade a package.” If the ILM policy is updated in the package of let’s say a major update it will overwrite though, correct?
3. my understanding is that fleet outputs have been limited to current cluster. So that the cluster where data is shipped to must be the cluster fleet is running on. Again, at least at launch?

[jordansissel]

Can we read about this anywhere? All my fleet information comes from things I hear or ask about or see in slide decks. “Packages” is a new concept to me as of your comment, so I’m a bit lost on such things.

…

On Thu, Oct 24, 2019 at 4:42 AM Matt Apperson ***@***.***> wrote: @roncohen <https://github.com/roncohen> a couple of questions: 1. “creating your own custom package” is, if my understanding is correct, a roadmap item, not on initial release? 2. “ fleet will not revert it just because you upgrade a package.” If the ILM policy is updated in the package of let’s say a major update it will overwrite though, correct? 3. my understanding is that fleet outputs have been limited to current cluster. So that the cluster where data is shipped to must be the cluster fleet is running on. Again, at least at launch? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49121?email_source=notifications&email_token=AABAF2VH435RF66VGBBJ4C3QQGCZBA5CNFSM4JEKHRDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECEXG7I#issuecomment-545878909>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAF2XFBQ7XMTR35LYKMBDQQGCZBANCNFSM4JEKHRDA> .

[roncohen]

“creating your own custom package” is, if my understanding is correct, a roadmap item, not on initial release?

packages are tarballs with assets and a manifest. Given an existing one, it should be pretty straightforward to create a new one, even without UI support in the MVP.

“ fleet will not revert it just because you upgrade a package.” If the ILM policy is updated in the package of let’s say a major update it will overwrite though, correct?

If a user has set up a custom ILM policy to control data retention, we'll not be overriding that if they update a package.

A bit more detail: if a user edits "managed" items (dashboards etc.) and then upgrades a package, we'd override their edits. We need to clearly signal which items are managed - and in the future ask the the user to edit duplicates instead the managed items.

my understanding is that fleet outputs have been limited to current cluster. So that the cluster where data is shipped to must be the cluster fleet is running on. Again, at least at launch?

this is still being decided. Due to the fact that the data sources you set up in Kibana must be properly installed in the destination cluster, we're planning for the Elasticsearch output to be limited to the same cluster as the one Kibana is talking for for the MVP. Logstash/Kafka outputs should not have this limitation, but it's being discussed if we can have those in for the MVP.

[jordansissel]

packages are tarballs with assets and a manifest

Can I read about this anywhere or is there a sample? As in, implementation details or drafts of some kind? Having this might help me answer my own questions. The alternative is me asking a bunch of yes/no questions "Can we ..." that might get tedious especially as I have new questions over time.

If a user has set up a custom ILM policy to control data retention, we'll not be overriding that if they update a package.

For context. beats pushes a brand new template/ilm/etc for every version and every beat (filebeat, auditbeat, etc). What do template/ilm/alias objects look like under Fleet (or EPM? i'm not sure yet which project this is under).

[ruflin]

All the details about the packages can be found here: https://github.com/elastic/integrations-registry. As an example, have a look at coredns: https://github.com/elastic/integrations-registry/tree/master/dev/package-examples/coredns-1.0.1

For template / ilm / alias: There is 1 index, 1 template per input. If you think of the nginx module today, it has 3 inputs: access logs, error logs, stubstatus metrics. There are 2 default ILM policies for logs and metrics, but you can use your own per input if you want.

The project installing all these assets is EPM #36708

[ruflin]

I'm closing this issue as I think we can address the above concerns with how Fleet / EPM works with the indexing strategy. @jordansissel Happy to go through it in detail with you.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)