Cross-Stack Central Config Management

[elasticmachine]

Original comment by @tbragin:

Cross Stack Config Management

Why?
The requirements for this project mostly came from Logstash and Beats products who store and manage configuration files locally. The goals are to centralize configuration to a central store - Elasticsearch - and use APIs/UI to manage them.

Benefits

• Ease of deployment and ongoing management. Currently, Logstash is managed via configuration files local to each instance. This approach has a number of drawbacks that result in poor ease-of-management. Clustering provides an enterprise deployment mode for Logstash that resolves a number of these issues and results in a much better experience, especially when managing Logstash at scale.
• No more restarts on config change. Whenever configuration has to be modified, the Logstash instances have to be restarted. This flies in the face of the business-critical, always-on use cases for which the Elastic stack is increasingly used. Another benefit of this theme is to eliminate the need to restart Logstash processes during routine configuration changes.
• Reduce reliance on 3rd party management tools. When deploying and running many instances of Logstash, users have to manage them manually or use 3rd party management tools (e.g. Puppet), increasing learning curve and complicating deployment. One benefit of this theme is to reduce reliance on external tools for Logstash management.

Requirements (Phase 1):

Use Elasticsearch as the centralized config store

• Add API based management to manage and fetch config files
• CRUD operations to handle configurations
• Option to read configuration from Elasticsearch on Logstash/Beats startup
• Create a x-plugin to abstract config management on ES-side and provide a better user experience by exposing specialized API endpoints (like /_logstash/ and /_beats/)

Dynamic configuration reloading

• Logstash/Beats instances will be able to pickup configuration changes from Elasticsearch without
  the need to restart the process
• Not configurable, if using centralized config, should automatically reload.

Versioning
Track config history for rollback scenarios

Security
Initially, can just be simply facilitated with Shield authentication

Migration
Existing Logstash/Beats nodes will be able to migrate to Elasticsearch-based configuration by uploading current config into Elasticsearch

Backward-compatibility
Retain the standalone deployment with local config file management, centralized config management is just an option. Statistically, most Logstash users use Elasticsearch, but not all.

Future Requirements (Phase 2):

UI integration
Full config management user workflow via UI
“Cluster” management, potentially integrate the centralized config mgmt with the monitoring features, allowing Logstash instances running the same configs to be monitored together

Clustering

• Associate nodes with roles
• Introduce concept of role tagging
• Heterogenous clusters
• Multi-role nodes (with multi-pipelines in the future) - pipeline/role correlation
• Config proxying: Logstash should be able to act as proxy for another Logstash (or Beats) instance for cases where its remote/edge instances can't reach ES directly, yet still need to be centrally managed
• Load balancing
• Automatic failover

---

This meta-issue is to track the effort of introducing cross-stack central configuration storage and management in Elasticsearch. The rough pieces we initially defined for 5.1 are:

Elasticsearch Management

• [ES]: To be defined - is there a set of important configuration management tasks we should target for this phase?

Kibana Management

• [Kibana - OSS]: Kibana already stores its configuration in an Elasticsearch index.
• [Kibana - OSS]: To be defined. Kibana already has a Settings app designed to manage configuration stored in Elasticsearch, but there is currently no way to see/manage multiple Kibana instances using a single UI.

Beats Management

• [Beats - OSS]: Add ability to read configuration from Elasticsearch, in addition to local config file. Use Elasticsearch as a central config store for beats - Phase 1 beats#464
• [Kibana - X]: Expose a basic Beats central management UI: show all Beats, add ability to edit and reload config for each beat, add ability to group Beats with common config using tags.

Logstash Management

• [Logstash - OSS]: Add ability to read configuration from Elasticsearch, in addition to local config file.
• [Kibana - X]: Expose a basic Logstash central management UI: show all Logstash instances, add ability to edit and reload config for each beat, add ability to group Beats with common config using tags.

Other

• Standardize how we store central config in ES across projects
• Competitive research on other UIs present this information

[elasticmachine]

Original comment by @suyograo:

@tbragin I updated this issue with details from our discussions/requirements in GDoc. /cc @tsg

[elasticmachine]

Original comment by @djschny:

My apologies if I didn't get this from reading the writeup above, but will there be an abstraction layer present so that centralized config management does not require Elasticsearch? Logstash and Beats are services that do not require Elasticsearch and keeping them loosely coupled is important. For example:

• One LS cluster may support writing to many ES clusters, not just one. So picking just one to use or setting up a standalone cluster for the config management would not be ideal.
• One LS cluster may never even write to ES so there is not an ES in the environment to use.

Providing an abstraction layer and then by us proving it out by creating the first implementation with ES would be great. Then we can work on other implementations of the community may choose to create other implementations as well (JDBC, ZK, Mongo, Cassandra, GIT, etc.). I would personally love to see GIT as a backing store for centralized config persistence.

[elasticmachine]

Original comment by @suyograo:

@djschny absolutely, there will be an abstraction layer (pluggable API layer) from LS agent side so the community can use something other than ES for centralized config management. As you mention, the OOTB feature will use ES. Also, the ES cluster to use for config management could be a different one than one that houses regular data, similar to marvel using a separate cluster.

[elasticmachine]

Original comment by @alexbrasetvik:

Love the effort, this is going to be key to make Logstash-aaS a thing on Cloud. :)

[aagupta1]

+1. Centralised config management for Beats would be very useful.

[cjcenizal]

@tbragin With the addition of Beats Central Management, do you feel like we can close this issue?

[tbragin]

@cjcenizal Yes, I think we can close it.