diff --git a/docs/reference/monitoring/configuring-metricbeat.asciidoc b/docs/reference/monitoring/configuring-metricbeat.asciidoc new file mode 100644 index 0000000000000..3d4a37861f1ae --- /dev/null +++ b/docs/reference/monitoring/configuring-metricbeat.asciidoc @@ -0,0 +1,195 @@ +[role="xpack"] +[testenv="gold"] +[[configuring-metricbeat]] +=== Monitoring {es} with {metricbeat} + +beta[] In 6.5 and later, you can use {metricbeat} to collect data about {es} +and ship it to the monitoring cluster, rather than routing it through exporters +as described in <>. + +image::monitoring/images/metricbeat.png[Example monitoring architecture] + +To learn about monitoring in general, see +{stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. + +. Enable the collection of monitoring data. Set +`xpack.monitoring.collection.enabled` to `true` on the production cluster. + ++ +-- +For example, you can use the following APIs to review and change this setting: + +[source,js] +---------------------------------- +GET _cluster/settings + +PUT _cluster/settings +{ + "persistent": { + "xpack.monitoring.collection.enabled": true + } +} +---------------------------------- +// CONSOLE + +For more information, see <> and <>. +-- + +. Disable the default collection of {es} monitoring metrics. Set +`xpack.monitoring.elasticsearch.collection.enabled` to `false` on the production +cluster. + ++ +-- +For example, you can use the following API to change this setting: + +[source,js] +---------------------------------- +PUT _cluster/settings +{ + "persistent": { + "xpack.monitoring.elasticsearch.collection.enabled": false + } +} +---------------------------------- +// CONSOLE + +Leave `xpack.monitoring.enabled` set to its default value (`true`). +-- + +. On each {es} node in the production cluster: + +.. {metricbeat-ref}/metricbeat-installation.html[Install {metricbeat}]. + +.. Enable the {es} module in {metricbeat}. + ++ +-- +For example, to enable the default configuration in the `modules.d` directory, +run the following command: + +["source","sh",subs="attributes,callouts"] +---------------------------------------------------------------------- +metricbeat modules enable elasticsearch +---------------------------------------------------------------------- + +For more information, see +{metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and +{metricbeat-ref}/metricbeat-module-elasticsearch.html[{es} module]. +-- + +.. Configure the {es} module in {metricbeat}. + ++ +-- +You must specify the following settings in the `modules.d/elasticsearch.yml` file: + +[source,yaml] +---------------------------------- +- module: elasticsearch + metricsets: + - ccr + - cluster_stats + - index + - index_recovery + - index_summary + - ml_job + - node_stats + - shard + period: 10s + hosts: ["http://localhost:9200"] <1> + xpack.enabled: true +---------------------------------- +<1> This setting identifies the host and port number that are used to access {es}. +-- + +.. If {security} is enabled, you must also provide a user ID and password so that +{metricbeat} can collect metrics successfully. + +... Create or identify a user that you want to use to collect the metrics. ++ +-- +TIP: There is a `remote_monitoring_user` built-in user that grants the privileges +necessary for {metricbeat} to monitor {stack} products. See +{stack-ov}/built-in-users.html[Built-in users]. + +Alternatively, you can choose a different user and give them the +`remote_monitoring_collector` {stack-ov}/built-in-roles.html[built-in role]. +-- + +... Add the `username` and `password` settings to the {es} module configuration +file. ++ +-- +For example, add the following settings in the `modules.d/kibana.yml` file: + +[source,yaml] +---------------------------------- +- module: elasticsearch + ... + username: remote_monitoring_user + password: YOUR_PASSWORD +---------------------------------- +-- + +.. If you configured {es} to use <>, +you must access it via HTTPS. For example, use a `hosts` setting like +`https://localhost:9200` in the `modules.d/elasticsearch.yml` file. + +.. Identify where to send the monitoring data. + ++ +-- +TIP: In production environments, we strongly recommend using a separate cluster +(referred to as the _monitoring cluster_) to store the data. Using a separate +monitoring cluster prevents production cluster outages from impacting your +ability to access your monitoring data. It also prevents monitoring activities +from impacting the performance of your production cluster. + +For example, specify the {es} output information in the {metricbeat} +configuration file (`metricbeat.yml`): + +[source,yaml] +---------------------------------- +output.elasticsearch: + hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1> +---------------------------------- +<1> In this example, the data is stored on a monitoring cluster with nodes +`es-mon-1` and `es-mon-2`. + +For more information about these configuration options, see +{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output]. +-- + +.. If {security} is enabled on the monitoring cluster, you must provide a valid +user ID and password so that {metricbeat} can send metrics successfully. + +... Create or identify a user that you want to use to send the metrics. ++ +-- +TIP: There is a `remote_monitoring_user` built-in user that grants the privileges +necessary for {metricbeat} to monitor {stack} products. See +{stack-ov}/built-in-users.html[Built-in users]. + +Alternatively, you can choose a different user and give them the +`remote_monitoring_agent` {stack-ov}/built-in-roles.html[built-in role]. +-- + +... Add the `username` and `password` settings to the {es} output information in +the {metricbeat} configuration file (`metricbeat.yml`): ++ +-- +[source,yaml] +---------------------------------- +output.elasticsearch: + ... + username: remote_monitoring_user + password: YOUR_PASSWORD +---------------------------------- +-- + +.. If you configured the monitoring cluster to use +<>, you must access it via +HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200` in the +`metricbeat.yml` file. + +. <>. + +. {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}]. + +. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. diff --git a/docs/reference/monitoring/configuring-monitoring.asciidoc b/docs/reference/monitoring/configuring-monitoring.asciidoc index 6708b791036a9..81a9cce4f12ec 100644 --- a/docs/reference/monitoring/configuring-monitoring.asciidoc +++ b/docs/reference/monitoring/configuring-monitoring.asciidoc @@ -6,18 +6,27 @@ Configuring monitoring ++++ -By default, {monitoring} is enabled but data collection is disabled. Advanced -monitoring settings enable you to control how frequently data is collected, -configure timeouts, and set the retention period for locally-stored monitoring -indices. You can also adjust how monitoring data is displayed. +If you enable the collection of monitoring data in your cluster, you can +optionally collect metrics about {es}. By default, {monitoring} is enabled but +data collection is disabled. + +The following method involves sending the metrics to the monitoring cluster by +using exporters. For an alternative method, see <>. + +Advanced monitoring settings enable you to control how frequently data is +collected, configure timeouts, and set the retention period for locally-stored +monitoring indices. You can also adjust how monitoring data is displayed. + +To learn about monitoring in general, see +{stack-ov}/xpack-monitoring.html[Monitoring the {stack}]. . To collect monitoring data about your {es} cluster: .. Verify that the `xpack.monitoring.enabled`, `xpack.monitoring.collection.enabled`, and `xpack.monitoring.elasticsearch.collection.enabled` settings are `true` on each -node in the cluster. By default xpack.monitoring.collection.enabled is disabled -(`false`), and that overrides xpack.monitoring.elasticsearch.collection.enabled, +node in the cluster. By default `xpack.monitoring.collection.enabled` is disabled +(`false`), and that overrides `xpack.monitoring.elasticsearch.collection.enabled`, which defaults to being enabled (`true`). Both settings can be set dynamically at runtime. For more information, see <>. @@ -69,8 +78,9 @@ see {stack-ov}/how-monitoring-works.html[How Monitoring Works]. a dedicated monitoring cluster: .. Create a user on the monitoring cluster that has the -{xpack-ref}/built-in-roles.html#built-in-roles-remote-monitoring-agent[`remote_monitoring_agent` built-in role]. For example, the following request -creates a `remote_monitor` user that has the `remote_monitoring_agent` role: +{stack-ov}/built-in-roles.html#built-in-roles-remote-monitoring-agent[`remote_monitoring_agent` built-in role]. +For example, the following request creates a `remote_monitor` user that has the +`remote_monitoring_agent` role: + -- [source, sh] @@ -87,12 +97,17 @@ POST /_xpack/security/user/remote_monitor -- .. On each node in the cluster that is being monitored, configure the `http` -exporter to use the appropriate credentials when data is shipped to the monitoring cluster. +exporter to use the appropriate credentials when data is shipped to the +monitoring cluster. + -- -If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol in the `host` setting. You must also include the CA certificate in each node's trusted certificates in order to verify the identities of the nodes in the monitoring cluster. +If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol +in the `host` setting. You must also include the CA certificate in each node's +trusted certificates in order to verify the identities of the nodes in the +monitoring cluster. -The following example specifies the location of the PEM encoded certificate with the `certificate_authorities` setting: +The following example specifies the location of the PEM encoded certificate with +the `certificate_authorities` setting: [source,yaml] -------------------------------------------------- @@ -144,5 +159,8 @@ stored, that is to say the monitoring cluster. To grant all of the necessary per . Optional: <>. +. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. + +include::configuring-metricbeat.asciidoc[] include::indices.asciidoc[] include::{es-repo-dir}/settings/monitoring-settings.asciidoc[] \ No newline at end of file diff --git a/docs/reference/monitoring/images/metricbeat.png b/docs/reference/monitoring/images/metricbeat.png new file mode 100644 index 0000000000000..bf6434dc4b40c Binary files /dev/null and b/docs/reference/monitoring/images/metricbeat.png differ diff --git a/docs/reference/monitoring/indices.asciidoc b/docs/reference/monitoring/indices.asciidoc index 658ac389ae80a..34cbced1c4332 100644 --- a/docs/reference/monitoring/indices.asciidoc +++ b/docs/reference/monitoring/indices.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[config-monitoring-indices]] -=== Configuring Indices for Monitoring +=== Configuring indices for monitoring <> are used to configure the indices that store the monitoring data collected from a cluster. diff --git a/docs/reference/settings/monitoring-settings.asciidoc b/docs/reference/settings/monitoring-settings.asciidoc index a039084412cda..92c51772720c9 100644 --- a/docs/reference/settings/monitoring-settings.asciidoc +++ b/docs/reference/settings/monitoring-settings.asciidoc @@ -1,8 +1,8 @@ [role="xpack"] [[monitoring-settings]] -=== Monitoring Settings in Elasticsearch +=== Monitoring settings in Elasticsearch ++++ -Monitoring Settings +Monitoring settings ++++ By default, monitoring is enabled but data collection is disabled. To enable @@ -43,17 +43,14 @@ to `true`. Its default value is `false`. The `xpack.monitoring.collection` settings control how data is collected from your Elasticsearch nodes. -`xpack.monitoring.collection.enabled`:: +`xpack.monitoring.collection.enabled`:: (<>) added[6.3.0] Set to `true` to enable the collection of monitoring data. When this setting is `false` (default), {es} monitoring data is not collected and all monitoring data from other sources such as {kib}, Beats, and Logstash is ignored. -+ -You can update this setting through the -<>. -`xpack.monitoring.collection.interval`:: +`xpack.monitoring.collection.interval`:: (<>) Setting to `-1` to disable data collection is no longer supported beginning with 7.0.0. deprecated[6.3.0, Use `xpack.monitoring.collection.enabled` set to @@ -62,35 +59,26 @@ Setting to `-1` to disable data collection is no longer supported beginning with Controls how often data samples are collected. Defaults to `10s`. If you modify the collection interval, set the `xpack.monitoring.min_interval_seconds` option in `kibana.yml` to the same value. -+ -You can update this setting through the -<>. -`xpack.monitoring.elasticsearch.collection.enabled`:: +`xpack.monitoring.elasticsearch.collection.enabled`:: (<>) Controls whether statistics about your {es} cluster should be collected. Defaults to `true`. This is different from xpack.monitoring.collection.enabled, which allows you to enable or disable all monitoring collection. However, this setting simply disables the collection of Elasticsearch data while still allowing other data (e.g., Kibana, Logstash, Beats, or APM Server monitoring data) to pass through this cluster. -+ -You can update this setting through the -<>. `xpack.monitoring.collection.cluster.stats.timeout`:: Sets the timeout for collecting the cluster statistics. Defaults to `10s`. -`xpack.monitoring.collection.indices`:: +`xpack.monitoring.collection.indices`:: (<>) Controls which indices Monitoring collects data from. Defaults to all indices. Specify the index names as a comma-separated list, for example `test1,test2,test3`. Names can include wildcards, for example `test*`. You can explicitly include or exclude indices by prepending `+` to include the index, or `-` to exclude the index. For example, to include all indices that start with `test` except `test3`, you could specify `+test*,-test3`. -+ -You can update this setting through the -<>. `xpack.monitoring.collection.index.stats.timeout`::