diff --git a/docs/en/ingest-management/elastic-agent/configuration/authentication/kerberos-shared-settings.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/authentication/kerberos-shared-settings.asciidoc new file mode 100644 index 0000000000..216e339b50 --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/configuration/authentication/kerberos-shared-settings.asciidoc @@ -0,0 +1,121 @@ +// These settings are shared across some inputs and outputs. + +// You can include this whole block, or individual settings +// tag::kerberos-all-settings[] + + +[cols="2*>. These settings are valid in both client and +server configurations. + +* <<{type}-client-ssl-options>> + +* <<{type}-server-ssl-options>> + +TIP: For more information about using certificates, refer to +<>. + +[id="{type}-common-ssl-options"] +.Common configuration options +[cols="2*>. To apply +changes after installation, you must modify the installed file. + +The following sections describe some settings you might need to configure to +run an {agent} standalone. For a full reference example, refer to the +<> file. + +The settings described here are available for standalone {agent}s. Settings for +{fleet}-managed agents are specified through the UI. You do not set them +explicitly in a configuration file. + +TIP: To get started quickly, you can use {fleet} to generate a standalone +configuration. You'll still need to deploy and manage the file, though. For more +information, see <>. + +//TODO: Explain the structure of the file, how it's used, etc. + +include::inputs/input-configuration.asciidoc[leveloffset=+1] + +include::outputs/output-configuration.asciidoc[leveloffset=+1] + +include::elastic-agent-standalone-logging.asciidoc[leveloffset=+1] + +include::elastic-agent-monitoring.asciidoc[leveloffset=+1] + +include::yaml/elastic-agent-reference-yaml.asciidoc[leveloffset=+1] diff --git a/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-monitoring.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-monitoring.asciidoc new file mode 100644 index 0000000000..9d5aaff376 --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-monitoring.asciidoc @@ -0,0 +1,34 @@ +[[elastic-agent-monitoring-configuration]] +[role="xpack"] += Configure monitoring for standalone {agent}s + +++++ +Monitoring +++++ + +{agent} monitors {beats} by default. To turn off or change monitoring +settings, set options under `agent.monitoring` in the `elastic-agent.yml` file. + +This example configures {agent} monitoring: + +[source,yaml] +---- +agent.monitoring: + # enabled turns on monitoring of running processes + enabled: true + # enables log monitoring + logs: true + # enables metrics monitoring + metrics: true + # specifies output to be used + use_output: monitoring +---- + +To turn off monitoring, set `agent.monitoring.enabled` to `false`. When set to +`false`, {beats} monitoring is turned off, and all other options in this section +are ignored. + +To enable monitoring, set `agent.monitoring.enabled` to `true`. Also set the +`logs` and `metrics` settings to control whether logs, metrics, or both are +collected. If neither setting is specified, monitoring is turned off. Set +`use_output` to specify the output to which monitoring events are sent. \ No newline at end of file diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-standalone-logging.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc similarity index 97% rename from docs/en/ingest-management/elastic-agent/elastic-agent-standalone-logging.asciidoc rename to docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc index dbf8ce8a67..364d785ed4 100644 --- a/docs/en/ingest-management/elastic-agent/elastic-agent-standalone-logging.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/elastic-agent-standalone-logging.asciidoc @@ -2,10 +2,16 @@ [role="xpack"] = Configure logging for standalone {agent}s +++++ +Logging +++++ + The Logging section of the `elastic-agent.yml` config file contains settings for configuring the logging output. The logging system can write logs to the `syslog`, `file`, `stderr`, `eventlog`, or rotate log files. If you do not explicitly configure logging, the `stderr` output is used. +This example configures {agent} logging: + ["source","yaml",subs="attributes"] ---- agent.logging.level: info @@ -21,7 +27,8 @@ agent.logging.files: [[elastic-agent-standalone-logging-settings]] == Logging configuration settings -You can specify the following settings in the Logging section of the `elastic-agent.yml` config file. +You can specify the following settings in the Logging section of the +`elastic-agent.yml` config file. [cols="2*Inputs +++++ + +The `inputs` section of the `elastic-agent.yml` file specifies how {agent} +locates and processes input data. + +By default {agent} collects system metrics, such as cpu, memory, network, and +filesystem metrics, and sends them to the default output. For example: + +[source,yaml] +------------------------------------------------------------------------------------- +inputs: + - type: system/metrics + data_stream.namespace: default + use_output: default + streams: + - metricset: cpu + data_stream.dataset: system.cpu + - metricset: memory + data_stream.dataset: system.memory + - metricset: network + data_stream.dataset: system.network + - metricset: filesystem + data_stream.dataset: system.filesystem +------------------------------------------------------------------------------------- + +If `use_output` is not specified, the `default` output is used. diff --git a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-configuration.asciidoc new file mode 100644 index 0000000000..2d40af0d2b --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-configuration.asciidoc @@ -0,0 +1,46 @@ +[[elastic-agent-output-configuration]] += Configure outputs for standalone {agent}s + +++++ +Outputs +++++ + +The `outputs` section of the `elastic-agent.yml` file specifies where to +send data. You can specify multiple outputs to pair specific inputs with +specific outputs. + +This example configures two outputs: `default` and `monitoring`: + +[source,yaml] +------------------------------------------------------------------------------------- +outputs: + default: + type: elasticsearch + hosts: [127.0.0.1:9200] + username: elastic + password: changeme + + monitoring: + type: elasticsearch + api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw + hosts: ["localhost:9200"] + ca_sha256: "7lHLiyp4J8m9kw38SJ7SURJP4bXRZv/BNxyyXkCcE/M=" +------------------------------------------------------------------------------------- + + +Notice that they use different authentication methods. The first one uses a +username and password pair, and the second one contains an API key. + +[NOTE] +============== +A default output configuration is required. +============== + +{agent} currently supports these outputs: + +* <> +* <> (standalone mode only) + +include::output-elasticsearch.asciidoc[leveloffset=+1] + +include::output-logstash.asciidoc[leveloffset=+1] diff --git a/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc new file mode 100644 index 0000000000..8a7fab532e --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc @@ -0,0 +1,575 @@ +:type: output-elasticsearch + +[[elasticsearch-output]] += Configure the {es} output + +++++ +Elasticsearch +++++ + +//QUESTION: Do we/are we going to support cloud ID? + +The {es} output sends events directly to {es} by using the {es} HTTP API. + +*Compatibility:* This output works with all compatible versions of {es}. See the +https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support +Matrix]. + +This example configures an {es} output called `default` in the +`elastic-agent.yml` file: + +//TODO: Provide a example that shows more of the settings users are likely to +//change. + +[source,yaml] +---- +outputs: + default: + type: elasticsearch + hosts: [127.0.0.1:9200] + username: elastic + password: changeme +---- + +== {es} output configuration settings + +The `elasticsearch` output type supports the following settings, grouped by +category. Many of these settings have sensible defaults that allow you to run +{agent} with minimal configuration. + +* <> + +* <> + +* <> + +* <> + +* <> + +[[output-elasticsearch-commonly-used-settings]] +== Commonly used settings + +[cols="2* + protocol: https + path: /elasticsearch +------------------------------------------------------------------------------ +<1> In this example, the {es} nodes are available at +`https://10.45.3.2:9220/elasticsearch` and +`https://10.45.3.1:9230/elasticsearch`. +// end::hosts-setting[] + +// ============================================================================= + +// tag::protocol-setting[] +| +[id="{type}-protocol-setting"] +`protocol` + +| (string) The name of the protocol {es} is reachable on. The options are: +`http` or `https`. The default is `http`. However, if you specify a URL for +`hosts`, the value of `protocol` is overridden by whatever scheme you specify in +the URL. +// end::protocol-setting[] + +// ============================================================================= + +// tag::proxy_disable-setting[] +| +[id="{type}-proxy_disable-setting"] +`proxy_disable` + +| (boolean) If set to `true`, all proxy settings, including `HTTP_PROXY` and +`HTTPS_PROXY` variables, are ignored. + +*Default:* `false` + +// end::proxy_disable-setting[] + +// ============================================================================= + +// tag::proxy_headers-setting[] +| +[id="{type}-proxy_headers-setting"] +`proxy_headers` + +| (string) Additional headers to send to proxies during CONNECT requests. + +// end::proxy_headers-setting[] + +// ============================================================================= +// tag::proxy_url-setting[] +| +[id="{type}-proxy_url-setting"] +`proxy_url` + +| (string) The URL of the proxy to use when connecting to the {es} servers. The +value may be either a complete URL or a `host[:port]`, in which case the `http` +scheme is assumed. If a value is not specified through the configuration file +then proxy environment variables are used. See the +https://golang.org/pkg/net/http/#ProxyFromEnvironment[Go documentation] +for more information about the environment variables. +// end::proxy_url-setting[] + +// ============================================================================= + +|=== + +[[output-elasticsearch-authentication-settings]] +== Authentication settings + +Settings for authenticating with {es}. + +When sending data to a secured cluster through the `elasticsearch` +output, {agent} can use any of the following authentication methods: + +* <> +* <> +* <> +* <> + +[[output-elasticsearch-basic-authentication-settings]] +=== Basic authentication credentials + +[source,yaml] +---- +outputs: + default: + type: elasticsearch + hosts: ["https://myEShost:9200"] + username: "your-username" + password: "your-password" +---- + +[cols="2*>. +// end::username-setting[] + +// ============================================================================= + +|=== + +[[output-elasticsearch-apikey-authentication-settings]] +=== Token-based (API key) authentication + +[source,yaml] +---- +outputs: + default: + type: elasticsearch + hosts: ["https://myEShost:9200"] + api_key: "KnR6yE41RrSowb0kQ0HWoA" +---- + +[cols="2*Logstash +++++ + +IMPORTANT: The {ls} output is currently only supported for {agent}s in +standalone mode. {fleet}-managed agents are not supported. + +The {ls} output uses an internal protocol to send events directly to {ls} over +TCP. {ls} provides additional parsing, transformation, and routing of data +collected by {agent}. + +*Compatibility:* This output works with all compatible versions of {ls}. Refer +to the https://www.elastic.co/support/matrix#matrix_compatibility[Elastic +Support Matrix]. + +This example configures a {ls} output called `default` in the +`elastic-agent.yml` file: + +//TODO: Provide a example that shows more of the settings users are likely to +//change. + +[source,yaml] +---- +outputs: + default: + type: logstash + hosts: ["127.0.0.1:5044"] <1> +---- +<1> The {ls} server and the port (`5044`) where {ls} is configured to listen for +incoming {agent} connections. + +To receive the events in {ls}, you also need to create a {ls} configuration pipeline. +The {ls} configuration pipeline listens for incoming {agent} connections, +processes received events, and then sends the events to {es}. + +The following example configures a {ls} pipeline that listens on port `5044` for +incoming {agent} connections and routes received events to {es}: + +[source,yaml] +---- +input { + elastic_agent { + port => 5044 + } +} + +output { + elasticsearch { + hosts => ["http://localhost:9200"] <1> + data_stream => "true" + } +} +---- +<1> The {es} server and the port (`9200`) where {es} is running. + +For more information about configuring {ls}, refer to +{logstash-ref}/configuration.html[Configuring {ls}] and +{logstash-ref}/plugins-inputs-elastic_agent.html[{agent} input plugin]. + +== {ls} output configuration settings + +The `logstash` output supports the following settings, grouped by category. +Many of these settings have sensible defaults that allow you to run {agent} with +minimal configuration. + +* <> + +* <> + +* <> + +[[output-logstash-commonly-used-settings]] +== Commonly used settings + +[cols="2*, |, ` ` (space character), ,, # - # Cannot start with -, _, + - # Cannot be . or .. + # Namespace name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes + # For index naming restrictions, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params data_stream.namespace: default use_output: default streams: - metricset: cpu - # The only two requirement are that it has only characters allowed in an Elasticsearch index name - # Index names must meet the following criteria: - # Lowercase only - # Cannot include \, /, *, ?, ", <, >, |, ` ` (space character), ,, # - # Cannot start with -, _, + - # Cannot be . or .. + # Dataset name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes data_stream.dataset: system.cpu - metricset: memory data_stream.dataset: system.memory @@ -116,8 +113,17 @@ inputs: # logs: false # # enables metrics monitoring # metrics: false - -# # Allow fleet to reload his configuration locally on disk. +# # exposes agent metrics using http, by default sockets and named pipes are used +# http: +# # enables http endpoint +# enabled: false +# # The HTTP endpoint will bind to this hostname, IP address, unix socket or named pipe. +# # When using IP addresses, it is recommended to only use localhost. +# host: localhost +# # Port on which the HTTP endpoint will bind. Default is 0 meaning feature is disabled. +# port: 6791 + +# # Allow fleet to reload its configuration locally on disk. # # Notes: Only specific process configuration will be reloaded. # agent.reload: # # enabled configure the Elastic Agent to reload or not the local configuration. @@ -200,3 +206,48 @@ agent.logging.to_stderr: true # information. Recommended to use in combination with `logging.json=true` # Defaults to false. #agent.logging.ecs: false + +# Providers + +# Providers supply the key/values pairs that are used for variable substitution +# and conditionals. Each provider's keys are automatically prefixed with the name +# of the provider. + +#providers: + +# Agent provides information about the running agent. +# agent: +# enabled: true + +# Docker provides inventory information from Docker. +# docker: +# enabled: true +# host: "unix:///var/run/docker.sock" +# cleanup_timeout: 60 + +# Env providers information about the running environment. +# env: +# enabled: true + +# Host provides information about the current host. +# host: +# enabled: true + +# Local provides custom keys to use as variable. +# local: +# enabled: true +# vars: +# foo: bar + +# Local dynamic allows you to define multiple key/values to generate multiple configurations. +# local_dynamic: +# enabled: true +# items: +# - vars: +# my_var: key1 +# - vars: +# my_var: key2 +# - vars: +# my_var: key3 + + diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent-configuration.asciidoc deleted file mode 100644 index bae8e6790f..0000000000 --- a/docs/en/ingest-management/elastic-agent/elastic-agent-configuration.asciidoc +++ /dev/null @@ -1,193 +0,0 @@ -[[elastic-agent-configuration]] -[role="xpack"] -= Agent policy settings - -The policy settings for {fleet}-managed agents are specified through the UI. -You do not set them explicitly in a configuration file. - -For standalone agents, you need to configure settings in the `elastic-agent.yml` -file. Prior to installation, edit the file located in the extracted {agent} -package. After installation, edit the file located in the directory -described in <>. - -TIP: To get started quickly, you can use {fleet} to generate a standalone -configuration. For more information, see <>. - -The following sections describe some settings you might need to configure to -run an {agent} standalone. For a full reference example, refer to the -<> file. - - -[discrete] -[[elastic-agent-output-configuration]] -== Output settings - -Output settings specify where to send data. You can specify multiple outputs to -pair specific inputs with specific outputs. - -{agent} currently supports these outputs: - -* <> -* <> (standalone mode only) - -[discrete] -[[elasticsearch-output]] -=== {es} output - -The {es} output sends events directly to {es} by using the {es} HTTP API. - -*Compatibility:* This output works with all compatible versions of {es}. See the -https://www.elastic.co/support/matrix#matrix_compatibility[Elastic Support -Matrix]. - -This example configures two {es} outputs: `default` and `monitoring`: - -[source,yaml] -------------------------------------------------------------------------------------- -outputs: - default: - type: elasticsearch - hosts: [127.0.0.1:9200] - username: elastic - password: changeme - - monitoring: - type: elasticsearch - api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw - hosts: ["localhost:9200"] - ca_sha256: "7lHLiyp4J8m9kw38SJ7SURJP4bXRZv/BNxyyXkCcE/M=" -------------------------------------------------------------------------------------- - -Notice that they use different authentication methods. The first one uses a -username and password pair, and the second one contains an API key. - -[NOTE] -============== -A default output configuration is required. -============== - -[discrete] -[[logstash-output]] -= {ls} output - -IMPORTANT: The {ls} output is currently only supported for {agent}s in -standalone mode. {fleet}-managed agents are not supported. - -The {ls} output uses an internal protocol to send events directly to {ls} over -TCP. {ls} provides additional parsing, transformation, and routing of data -collected by {agent}. - -*Compatibility:* This output works with all compatible versions of {ls}. Refer -to the https://www.elastic.co/support/matrix#matrix_compatibility[Elastic -Support Matrix]. - -This example configures a {ls} output called `default` in the -`elastic-agent.yml` file: - -[source,yaml] ----- -outputs: - default: - type: logstash - hosts: ["127.0.0.1:5044"] <1> ----- -<1> The {ls} server and the port (`5044`) where {ls} is configured to listen for -incoming {agent} connections. - -To send events to {ls}, you also need to create a {ls} configuration pipeline. -The {ls} configuration pipeline listens for incoming {agent} connections, -processes received events, and then sends the events to {es}. - -The following example configures a {ls} pipeline that listens on port `5044` for -incoming {agent} connections and routes received events to {es}: - -[source,yaml] ----- -input { - elastic_agent { - port => 5044 - } -} -output { - elasticsearch { - hosts => ["http://localhost:9200"] <1> - data_stream => "true" - } -} ----- -<1> The {es} server and the port (`9200`) where {es} is running. - -For more information about configuring {ls}, refer to -{logstash-ref}/configuration.html[Configuring {ls}] and -{logstash-ref}/plugins-inputs-elastic_agent.html[{agent} input plugin]. - -[discrete] -[[elastic-agent-monitoring-configuration]] -== {beats} monitoring settings - -{agent} monitors {beats} by default. To disable or change monitoring -settings, set options under `agent.monitoring`: - -[source,yaml] -------------------------------------------------------------------------------------- -agent.monitoring: - # enabled turns on monitoring of running processes - enabled: true - # enables log monitoring - logs: true - # enables metrics monitoring - metrics: true - # specifies output to be used - use_output: monitoring -------------------------------------------------------------------------------------- - - -To disable monitoring, set `agent.monitoring.enabled` to `false`. When set to -`false`, {beats} monitoring is turned off, and all other options in this section -are ignored. - -To enable monitoring, set `agent.monitoring.enabled` to `true`. Also set the -`logs` and `metrics` settings to control whether logs, metrics, or both are -collected. If neither setting is specified, monitoring is disabled. Set -`use_output` to specify the output to which monitoring events are sent. - -[discrete] -[[elastic-agent-input-configuration]] -== Input settings - -By default {agent} collects system metrics, such as cpu, memory, network, and -filesystem metrics, and sends them to the default output. For example: - - -[source,yaml] -------------------------------------------------------------------------------------- -inputs: - - type: system/metrics - data_stream.namespace: default - use_output: default - streams: - - metricset: cpu - data_stream.dataset: system.cpu - - metricset: memory - data_stream.dataset: system.memory - - metricset: network - data_stream.dataset: system.network - - metricset: filesystem - data_stream.dataset: system.filesystem -------------------------------------------------------------------------------------- - -If `use_output` is not specified, the `default` output is used. - -[discrete] -[[elastic-agent-reference-yaml]] -== Reference yaml - -The {agent} installation includes an `elastic-agent.reference.yml` file that -describes settings available in a standalone configuration. - -The contents of the file are included here for your convenience. - -[source,yaml] ----- -include::elastic-agent-reference-yaml.yml[] ----- diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent.asciidoc index 7218f9e99b..c708c85ed6 100644 --- a/docs/en/ingest-management/elastic-agent/elastic-agent.asciidoc +++ b/docs/en/ingest-management/elastic-agent/elastic-agent.asciidoc @@ -20,11 +20,9 @@ To learn how to install, configure, and run your {agent}s, see: * <> * <> * <> -* <> * <> * <> * <> -* <> include::install-elastic-agent.asciidoc[leveloffset=+1] @@ -32,8 +30,6 @@ include::uninstall-elastic-agent.asciidoc[leveloffset=+1] include::run-elastic-agent-standalone.asciidoc[leveloffset=+1] -include::elastic-agent-configuration.asciidoc[leveloffset=+2] - include::elastic-agent-container.asciidoc[leveloffset=+1] include::running-on-kubernetes-managed-by-fleet.asciidoc[leveloffset=+1] @@ -57,5 +53,3 @@ include::elastic-agent-capabilities.asciidoc[leveloffset=+1] include::configuration/env/container-envs.asciidoc[leveloffset=+1] include::elastic-agent-logging.asciidoc[leveloffset=+1] - -include::elastic-agent-standalone-logging.asciidoc[leveloffset=+1] diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 837d99d2b7..107336e22e 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -40,12 +40,14 @@ include::getting-started-traces.asciidoc[leveloffset=+1] include::elastic-agent/elastic-agent.asciidoc[leveloffset=+1] +include::agent-policies.asciidoc[leveloffset=+1] + +include::elastic-agent/configuration/elastic-agent-configuration.asciidoc[leveloffset=+1] + include::fleet/fleet-settings.asciidoc[leveloffset=+1] include::fleet/fleet-server.asciidoc[leveloffset=+1] -include::agent-policies.asciidoc[leveloffset=+1] - include::security/enrollment-tokens.asciidoc[leveloffset=+1] include::security/certificates.asciidoc[leveloffset=+1]