From 3f22c754a8cde39e8368530ac6a702eb4308f116 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 26 Oct 2023 18:05:19 +0200 Subject: [PATCH 01/37] Redo URLs --- content/en/docs/collector/configuration.md | 51 +++++++++++----------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index a7cca798af5f..ff16a1b6d016 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -1,6 +1,7 @@ --- title: Configuration weight: 20 +description: Learn how to configure the Collector to suit your needs # prettier-ignore cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca loglevel OIDC oidc otlphttp pprof prodevent prometheusremotewrite servicegraph spanevents spanmetrics upsert zpages --- @@ -330,25 +331,25 @@ processors: ## Exporters {#exporters} -An exporter, which can be push or pull based, is how you send data to one or -more backends/destinations. Exporters may support one or more +Exporters send data to one or more backends or destinations. Exporters can be +pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -The `exporters:` section is how exporters are configured. Exporters may come -with default settings, but many require configuration to specify at least the -destination and security settings. Any configuration for an exporter must be -done in this section. Configuration parameters specified for which the exporter -provides a default configuration are overridden. +The `exporters:` section contains exporters configuration. Most exporters +require configuration to specify at least the destination, as well as +security settings, like authentication tokens or TLS certificates. Any setting +you specify overrides the default values, if present. -> Configuring an exporter does not enable it. Exporters are enabled via -> pipelines within the [service](#service) section. +> Configuring an exporter does not enable it. Exporters are enabled by +> adding them to the appropriate pipelines within the +> [service](#service) section. -One or more exporters must be configured. By default, no exporters are -configured. A basic example of exporters is provided below. Certain exporter -configurations require x.509 certificates to be created in order to be secure, -as described in [setting up certificates](#setting-up-certificates). +The Collector requires one or more exporters. The following example shows +various exporters. Notice that some exporters require x.509 certificates in +order to establish secure connections, as described in +[setting up certificates](#setting-up-certificates). -> For detailed exporter configuration, see the +> For more information on exporter configuration, see the > [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). ```yaml @@ -359,7 +360,7 @@ exporters: # Data sources: traces otlp/jaeger: - endpoint: jaeger-all-in-one:4317 + endpoint: jaeger.example.com:4317 tls: cert_file: cert.pem key_file: cert-key.pem @@ -386,24 +387,24 @@ exporters: # Data sources: traces, metrics otlphttp: - endpoint: https://example.com:4318 + endpoint: https://otlp.example.com:4318 # Data sources: metrics prometheus: - endpoint: prometheus:8889 + endpoint: localhost:8889 namespace: default # Data sources: metrics prometheusremotewrite: - endpoint: http://some.url:9411/api/prom/push - # For official Prometheus (e.g. running via Docker) - # endpoint: 'http://prometheus:9090/api/v1/write' + endpoint: http://prometheus.example.com:9411/api/prom/push + # When using the official Prometheus (running via Docker) + # endpoint: 'http://prometheus:9090/api/v1/write', add: # tls: # insecure: true # Data sources: traces zipkin: - endpoint: http://localhost:9411/api/v2/spans + endpoint: http://zipkin.example.com:9411/api/v2/spans ``` ## Connectors {#connectors} @@ -602,15 +603,15 @@ exporters: ### Proxy Support -Exporters that leverage the `net/http` package (all do today) respect the -following proxy environment variables: +Exporters that leverage the [net/http](https://pkg.go.dev/net/http) package +respect the following proxy environment variables: - HTTP_PROXY - HTTPS_PROXY - NO_PROXY -If set at Collector start time then exporters, regardless of protocol, will or -will not proxy traffic as defined by these environment variables. +If set at Collector start time, exporters, regardless of protocol, will proxy +traffic or bypass proxy traffic as defined by these environment variables. ### Authentication From 778d19d7e685a1012542bac3947cd05b0bdc31cd Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 26 Oct 2023 18:07:51 +0200 Subject: [PATCH 02/37] Format fixes --- content/en/docs/collector/configuration.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index ff16a1b6d016..6ab88396f297 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -335,14 +335,13 @@ Exporters send data to one or more backends or destinations. Exporters can be pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -The `exporters:` section contains exporters configuration. Most exporters -require configuration to specify at least the destination, as well as -security settings, like authentication tokens or TLS certificates. Any setting -you specify overrides the default values, if present. +The `exporters:` section contains exporters configuration. Most exporters +require configuration to specify at least the destination, as well as security +settings, like authentication tokens or TLS certificates. Any setting you +specify overrides the default values, if present. -> Configuring an exporter does not enable it. Exporters are enabled by -> adding them to the appropriate pipelines within the -> [service](#service) section. +> Configuring an exporter does not enable it. Exporters are enabled by adding +> them to the appropriate pipelines within the [service](#service) section. The Collector requires one or more exporters. The following example shows various exporters. Notice that some exporters require x.509 certificates in From 90fc621a5645240f04eba9404c50af84df98ba21 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 26 Oct 2023 18:17:16 +0200 Subject: [PATCH 03/37] Update lint --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 6ab88396f297..e2649c8bbfbd 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -602,7 +602,7 @@ exporters: ### Proxy Support -Exporters that leverage the [net/http](https://pkg.go.dev/net/http) package +Exporters that leverage the [`net/http`](https://pkg.go.dev/net/http) package respect the following proxy environment variables: - HTTP_PROXY From e42180c6538e07ef63fe118273cfb85305950920 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 11:15:37 +0100 Subject: [PATCH 04/37] Further edits --- content/en/docs/collector/configuration.md | 374 +++++++++++---------- 1 file changed, 190 insertions(+), 184 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index e2649c8bbfbd..2c00240ec008 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -8,15 +8,18 @@ cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca l -Familiarity with the following pages is assumed: +You can configure the Collector to suit your observability needs. Before you +learn how Collector configuration works, familiarize yourself with the following +content: - [Data collection concepts][dcc] in order to understand the repositories applicable to the OpenTelemetry Collector. - [Security guidance](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md) -## Basics +## Configuration structure -The Collector consists of four components that access telemetry data: +The structure of any Collector configuration file consists of four classes of +pipeline components that access telemetry data: - [Receivers](#receivers) @@ -27,15 +30,16 @@ The Collector consists of four components that access telemetry data: - [Connectors](#connectors) -These components once configured must be enabled via pipelines within the -[service](#service) section. +After each pipeline component is configured you must enable it using the +pipelines within the [service](#service) section of the configuration file. -Secondarily, there are [extensions](#extensions), which provide capabilities -that can be added to the Collector, but which do not require direct access to -telemetry data and are not part of pipelines. They are also enabled within the -[service](#service) section. +Besides pipeline components you can also configure [extensions](#extensions), +which provide capabilities that can be added to the Collector, such as +diagnostic tools. Extensions don't require direct access to telemetry data and +are enabled through the [service](#service) section. -An example configuration would look like: +The following is an example of Collector configuration with a receiver, a +processor, an exporter, and three extensions: ```yaml receivers: @@ -73,10 +77,10 @@ service: exporters: [otlp] ``` -Note that receivers, processors, exporters and/or pipelines are defined via -component identifiers in `type[/name]` format (e.g. `otlp` or `otlp/2`). -Components of a given type can be defined more than once as long as the -identifiers are unique. For example: +Note that receivers, processors, exporters and pipelines are defined through +component identifiers following the `type[/name]` format, for example `otlp` or +`otlp/2`. You can define components of a given type more than once as long as +the identifiers are unique. For example: ```yaml receivers: @@ -126,7 +130,7 @@ service: ``` The configuration can also include other files, causing the Collector to merge -the two files in a single in-memory representation of the YAML configuration: +them in a single in-memory representation of the YAML configuration: ```yaml receivers: @@ -175,25 +179,20 @@ service: ## Receivers {#receivers} -A receiver, which can be push or pull based, is how data gets into the -Collector. Receivers may support one or more -[data sources](/docs/concepts/signals/). +Receivers collect telemetry from one or more sources. They can be pull or push +based, and may support one or more [data sources](/docs/concepts/signals/). -The `receivers:` section is how receivers are configured. Many receivers come -with default settings so simply specifying the name of the receiver is enough to -configure it (for example, `zipkin:`). If configuration is required or a user -wants to change the default configuration then such configuration must be -defined in this section. Configuration parameters specified for which the -receiver provides a default configuration are overridden. +Receivers are configured in the `receivers:` section. Many receivers come with +default settings, so that specifying the name of the receiver is enough to +configure it. If you need to configure a receiver or want to change the default +configuration, you can do so in this section. Any setting you specify overrides +the default values, if present. -> Configuring a receiver does not enable it. Receivers are enabled via pipelines -> within the [service](#service) section. +> Configuring a receiver does not enable it. Receivers are enabled by adding +> them to the appropriate pipelines within the [service](#service) section. -One or more receivers must be configured. By default, no receivers are -configured. A basic example of receivers is provided below. - -> For detailed receiver configuration, see the -> [receiver README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md). +The Collector requires one or more receivers. The following example shows +various receivers in the same configuration file: ```yaml receivers: @@ -248,28 +247,31 @@ receivers: zipkin: ``` +> For detailed receiver configuration, see the +> [receiver README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md). + ## Processors {#processors} -Processors are run on data between being received and being exported. Processors -are optional though -[some are recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). +Processors take the data collected by receivers and modify or transform it +before sending it to the exporters. Data processing happens according to rules +or settings defined for each processor, which might include filtering, dropping, +renaming, or recalculating telemetry, among other operations. -The `processors:` section is how processors are configured. Processors may come -with default settings, but many require configuration. Any configuration for a -processor must be done in this section. Configuration parameters specified for -which the processor provides a default configuration are overridden. +Processors are optional, although some +[are recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). -> Configuring a processor does not enable it. Processors are enabled via -> pipelines within the [service](#service) section. +You can configure processors using the `processors:` section of the Collector +configuration file. Some processors may require security settings. Any setting +you specify overrides the default values, if present. -A basic example of the default processors is provided below. The full list of -processors can be found by combining the list found -[here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor) -and -[here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor). +> Configuring a processor does not enable it. Processors are enabled by adding +> them to the appropriate pipelines within the [service](#service) section. -> For detailed processor configuration, see the -> [processor README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md). +The following example shows several default processors in the same configuration +file. You can find the full list of processors by combining the list from +[opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor) +and the list from +[opentelemetry-collector](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor). ```yaml processors: @@ -329,6 +331,9 @@ processors: separator: '::' ``` +> For detailed processor configuration, see the +> [processor README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md). + ## Exporters {#exporters} Exporters send data to one or more backends or destinations. Exporters can be @@ -344,12 +349,7 @@ specify overrides the default values, if present. > them to the appropriate pipelines within the [service](#service) section. The Collector requires one or more exporters. The following example shows -various exporters. Notice that some exporters require x.509 certificates in -order to establish secure connections, as described in -[setting up certificates](#setting-up-certificates). - -> For more information on exporter configuration, see the -> [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). +various exporters in the same configuration file: ```yaml exporters: @@ -406,30 +406,41 @@ exporters: endpoint: http://zipkin.example.com:9411/api/v2/spans ``` +Notice that some exporters require x.509 certificates in order to establish +secure connections, as described in +[setting up certificates](#setting-up-certificates). + +> For more information on exporter configuration, see the +> [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). + ## Connectors {#connectors} -A connector is both an exporter and receiver. As the name suggests a Connector -connects two pipelines: It consumes data as an exporter at the end of one -pipeline and emits data as a receiver at the start of another pipeline. It may -consume and emit data of the same data type, or of different data types. A -connector may generate and emit data to summarize the consumed data, or it may -simply replicate or route data. +Connectors join two pipelines, acting as both exporter and receiver. A connector +consumes data as an exporter at the end of one pipeline and emits data as a +receiver at the beginning of another pipeline. The data consumed and emitted may +be of the same type or of different data types. You can use connectors to +summarize consumed data, replicate it, or route it. -The `connectors:` section is how connectors are configured. +You can configure one or more connectors using the `connectors:` section of the +Collector configuration file. By default, no connectors are configured. Each +type of connector is designed to work with one or more pairs of data types and +may only be used to connect pipelines accordingly. -> Configuring a connectors does not enable it. Connectors are enabled via +> Configuring a connector doesn't enable it. Connectors are enabled through > pipelines within the [service](#service) section. -One or more connectors may be configured. By default, no connectors are -configured. A basic example of connectors is provided below. - -> For detailed connector configuration, see the -> [connector README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md). +The following example shows the `count` connector and how it's configured in the +`pipelines:` section. Notice that the connector acts as an exporter for traces +and as a receiver for metrics, connecting both pipelines: ```yaml -connectors: - forward: +receivers: + foo: + +exporters: + bar: +connectors: count: spanevents: my.prod.event.count: @@ -438,48 +449,36 @@ connectors: - 'attributes["env"] == "prod"' - 'name == "prodevent"' - spanmetrics: - histogram: - explicit: - buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms] - dimensions: - - name: http.method - default: GET - - name: http.status_code - dimensions_cache_size: 1000 - aggregation_temporality: 'AGGREGATION_TEMPORALITY_CUMULATIVE' - - servicegraph: - latency_histogram_buckets: [1, 2, 3, 4, 5] - dimensions: - - dimension-1 - - dimension-2 - store: - ttl: 1s - max_items: 10 +service: + pipelines: + traces: + receivers: [foo] + exporters: [count] + metrics: + receivers: [count] + exporters: [bar] ``` +> For detailed connector configuration, see the +> [connector README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md). + ## Extensions -Extensions are available primarily for tasks that do not involve processing -telemetry data. Examples of extensions include health monitoring, service -discovery, and data forwarding. Extensions are optional. +Extensions are optional components that expand the capabilities of the Collector +to accomplish tasks not directly involved with processing telemetry data. For +example, you can add extensions for Collector health monitoring, service +discovery, or data forwarding, among others. -The `extensions:` section is how extensions are configured. Many extensions come -with default settings so simply specifying the name of the extension is enough -to configure it (for example, `health_check:`). If configuration is required or -a user wants to change the default configuration then such configuration must be -defined in this section. Configuration parameters specified for which the -extension provides a default configuration are overridden. +You can configure extensions through the `extensions:` section of the Collector +configuration file. Most extensions come with default settings, so you can +configure them just by specifying the name of the extension. Any setting you +specify overrides the default values, if present. -> Configuring an extension does not enable it. Extensions are enabled within the +> Configuring an extension doesn't enable it. Extensions are enabled within the > [service](#service) section. -By default, no extensions are configured. A basic example of extensions is -provided below. - -> For detailed extension configuration, see the -> [extension README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). +By default, no extensions are configured. The following example shows several +extensions configured in the same file: ```yaml extensions: @@ -490,21 +489,26 @@ extensions: size_mib: 512 ``` -## Service +> For detailed extension configuration, see the +> [extension README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). + +## Service section The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, exporters, and extensions sections. If a component is configured, but not -defined within the service section then it is not enabled. The service section -consists of three sub-sections: +defined within the service section, then it's not enabled. + +The service section consists of three subsections: -- extensions -- pipelines -- telemetry +- `extensions:` +- `pipelines:` +- `telemetry:` ### Extensions {#service-extensions} -Extensions consist of a list of all extensions to enable. For example: +The `extensions:` subsection consists of a list of all extensions to enable. For +example: ```yaml service: @@ -513,54 +517,56 @@ service: ### Pipelines -Pipelines can be of the following types: +Pipelines are defined in the `pipelines:` subsection and can be of the following +types: -- traces: collects and processes trace data. -- metrics: collects and processes metric data. -- logs: collects and processes log data. +- `traces:` collects and processes trace data. +- `metrics:` collects and processes metric data. +- `logs:` collects and processes log data. -A pipeline consists of a set of receivers, processors and exporters. Each -receiver/processor/exporter must be defined in the configuration outside of the -service section to be included in a pipeline. +A pipeline consists of a set of receivers, processors and exporters. Before +including a receiver, processor, or exporter in a pipeline, make sure to define +its configuration in the appropriate section. -_Note:_ Each receiver/processor/exporter can be used in more than one pipeline. -For processor(s) referenced in multiple pipelines, each pipeline will get a -separate instance of that processor(s). This is in contrast to -receiver(s)/exporter(s) referenced in multiple pipelines, where only one -instance of a receiver/exporter is used for all pipelines. Also note that the -order of processors dictates the order in which data is processed. +You can use the same receiver, processor, or exporter in more than one pipeline. +When a processor is referenced in multiple pipelines, each pipeline gets a +separate instance of the processor. -The following is an example pipeline configuration: +The following is an example of pipeline configuration. Note that the order of +processors dictates the order in which data is processed: ```yaml service: pipelines: metrics: receivers: [opencensus, prometheus] + processors: [batch] exporters: [opencensus, prometheus] traces: receivers: [opencensus, jaeger] - processors: [batch] + processors: [batch, memory_limiter] exporters: [opencensus, zipkin] ``` ### Telemetry -Telemetry is where the telemetry for the collector itself can be configured. It -has two subsections: `logs` and `metrics`. +You can configure telemetry for the Collector itself in the `telemetry:` +subsection inside the `service:` section. Collector telemetry can be useful when +troubleshooting Collector issues. -The `logs` subsection allows configuration of the logs generated by the -collector. By default the collector will write its logs to stderr with a log -level of `INFO`. You can also add static key-value pairs to all logs using the -`initial_fields` section. -[View the full list of `logs` options here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L41-L97) +The `logs:` subsection lets you configure logs generated by the Collector. By +default, the Collector writes its logs to stderr with a log level of `INFO`. You +can also add static key-value pairs to all logs using the `initial_fields` +section. See the full list of `logs:` options +[here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L41-L97) -The `metrics` subsection allows configuration of the metrics generated by the -collector. By default the collector will generate basic metrics about itself and -expose them for scraping at `localhost:8888/metrics` -[View the full list of `metrics` options here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) +The `metrics:` subsection lets you configure metrics generated by the Collector. +By default, the Collector generates basic metrics about itself and expose them +for scraping at `localhost:8888/metrics`. See the full list of options for the +`metrics:` subsection +[here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) -The following is an example telemetry configuration: +The following example shows Collector telemetry configuration: ```yaml service: @@ -574,9 +580,9 @@ service: address: 0.0.0.0:8888 ``` -## Other Information +## Additional information -### Configuration Environment Variables +### Environment variables The use and expansion of environment variables is supported in the Collector configuration. For example to use the values stored on the `DB_KEY` and @@ -600,48 +606,47 @@ exporters: namespace: $$DataVisualization ``` -### Proxy Support +### Proxy support -Exporters that leverage the [`net/http`](https://pkg.go.dev/net/http) package -respect the following proxy environment variables: +Exporters that use the [`net/http`](https://pkg.go.dev/net/http) package respect +the following proxy environment variables: -- HTTP_PROXY -- HTTPS_PROXY -- NO_PROXY +- `HTTP_PROXY`: Address of the HTTP proxy +- `HTTPS_PROXY`: Address of the HTTPS proxy +- `NO_PROXY`: Addresses that must not use the proxy -If set at Collector start time, exporters, regardless of protocol, will proxy +If set at Collector start time, exporters, regardless of the protocol, proxy traffic or bypass proxy traffic as defined by these environment variables. ### Authentication -Most receivers exposing an HTTP or gRPC port are able to be protected using the -collector's authentication mechanism, and most exporters using HTTP or gRPC -clients are able to add authentication data to the outgoing requests. +Most receivers exposing an HTTP or gRPC port can be protected using the +Collector's authentication mechanism. Similarly, most exporters using HTTP or +gRPC clients can add authentication to outgoing requests. + +The authentication mechanism in the Collector uses the extensions mechanism, +allowing for custom authenticators to be plugged into Collector distributions. +Each authentication extension has two possible usages: + +- As client authenticator for exporters, adding auth data to outgoing requests +- As server authenticator for receivers, authenticating incoming connections. -The authentication mechanism in the collector uses the extensions mechanism, -allowing for custom authenticators to be plugged into collector distributions. -If you are interested in developing a custom authenticator, check out the +For a list of known authenticators, see the +[Registry](/ecosystem/registry/?s=authenticator&component=extension). If you're +interested in developing a custom authenticator, check out the ["Building a custom authenticator"](../custom-auth) document. -Each authentication extension has two possible usages: as client authenticator -for exporters, adding auth data to outgoing requests, and as server -authenticator for receivers, authenticating incoming connections. Refer to the -authentication extension for a list of its capabilities, but in general, an -authentication extension would only implement one of those traits. For a list of -known authenticators, use the -[Registry](/ecosystem/registry/?s=authenticator&component=extension) available -in this website. - -To add a server authenticator to a receiver in your collector, make sure to: - -1. add the authenticator extension and its configuration under `.extensions` -1. add a reference to the authenticator to `.services.extensions`, so that it's - loaded by the collector -1. add a reference to the authenticator under - `.receivers...auth` - -Here's an example that uses the OIDC authenticator on the receiver side, making -this suitable for a remote collector that receives data from an OpenTelemetry +To add a server authenticator to a receiver in the Collector, follow these steps +: + +1. Add the authenticator extension and its configuration under `.extensions`. +1. Add a reference to the authenticator to `.services.extensions`, so that it's + loaded by the Collector. +1. Add a reference to the authenticator under + `.receivers...auth`. + +The following example uses the OIDC authenticator on the receiver side, making +this suitable for a remote Collector that receives data from an OpenTelemetry Collector acting as agent: ```yaml @@ -676,7 +681,7 @@ service: ``` On the agent side, this is an example that makes the OTLP exporter obtain OIDC -tokens, adding them to every RPC made to a remote collector: +tokens, adding them to every RPC made to a remote Collector: ```yaml extensions: @@ -711,16 +716,15 @@ service: - otlp/auth ``` -### Setting up certificates +### Configuring certificates -For a production setup, we strongly recommend using TLS certificates, either for -secure communication or mTLS for mutual authentication. See the below steps to -generate self-signed certificates used in this example. You might want to use -your current cert provisioning procedures to procure a certificate for -production usage. +In a production environment, use TLS certificates for secure communication or +mTLS for mutual authentication. Follow these steps to generate self-signed +certificates as in this example. You might want to use your current cert +provisioning procedures to procure a certificate for production usage. -Install [cfssl](https://github.com/cloudflare/cfssl), and create the following -`csr.json` file: +1. Install [cfssl](https://github.com/cloudflare/cfssl) and create the following + `csr.json` file: ```json { @@ -737,16 +741,18 @@ Install [cfssl](https://github.com/cloudflare/cfssl), and create the following } ``` -Now, run the following commands: +2. Run the following commands: ```bash cfssl genkey -initca csr.json | cfssljson -bare ca cfssl gencert -ca ca.pem -ca-key ca-key.pem csr.json | cfssljson -bare cert ``` -This will create two certificates; first, an "OpenTelemetry Example" Certificate -Authority (CA) in `ca.pem` and the associated key in `ca-key.pem`, and second a -client certificate in `cert.pem` (signed by the OpenTelemetry Example CA) and -the associated key in `cert-key.pem`. +This creates two certificates: + +- An "OpenTelemetry Example" Certificate Authority (CA) in `ca.pem`, with the + associated key in `ca-key.pem` +- A client certificate in `cert.pem`, signed by the OpenTelemetry Example CA, + with the associated key in `cert-key.pem`. [dcc]: /docs/concepts/components/#collector From 1e790d8f7669b96e5232b794c4b3dadc59a07ae6 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 11:18:39 +0100 Subject: [PATCH 05/37] Add refs --- content/en/docs/collector/configuration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 2c00240ec008..86c0f71a0225 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -16,7 +16,7 @@ content: applicable to the OpenTelemetry Collector. - [Security guidance](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md) -## Configuration structure +## Configuration structure {#basics} The structure of any Collector configuration file consists of four classes of pipeline components that access telemetry data: @@ -359,7 +359,7 @@ exporters: # Data sources: traces otlp/jaeger: - endpoint: jaeger.example.com:4317 + endpoint: jaeger-server:4317 tls: cert_file: cert.pem key_file: cert-key.pem @@ -492,7 +492,7 @@ extensions: > For detailed extension configuration, see the > [extension README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). -## Service section +## Service section {#service} The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, @@ -716,7 +716,7 @@ service: - otlp/auth ``` -### Configuring certificates +### Configuring certificates {#setting-up-certificates} In a production environment, use TLS certificates for secure communication or mTLS for mutual authentication. Follow these steps to generate self-signed From 516247700380a315a8d768c79bd8452f332bcb6a Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 11:21:18 +0100 Subject: [PATCH 06/37] Linting issues --- content/en/docs/collector/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 86c0f71a0225..3cb91b257467 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -723,8 +723,8 @@ mTLS for mutual authentication. Follow these steps to generate self-signed certificates as in this example. You might want to use your current cert provisioning procedures to procure a certificate for production usage. -1. Install [cfssl](https://github.com/cloudflare/cfssl) and create the following - `csr.json` file: +Install [cfssl](https://github.com/cloudflare/cfssl) and create the following +`csr.json` file: ```json { @@ -741,7 +741,7 @@ provisioning procedures to procure a certificate for production usage. } ``` -2. Run the following commands: +Then run the following commands: ```bash cfssl genkey -initca csr.json | cfssljson -bare ca From fa3c1acd8f23266cc47fce7672401d62b9e473cb Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 26 Oct 2023 18:05:19 +0200 Subject: [PATCH 07/37] Redo URLs --- content/en/docs/collector/configuration.md | 51 +++++++++++----------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index a7cca798af5f..ff16a1b6d016 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -1,6 +1,7 @@ --- title: Configuration weight: 20 +description: Learn how to configure the Collector to suit your needs # prettier-ignore cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca loglevel OIDC oidc otlphttp pprof prodevent prometheusremotewrite servicegraph spanevents spanmetrics upsert zpages --- @@ -330,25 +331,25 @@ processors: ## Exporters {#exporters} -An exporter, which can be push or pull based, is how you send data to one or -more backends/destinations. Exporters may support one or more +Exporters send data to one or more backends or destinations. Exporters can be +pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -The `exporters:` section is how exporters are configured. Exporters may come -with default settings, but many require configuration to specify at least the -destination and security settings. Any configuration for an exporter must be -done in this section. Configuration parameters specified for which the exporter -provides a default configuration are overridden. +The `exporters:` section contains exporters configuration. Most exporters +require configuration to specify at least the destination, as well as +security settings, like authentication tokens or TLS certificates. Any setting +you specify overrides the default values, if present. -> Configuring an exporter does not enable it. Exporters are enabled via -> pipelines within the [service](#service) section. +> Configuring an exporter does not enable it. Exporters are enabled by +> adding them to the appropriate pipelines within the +> [service](#service) section. -One or more exporters must be configured. By default, no exporters are -configured. A basic example of exporters is provided below. Certain exporter -configurations require x.509 certificates to be created in order to be secure, -as described in [setting up certificates](#setting-up-certificates). +The Collector requires one or more exporters. The following example shows +various exporters. Notice that some exporters require x.509 certificates in +order to establish secure connections, as described in +[setting up certificates](#setting-up-certificates). -> For detailed exporter configuration, see the +> For more information on exporter configuration, see the > [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). ```yaml @@ -359,7 +360,7 @@ exporters: # Data sources: traces otlp/jaeger: - endpoint: jaeger-all-in-one:4317 + endpoint: jaeger.example.com:4317 tls: cert_file: cert.pem key_file: cert-key.pem @@ -386,24 +387,24 @@ exporters: # Data sources: traces, metrics otlphttp: - endpoint: https://example.com:4318 + endpoint: https://otlp.example.com:4318 # Data sources: metrics prometheus: - endpoint: prometheus:8889 + endpoint: localhost:8889 namespace: default # Data sources: metrics prometheusremotewrite: - endpoint: http://some.url:9411/api/prom/push - # For official Prometheus (e.g. running via Docker) - # endpoint: 'http://prometheus:9090/api/v1/write' + endpoint: http://prometheus.example.com:9411/api/prom/push + # When using the official Prometheus (running via Docker) + # endpoint: 'http://prometheus:9090/api/v1/write', add: # tls: # insecure: true # Data sources: traces zipkin: - endpoint: http://localhost:9411/api/v2/spans + endpoint: http://zipkin.example.com:9411/api/v2/spans ``` ## Connectors {#connectors} @@ -602,15 +603,15 @@ exporters: ### Proxy Support -Exporters that leverage the `net/http` package (all do today) respect the -following proxy environment variables: +Exporters that leverage the [net/http](https://pkg.go.dev/net/http) package +respect the following proxy environment variables: - HTTP_PROXY - HTTPS_PROXY - NO_PROXY -If set at Collector start time then exporters, regardless of protocol, will or -will not proxy traffic as defined by these environment variables. +If set at Collector start time, exporters, regardless of protocol, will proxy +traffic or bypass proxy traffic as defined by these environment variables. ### Authentication From ba2173350fd3d652df481c2d76262e3b9b33120b Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 26 Oct 2023 18:07:51 +0200 Subject: [PATCH 08/37] Format fixes --- content/en/docs/collector/configuration.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index ff16a1b6d016..6ab88396f297 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -335,14 +335,13 @@ Exporters send data to one or more backends or destinations. Exporters can be pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -The `exporters:` section contains exporters configuration. Most exporters -require configuration to specify at least the destination, as well as -security settings, like authentication tokens or TLS certificates. Any setting -you specify overrides the default values, if present. +The `exporters:` section contains exporters configuration. Most exporters +require configuration to specify at least the destination, as well as security +settings, like authentication tokens or TLS certificates. Any setting you +specify overrides the default values, if present. -> Configuring an exporter does not enable it. Exporters are enabled by -> adding them to the appropriate pipelines within the -> [service](#service) section. +> Configuring an exporter does not enable it. Exporters are enabled by adding +> them to the appropriate pipelines within the [service](#service) section. The Collector requires one or more exporters. The following example shows various exporters. Notice that some exporters require x.509 certificates in From 8e8673c6c5aec40b51d5062d5161138709be617a Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 26 Oct 2023 18:17:16 +0200 Subject: [PATCH 09/37] Update lint --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 6ab88396f297..e2649c8bbfbd 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -602,7 +602,7 @@ exporters: ### Proxy Support -Exporters that leverage the [net/http](https://pkg.go.dev/net/http) package +Exporters that leverage the [`net/http`](https://pkg.go.dev/net/http) package respect the following proxy environment variables: - HTTP_PROXY From 4aac692a28306e91e3e0a0470dbe947833f34926 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 11:15:37 +0100 Subject: [PATCH 10/37] Further edits --- content/en/docs/collector/configuration.md | 374 +++++++++++---------- 1 file changed, 190 insertions(+), 184 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index e2649c8bbfbd..2c00240ec008 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -8,15 +8,18 @@ cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca l -Familiarity with the following pages is assumed: +You can configure the Collector to suit your observability needs. Before you +learn how Collector configuration works, familiarize yourself with the following +content: - [Data collection concepts][dcc] in order to understand the repositories applicable to the OpenTelemetry Collector. - [Security guidance](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md) -## Basics +## Configuration structure -The Collector consists of four components that access telemetry data: +The structure of any Collector configuration file consists of four classes of +pipeline components that access telemetry data: - [Receivers](#receivers) @@ -27,15 +30,16 @@ The Collector consists of four components that access telemetry data: - [Connectors](#connectors) -These components once configured must be enabled via pipelines within the -[service](#service) section. +After each pipeline component is configured you must enable it using the +pipelines within the [service](#service) section of the configuration file. -Secondarily, there are [extensions](#extensions), which provide capabilities -that can be added to the Collector, but which do not require direct access to -telemetry data and are not part of pipelines. They are also enabled within the -[service](#service) section. +Besides pipeline components you can also configure [extensions](#extensions), +which provide capabilities that can be added to the Collector, such as +diagnostic tools. Extensions don't require direct access to telemetry data and +are enabled through the [service](#service) section. -An example configuration would look like: +The following is an example of Collector configuration with a receiver, a +processor, an exporter, and three extensions: ```yaml receivers: @@ -73,10 +77,10 @@ service: exporters: [otlp] ``` -Note that receivers, processors, exporters and/or pipelines are defined via -component identifiers in `type[/name]` format (e.g. `otlp` or `otlp/2`). -Components of a given type can be defined more than once as long as the -identifiers are unique. For example: +Note that receivers, processors, exporters and pipelines are defined through +component identifiers following the `type[/name]` format, for example `otlp` or +`otlp/2`. You can define components of a given type more than once as long as +the identifiers are unique. For example: ```yaml receivers: @@ -126,7 +130,7 @@ service: ``` The configuration can also include other files, causing the Collector to merge -the two files in a single in-memory representation of the YAML configuration: +them in a single in-memory representation of the YAML configuration: ```yaml receivers: @@ -175,25 +179,20 @@ service: ## Receivers {#receivers} -A receiver, which can be push or pull based, is how data gets into the -Collector. Receivers may support one or more -[data sources](/docs/concepts/signals/). +Receivers collect telemetry from one or more sources. They can be pull or push +based, and may support one or more [data sources](/docs/concepts/signals/). -The `receivers:` section is how receivers are configured. Many receivers come -with default settings so simply specifying the name of the receiver is enough to -configure it (for example, `zipkin:`). If configuration is required or a user -wants to change the default configuration then such configuration must be -defined in this section. Configuration parameters specified for which the -receiver provides a default configuration are overridden. +Receivers are configured in the `receivers:` section. Many receivers come with +default settings, so that specifying the name of the receiver is enough to +configure it. If you need to configure a receiver or want to change the default +configuration, you can do so in this section. Any setting you specify overrides +the default values, if present. -> Configuring a receiver does not enable it. Receivers are enabled via pipelines -> within the [service](#service) section. +> Configuring a receiver does not enable it. Receivers are enabled by adding +> them to the appropriate pipelines within the [service](#service) section. -One or more receivers must be configured. By default, no receivers are -configured. A basic example of receivers is provided below. - -> For detailed receiver configuration, see the -> [receiver README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md). +The Collector requires one or more receivers. The following example shows +various receivers in the same configuration file: ```yaml receivers: @@ -248,28 +247,31 @@ receivers: zipkin: ``` +> For detailed receiver configuration, see the +> [receiver README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md). + ## Processors {#processors} -Processors are run on data between being received and being exported. Processors -are optional though -[some are recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). +Processors take the data collected by receivers and modify or transform it +before sending it to the exporters. Data processing happens according to rules +or settings defined for each processor, which might include filtering, dropping, +renaming, or recalculating telemetry, among other operations. -The `processors:` section is how processors are configured. Processors may come -with default settings, but many require configuration. Any configuration for a -processor must be done in this section. Configuration parameters specified for -which the processor provides a default configuration are overridden. +Processors are optional, although some +[are recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). -> Configuring a processor does not enable it. Processors are enabled via -> pipelines within the [service](#service) section. +You can configure processors using the `processors:` section of the Collector +configuration file. Some processors may require security settings. Any setting +you specify overrides the default values, if present. -A basic example of the default processors is provided below. The full list of -processors can be found by combining the list found -[here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor) -and -[here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor). +> Configuring a processor does not enable it. Processors are enabled by adding +> them to the appropriate pipelines within the [service](#service) section. -> For detailed processor configuration, see the -> [processor README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md). +The following example shows several default processors in the same configuration +file. You can find the full list of processors by combining the list from +[opentelemetry-collector-contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor) +and the list from +[opentelemetry-collector](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor). ```yaml processors: @@ -329,6 +331,9 @@ processors: separator: '::' ``` +> For detailed processor configuration, see the +> [processor README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/README.md). + ## Exporters {#exporters} Exporters send data to one or more backends or destinations. Exporters can be @@ -344,12 +349,7 @@ specify overrides the default values, if present. > them to the appropriate pipelines within the [service](#service) section. The Collector requires one or more exporters. The following example shows -various exporters. Notice that some exporters require x.509 certificates in -order to establish secure connections, as described in -[setting up certificates](#setting-up-certificates). - -> For more information on exporter configuration, see the -> [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). +various exporters in the same configuration file: ```yaml exporters: @@ -406,30 +406,41 @@ exporters: endpoint: http://zipkin.example.com:9411/api/v2/spans ``` +Notice that some exporters require x.509 certificates in order to establish +secure connections, as described in +[setting up certificates](#setting-up-certificates). + +> For more information on exporter configuration, see the +> [exporter README.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/README.md). + ## Connectors {#connectors} -A connector is both an exporter and receiver. As the name suggests a Connector -connects two pipelines: It consumes data as an exporter at the end of one -pipeline and emits data as a receiver at the start of another pipeline. It may -consume and emit data of the same data type, or of different data types. A -connector may generate and emit data to summarize the consumed data, or it may -simply replicate or route data. +Connectors join two pipelines, acting as both exporter and receiver. A connector +consumes data as an exporter at the end of one pipeline and emits data as a +receiver at the beginning of another pipeline. The data consumed and emitted may +be of the same type or of different data types. You can use connectors to +summarize consumed data, replicate it, or route it. -The `connectors:` section is how connectors are configured. +You can configure one or more connectors using the `connectors:` section of the +Collector configuration file. By default, no connectors are configured. Each +type of connector is designed to work with one or more pairs of data types and +may only be used to connect pipelines accordingly. -> Configuring a connectors does not enable it. Connectors are enabled via +> Configuring a connector doesn't enable it. Connectors are enabled through > pipelines within the [service](#service) section. -One or more connectors may be configured. By default, no connectors are -configured. A basic example of connectors is provided below. - -> For detailed connector configuration, see the -> [connector README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md). +The following example shows the `count` connector and how it's configured in the +`pipelines:` section. Notice that the connector acts as an exporter for traces +and as a receiver for metrics, connecting both pipelines: ```yaml -connectors: - forward: +receivers: + foo: + +exporters: + bar: +connectors: count: spanevents: my.prod.event.count: @@ -438,48 +449,36 @@ connectors: - 'attributes["env"] == "prod"' - 'name == "prodevent"' - spanmetrics: - histogram: - explicit: - buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms] - dimensions: - - name: http.method - default: GET - - name: http.status_code - dimensions_cache_size: 1000 - aggregation_temporality: 'AGGREGATION_TEMPORALITY_CUMULATIVE' - - servicegraph: - latency_histogram_buckets: [1, 2, 3, 4, 5] - dimensions: - - dimension-1 - - dimension-2 - store: - ttl: 1s - max_items: 10 +service: + pipelines: + traces: + receivers: [foo] + exporters: [count] + metrics: + receivers: [count] + exporters: [bar] ``` +> For detailed connector configuration, see the +> [connector README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md). + ## Extensions -Extensions are available primarily for tasks that do not involve processing -telemetry data. Examples of extensions include health monitoring, service -discovery, and data forwarding. Extensions are optional. +Extensions are optional components that expand the capabilities of the Collector +to accomplish tasks not directly involved with processing telemetry data. For +example, you can add extensions for Collector health monitoring, service +discovery, or data forwarding, among others. -The `extensions:` section is how extensions are configured. Many extensions come -with default settings so simply specifying the name of the extension is enough -to configure it (for example, `health_check:`). If configuration is required or -a user wants to change the default configuration then such configuration must be -defined in this section. Configuration parameters specified for which the -extension provides a default configuration are overridden. +You can configure extensions through the `extensions:` section of the Collector +configuration file. Most extensions come with default settings, so you can +configure them just by specifying the name of the extension. Any setting you +specify overrides the default values, if present. -> Configuring an extension does not enable it. Extensions are enabled within the +> Configuring an extension doesn't enable it. Extensions are enabled within the > [service](#service) section. -By default, no extensions are configured. A basic example of extensions is -provided below. - -> For detailed extension configuration, see the -> [extension README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). +By default, no extensions are configured. The following example shows several +extensions configured in the same file: ```yaml extensions: @@ -490,21 +489,26 @@ extensions: size_mib: 512 ``` -## Service +> For detailed extension configuration, see the +> [extension README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). + +## Service section The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, exporters, and extensions sections. If a component is configured, but not -defined within the service section then it is not enabled. The service section -consists of three sub-sections: +defined within the service section, then it's not enabled. + +The service section consists of three subsections: -- extensions -- pipelines -- telemetry +- `extensions:` +- `pipelines:` +- `telemetry:` ### Extensions {#service-extensions} -Extensions consist of a list of all extensions to enable. For example: +The `extensions:` subsection consists of a list of all extensions to enable. For +example: ```yaml service: @@ -513,54 +517,56 @@ service: ### Pipelines -Pipelines can be of the following types: +Pipelines are defined in the `pipelines:` subsection and can be of the following +types: -- traces: collects and processes trace data. -- metrics: collects and processes metric data. -- logs: collects and processes log data. +- `traces:` collects and processes trace data. +- `metrics:` collects and processes metric data. +- `logs:` collects and processes log data. -A pipeline consists of a set of receivers, processors and exporters. Each -receiver/processor/exporter must be defined in the configuration outside of the -service section to be included in a pipeline. +A pipeline consists of a set of receivers, processors and exporters. Before +including a receiver, processor, or exporter in a pipeline, make sure to define +its configuration in the appropriate section. -_Note:_ Each receiver/processor/exporter can be used in more than one pipeline. -For processor(s) referenced in multiple pipelines, each pipeline will get a -separate instance of that processor(s). This is in contrast to -receiver(s)/exporter(s) referenced in multiple pipelines, where only one -instance of a receiver/exporter is used for all pipelines. Also note that the -order of processors dictates the order in which data is processed. +You can use the same receiver, processor, or exporter in more than one pipeline. +When a processor is referenced in multiple pipelines, each pipeline gets a +separate instance of the processor. -The following is an example pipeline configuration: +The following is an example of pipeline configuration. Note that the order of +processors dictates the order in which data is processed: ```yaml service: pipelines: metrics: receivers: [opencensus, prometheus] + processors: [batch] exporters: [opencensus, prometheus] traces: receivers: [opencensus, jaeger] - processors: [batch] + processors: [batch, memory_limiter] exporters: [opencensus, zipkin] ``` ### Telemetry -Telemetry is where the telemetry for the collector itself can be configured. It -has two subsections: `logs` and `metrics`. +You can configure telemetry for the Collector itself in the `telemetry:` +subsection inside the `service:` section. Collector telemetry can be useful when +troubleshooting Collector issues. -The `logs` subsection allows configuration of the logs generated by the -collector. By default the collector will write its logs to stderr with a log -level of `INFO`. You can also add static key-value pairs to all logs using the -`initial_fields` section. -[View the full list of `logs` options here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L41-L97) +The `logs:` subsection lets you configure logs generated by the Collector. By +default, the Collector writes its logs to stderr with a log level of `INFO`. You +can also add static key-value pairs to all logs using the `initial_fields` +section. See the full list of `logs:` options +[here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L41-L97) -The `metrics` subsection allows configuration of the metrics generated by the -collector. By default the collector will generate basic metrics about itself and -expose them for scraping at `localhost:8888/metrics` -[View the full list of `metrics` options here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) +The `metrics:` subsection lets you configure metrics generated by the Collector. +By default, the Collector generates basic metrics about itself and expose them +for scraping at `localhost:8888/metrics`. See the full list of options for the +`metrics:` subsection +[here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) -The following is an example telemetry configuration: +The following example shows Collector telemetry configuration: ```yaml service: @@ -574,9 +580,9 @@ service: address: 0.0.0.0:8888 ``` -## Other Information +## Additional information -### Configuration Environment Variables +### Environment variables The use and expansion of environment variables is supported in the Collector configuration. For example to use the values stored on the `DB_KEY` and @@ -600,48 +606,47 @@ exporters: namespace: $$DataVisualization ``` -### Proxy Support +### Proxy support -Exporters that leverage the [`net/http`](https://pkg.go.dev/net/http) package -respect the following proxy environment variables: +Exporters that use the [`net/http`](https://pkg.go.dev/net/http) package respect +the following proxy environment variables: -- HTTP_PROXY -- HTTPS_PROXY -- NO_PROXY +- `HTTP_PROXY`: Address of the HTTP proxy +- `HTTPS_PROXY`: Address of the HTTPS proxy +- `NO_PROXY`: Addresses that must not use the proxy -If set at Collector start time, exporters, regardless of protocol, will proxy +If set at Collector start time, exporters, regardless of the protocol, proxy traffic or bypass proxy traffic as defined by these environment variables. ### Authentication -Most receivers exposing an HTTP or gRPC port are able to be protected using the -collector's authentication mechanism, and most exporters using HTTP or gRPC -clients are able to add authentication data to the outgoing requests. +Most receivers exposing an HTTP or gRPC port can be protected using the +Collector's authentication mechanism. Similarly, most exporters using HTTP or +gRPC clients can add authentication to outgoing requests. + +The authentication mechanism in the Collector uses the extensions mechanism, +allowing for custom authenticators to be plugged into Collector distributions. +Each authentication extension has two possible usages: + +- As client authenticator for exporters, adding auth data to outgoing requests +- As server authenticator for receivers, authenticating incoming connections. -The authentication mechanism in the collector uses the extensions mechanism, -allowing for custom authenticators to be plugged into collector distributions. -If you are interested in developing a custom authenticator, check out the +For a list of known authenticators, see the +[Registry](/ecosystem/registry/?s=authenticator&component=extension). If you're +interested in developing a custom authenticator, check out the ["Building a custom authenticator"](../custom-auth) document. -Each authentication extension has two possible usages: as client authenticator -for exporters, adding auth data to outgoing requests, and as server -authenticator for receivers, authenticating incoming connections. Refer to the -authentication extension for a list of its capabilities, but in general, an -authentication extension would only implement one of those traits. For a list of -known authenticators, use the -[Registry](/ecosystem/registry/?s=authenticator&component=extension) available -in this website. - -To add a server authenticator to a receiver in your collector, make sure to: - -1. add the authenticator extension and its configuration under `.extensions` -1. add a reference to the authenticator to `.services.extensions`, so that it's - loaded by the collector -1. add a reference to the authenticator under - `.receivers...auth` - -Here's an example that uses the OIDC authenticator on the receiver side, making -this suitable for a remote collector that receives data from an OpenTelemetry +To add a server authenticator to a receiver in the Collector, follow these steps +: + +1. Add the authenticator extension and its configuration under `.extensions`. +1. Add a reference to the authenticator to `.services.extensions`, so that it's + loaded by the Collector. +1. Add a reference to the authenticator under + `.receivers...auth`. + +The following example uses the OIDC authenticator on the receiver side, making +this suitable for a remote Collector that receives data from an OpenTelemetry Collector acting as agent: ```yaml @@ -676,7 +681,7 @@ service: ``` On the agent side, this is an example that makes the OTLP exporter obtain OIDC -tokens, adding them to every RPC made to a remote collector: +tokens, adding them to every RPC made to a remote Collector: ```yaml extensions: @@ -711,16 +716,15 @@ service: - otlp/auth ``` -### Setting up certificates +### Configuring certificates -For a production setup, we strongly recommend using TLS certificates, either for -secure communication or mTLS for mutual authentication. See the below steps to -generate self-signed certificates used in this example. You might want to use -your current cert provisioning procedures to procure a certificate for -production usage. +In a production environment, use TLS certificates for secure communication or +mTLS for mutual authentication. Follow these steps to generate self-signed +certificates as in this example. You might want to use your current cert +provisioning procedures to procure a certificate for production usage. -Install [cfssl](https://github.com/cloudflare/cfssl), and create the following -`csr.json` file: +1. Install [cfssl](https://github.com/cloudflare/cfssl) and create the following + `csr.json` file: ```json { @@ -737,16 +741,18 @@ Install [cfssl](https://github.com/cloudflare/cfssl), and create the following } ``` -Now, run the following commands: +2. Run the following commands: ```bash cfssl genkey -initca csr.json | cfssljson -bare ca cfssl gencert -ca ca.pem -ca-key ca-key.pem csr.json | cfssljson -bare cert ``` -This will create two certificates; first, an "OpenTelemetry Example" Certificate -Authority (CA) in `ca.pem` and the associated key in `ca-key.pem`, and second a -client certificate in `cert.pem` (signed by the OpenTelemetry Example CA) and -the associated key in `cert-key.pem`. +This creates two certificates: + +- An "OpenTelemetry Example" Certificate Authority (CA) in `ca.pem`, with the + associated key in `ca-key.pem` +- A client certificate in `cert.pem`, signed by the OpenTelemetry Example CA, + with the associated key in `cert-key.pem`. [dcc]: /docs/concepts/components/#collector From 66a3fe89880cfb0960ff15d1bc6632c4693f5721 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 11:18:39 +0100 Subject: [PATCH 11/37] Add refs --- content/en/docs/collector/configuration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 2c00240ec008..86c0f71a0225 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -16,7 +16,7 @@ content: applicable to the OpenTelemetry Collector. - [Security guidance](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md) -## Configuration structure +## Configuration structure {#basics} The structure of any Collector configuration file consists of four classes of pipeline components that access telemetry data: @@ -359,7 +359,7 @@ exporters: # Data sources: traces otlp/jaeger: - endpoint: jaeger.example.com:4317 + endpoint: jaeger-server:4317 tls: cert_file: cert.pem key_file: cert-key.pem @@ -492,7 +492,7 @@ extensions: > For detailed extension configuration, see the > [extension README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/extension/README.md). -## Service section +## Service section {#service} The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, @@ -716,7 +716,7 @@ service: - otlp/auth ``` -### Configuring certificates +### Configuring certificates {#setting-up-certificates} In a production environment, use TLS certificates for secure communication or mTLS for mutual authentication. Follow these steps to generate self-signed From fd81f8ba69a7955f1ee9a369c488d6e619cdea09 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 11:21:18 +0100 Subject: [PATCH 12/37] Linting issues --- content/en/docs/collector/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 86c0f71a0225..3cb91b257467 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -723,8 +723,8 @@ mTLS for mutual authentication. Follow these steps to generate self-signed certificates as in this example. You might want to use your current cert provisioning procedures to procure a certificate for production usage. -1. Install [cfssl](https://github.com/cloudflare/cfssl) and create the following - `csr.json` file: +Install [cfssl](https://github.com/cloudflare/cfssl) and create the following +`csr.json` file: ```json { @@ -741,7 +741,7 @@ provisioning procedures to procure a certificate for production usage. } ``` -2. Run the following commands: +Then run the following commands: ```bash cfssl genkey -initca csr.json | cfssljson -bare ca From 21f7883c4961e3325aecedca0af3c9f6be9d2d64 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 16:00:18 +0100 Subject: [PATCH 13/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 3cb91b257467..49ad2483d2a9 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -340,7 +340,7 @@ Exporters send data to one or more backends or destinations. Exporters can be pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -The `exporters:` section contains exporters configuration. Most exporters +The `exporters` section contains exporters configuration. Most exporters require configuration to specify at least the destination, as well as security settings, like authentication tokens or TLS certificates. Any setting you specify overrides the default values, if present. From e986fbc29fc30d0982b0ac4ef6abca263efb96d9 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 16:00:28 +0100 Subject: [PATCH 14/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 49ad2483d2a9..90954e6bbf93 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -260,7 +260,7 @@ renaming, or recalculating telemetry, among other operations. Processors are optional, although some [are recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). -You can configure processors using the `processors:` section of the Collector +You can configure processors using the `processors` section of the Collector configuration file. Some processors may require security settings. Any setting you specify overrides the default values, if present. From 23616d89c9cd7edd5a413c0fe2f1e62d938071b5 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 16:00:35 +0100 Subject: [PATCH 15/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 90954e6bbf93..befc9c745ceb 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -182,7 +182,7 @@ service: Receivers collect telemetry from one or more sources. They can be pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -Receivers are configured in the `receivers:` section. Many receivers come with +Receivers are configured in the `receivers` section. Many receivers come with default settings, so that specifying the name of the receiver is enough to configure it. If you need to configure a receiver or want to change the default configuration, you can do so in this section. Any setting you specify overrides From f830907de6929a84fc43faf72334abb111ad9797 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 16:00:42 +0100 Subject: [PATCH 16/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index befc9c745ceb..0964a354d0ba 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -636,8 +636,7 @@ For a list of known authenticators, see the interested in developing a custom authenticator, check out the ["Building a custom authenticator"](../custom-auth) document. -To add a server authenticator to a receiver in the Collector, follow these steps -: +To add a server authenticator to a receiver in the Collector, follow these steps: 1. Add the authenticator extension and its configuration under `.extensions`. 1. Add a reference to the authenticator to `.services.extensions`, so that it's From 619dcaaff1bb7a1140e3cf09282a408861e9591f Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 16:01:28 +0100 Subject: [PATCH 17/37] Update configuration.md --- content/en/docs/collector/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 0964a354d0ba..d08e6c49456f 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -501,9 +501,9 @@ defined within the service section, then it's not enabled. The service section consists of three subsections: -- `extensions:` -- `pipelines:` -- `telemetry:` +- Extensions +- Pipelines +- Telemetry ### Extensions {#service-extensions} From 597cd8f1b87b5c670baf73e316b6edc87f98b67d Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Tue, 31 Oct 2023 16:15:59 +0100 Subject: [PATCH 18/37] Fix format --- content/en/docs/collector/configuration.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index d08e6c49456f..36f1090fe01d 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -340,10 +340,10 @@ Exporters send data to one or more backends or destinations. Exporters can be pull or push based, and may support one or more [data sources](/docs/concepts/signals/). -The `exporters` section contains exporters configuration. Most exporters -require configuration to specify at least the destination, as well as security -settings, like authentication tokens or TLS certificates. Any setting you -specify overrides the default values, if present. +The `exporters` section contains exporters configuration. Most exporters require +configuration to specify at least the destination, as well as security settings, +like authentication tokens or TLS certificates. Any setting you specify +overrides the default values, if present. > Configuring an exporter does not enable it. Exporters are enabled by adding > them to the appropriate pipelines within the [service](#service) section. @@ -636,7 +636,8 @@ For a list of known authenticators, see the interested in developing a custom authenticator, check out the ["Building a custom authenticator"](../custom-auth) document. -To add a server authenticator to a receiver in the Collector, follow these steps: +To add a server authenticator to a receiver in the Collector, follow these +steps: 1. Add the authenticator extension and its configuration under `.extensions`. 1. Add a reference to the authenticator to `.services.extensions`, so that it's From e579d0b256604bdc7bfa0cc482ac083fb0f23395 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:21:55 +0100 Subject: [PATCH 19/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 36f1090fe01d..3a08df2b6582 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -421,7 +421,7 @@ receiver at the beginning of another pipeline. The data consumed and emitted may be of the same type or of different data types. You can use connectors to summarize consumed data, replicate it, or route it. -You can configure one or more connectors using the `connectors:` section of the +You can configure one or more connectors using the `connectors` section of the Collector configuration file. By default, no connectors are configured. Each type of connector is designed to work with one or more pairs of data types and may only be used to connect pipelines accordingly. From ccc08a12490bdda7a3d8411cd50c2ec5f038a00c Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:22:06 +0100 Subject: [PATCH 20/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 3a08df2b6582..6901970fafb6 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -430,7 +430,7 @@ may only be used to connect pipelines accordingly. > pipelines within the [service](#service) section. The following example shows the `count` connector and how it's configured in the -`pipelines:` section. Notice that the connector acts as an exporter for traces +`pipelines` section. Notice that the connector acts as an exporter for traces and as a receiver for metrics, connecting both pipelines: ```yaml From c8de7f78083071d5c15dd3b359ee7ec0143e93fc Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:22:15 +0100 Subject: [PATCH 21/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 6901970fafb6..0fd05b23f866 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -469,7 +469,7 @@ to accomplish tasks not directly involved with processing telemetry data. For example, you can add extensions for Collector health monitoring, service discovery, or data forwarding, among others. -You can configure extensions through the `extensions:` section of the Collector +You can configure extensions through the `extensions` section of the Collector configuration file. Most extensions come with default settings, so you can configure them just by specifying the name of the extension. Any setting you specify overrides the default values, if present. From 7a623e68f7cece0ce713b6c94c8c8be368cf1529 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:22:24 +0100 Subject: [PATCH 22/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 0fd05b23f866..fc06b0986597 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -520,9 +520,9 @@ service: Pipelines are defined in the `pipelines:` subsection and can be of the following types: -- `traces:` collects and processes trace data. -- `metrics:` collects and processes metric data. -- `logs:` collects and processes log data. +- `traces` collects and processes trace data. +- `metrics` collects and processes metric data. +- `logs` collects and processes log data. A pipeline consists of a set of receivers, processors and exporters. Before including a receiver, processor, or exporter in a pipeline, make sure to define From ffe0f72514de6a79a0d5667a4c31482ba50fa0bd Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:22:36 +0100 Subject: [PATCH 23/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index fc06b0986597..60b2263a95ce 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -554,10 +554,10 @@ You can configure telemetry for the Collector itself in the `telemetry:` subsection inside the `service:` section. Collector telemetry can be useful when troubleshooting Collector issues. -The `logs:` subsection lets you configure logs generated by the Collector. By +The `logs` subsection lets you configure logs generated by the Collector. By default, the Collector writes its logs to stderr with a log level of `INFO`. You can also add static key-value pairs to all logs using the `initial_fields` -section. See the full list of `logs:` options +section. See the full list of `logs` options [here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L41-L97) The `metrics:` subsection lets you configure metrics generated by the Collector. From 7bd1989c5d13f3af1df01ccda48e8eb9ebdf6a33 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:22:46 +0100 Subject: [PATCH 24/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 60b2263a95ce..497e0a68fd03 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -517,7 +517,7 @@ service: ### Pipelines -Pipelines are defined in the `pipelines:` subsection and can be of the following +Pipelines are defined in the `pipelines` subsection and can be of the following types: - `traces` collects and processes trace data. From 0c3c657b147fc550a332f7cd52dbacf5ce60645c Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:22:54 +0100 Subject: [PATCH 25/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 497e0a68fd03..c3e2cbec5ebc 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -560,7 +560,7 @@ can also add static key-value pairs to all logs using the `initial_fields` section. See the full list of `logs` options [here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L41-L97) -The `metrics:` subsection lets you configure metrics generated by the Collector. +The `metrics` subsection lets you configure metrics generated by the Collector. By default, the Collector generates basic metrics about itself and expose them for scraping at `localhost:8888/metrics`. See the full list of options for the `metrics:` subsection From 04228846bf74825a9c49fb4e0fc880318714c24d Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:23:06 +0100 Subject: [PATCH 26/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index c3e2cbec5ebc..441d76cab9e4 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -550,8 +550,8 @@ service: ### Telemetry -You can configure telemetry for the Collector itself in the `telemetry:` -subsection inside the `service:` section. Collector telemetry can be useful when +You can configure telemetry for the Collector itself in the `telemetry` +subsection inside the `service` section. Collector telemetry can be useful when troubleshooting Collector issues. The `logs` subsection lets you configure logs generated by the Collector. By From 50d4020bd7df5aa706a4419ff0e8dd97718aad52 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:23:56 +0100 Subject: [PATCH 27/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 441d76cab9e4..1f1fffa5a3ee 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -563,7 +563,7 @@ section. See the full list of `logs` options The `metrics` subsection lets you configure metrics generated by the Collector. By default, the Collector generates basic metrics about itself and expose them for scraping at `localhost:8888/metrics`. See the full list of options for the -`metrics:` subsection +`metrics` subsection [here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) The following example shows Collector telemetry configuration: From c08e79c85f19b7d7a6c4d728b3d6c30afe105feb Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:24:10 +0100 Subject: [PATCH 28/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 1f1fffa5a3ee..40ede46df1ad 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -723,7 +723,7 @@ mTLS for mutual authentication. Follow these steps to generate self-signed certificates as in this example. You might want to use your current cert provisioning procedures to procure a certificate for production usage. -Install [cfssl](https://github.com/cloudflare/cfssl) and create the following +Install [`cfssl`](https://github.com/cloudflare/cfssl) and create the following `csr.json` file: ```json From c63b67dae7d426df5c712b1b1c0fb331e7380a6a Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 12:30:40 +0100 Subject: [PATCH 29/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 40ede46df1ad..57da01bce6c4 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -562,7 +562,7 @@ section. See the full list of `logs` options The `metrics` subsection lets you configure metrics generated by the Collector. By default, the Collector generates basic metrics about itself and expose them -for scraping at `localhost:8888/metrics`. See the full list of options for the +for scraping at . See the full list of options for the `metrics` subsection [here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) From b9406d0979a3aafa41e6dc17dcc8a9428e880b69 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Mon, 6 Nov 2023 14:16:02 +0100 Subject: [PATCH 30/37] Formatting --- content/en/docs/collector/configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 57da01bce6c4..3202818e0712 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -562,8 +562,8 @@ section. See the full list of `logs` options The `metrics` subsection lets you configure metrics generated by the Collector. By default, the Collector generates basic metrics about itself and expose them -for scraping at . See the full list of options for the -`metrics` subsection +for scraping at . See the full list of options +for the `metrics` subsection [here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) The following example shows Collector telemetry configuration: From ee43d571f323efcf75e799e5f49934a115186c46 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 8 Nov 2023 08:40:20 +0100 Subject: [PATCH 31/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 3202818e0712..3cfc81720999 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -633,8 +633,8 @@ Each authentication extension has two possible usages: For a list of known authenticators, see the [Registry](/ecosystem/registry/?s=authenticator&component=extension). If you're -interested in developing a custom authenticator, check out the -["Building a custom authenticator"](../custom-auth) document. +interested in developing a custom authenticator, see +[Building a custom authenticator](../custom-auth). To add a server authenticator to a receiver in the Collector, follow these steps: From 8c569df9289f730a7b8317da53e8280f9a2e794d Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 8 Nov 2023 08:40:27 +0100 Subject: [PATCH 32/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 3cfc81720999..5a719155e899 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -507,7 +507,7 @@ The service section consists of three subsections: ### Extensions {#service-extensions} -The `extensions:` subsection consists of a list of all extensions to enable. For +The `extensions` subsection consists of a list of all extensions to enable. For example: ```yaml From fc4bc91b33778518c73bce26fc507bf9f2164507 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 8 Nov 2023 08:40:33 +0100 Subject: [PATCH 33/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 5a719155e899..963ed82d43fb 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -520,9 +520,9 @@ service: Pipelines are defined in the `pipelines` subsection and can be of the following types: -- `traces` collects and processes trace data. -- `metrics` collects and processes metric data. -- `logs` collects and processes log data. +- `traces` collect and processes trace data. +- `metrics` collect and processes metric data. +- `logs` collect and processes log data. A pipeline consists of a set of receivers, processors and exporters. Before including a receiver, processor, or exporter in a pipeline, make sure to define From acef6563b2fa9d37caa05c915da01563b3d2bd09 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 8 Nov 2023 08:40:46 +0100 Subject: [PATCH 34/37] Update content/en/docs/collector/configuration.md Co-authored-by: Patrice Chalin --- content/en/docs/collector/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 963ed82d43fb..d7cf305ecc81 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -562,9 +562,9 @@ section. See the full list of `logs` options The `metrics` subsection lets you configure metrics generated by the Collector. By default, the Collector generates basic metrics about itself and expose them -for scraping at . See the full list of options -for the `metrics` subsection -[here.](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) +for scraping at . For the full list of options +for the `metrics` subsection, see the +[`ServiceTelemetryMetrics struct`](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) comments. The following example shows Collector telemetry configuration: From 9c85b18ad874c6034ec4a22a4795fc7bf4bd28fc Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 8 Nov 2023 14:49:34 +0100 Subject: [PATCH 35/37] Additional fixes --- content/en/docs/collector/configuration.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index d7cf305ecc81..592640e1732d 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -3,7 +3,7 @@ title: Configuration weight: 20 description: Learn how to configure the Collector to suit your needs # prettier-ignore -cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca loglevel OIDC oidc otlphttp pprof prodevent prometheusremotewrite servicegraph spanevents spanmetrics upsert zpages +cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca loglevel OIDC oidc otlphttp pprof prodevent prometheusremotewrite servicegraph spanevents spanmetrics upsert zpages struct --- @@ -563,8 +563,9 @@ section. See the full list of `logs` options The `metrics` subsection lets you configure metrics generated by the Collector. By default, the Collector generates basic metrics about itself and expose them for scraping at . For the full list of options -for the `metrics` subsection, see the -[`ServiceTelemetryMetrics struct`](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) comments. +for the `metrics` subsection, see the +[`ServiceTelemetryMetrics struct`](https://github.com/open-telemetry/opentelemetry-collector/blob/7666eb04c30e5cfd750db9969fe507562598f0ae/config/service.go#L99-L111) +comments. The following example shows Collector telemetry configuration: @@ -633,7 +634,7 @@ Each authentication extension has two possible usages: For a list of known authenticators, see the [Registry](/ecosystem/registry/?s=authenticator&component=extension). If you're -interested in developing a custom authenticator, see +interested in developing a custom authenticator, see [Building a custom authenticator](../custom-auth). To add a server authenticator to a receiver in the Collector, follow these From 935db7161e83fb87f58700e28fee04faa23d0d2c Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 8 Nov 2023 14:53:43 +0100 Subject: [PATCH 36/37] Fixes --- content/en/docs/collector/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index 592640e1732d..cd3b837514bd 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -3,7 +3,7 @@ title: Configuration weight: 20 description: Learn how to configure the Collector to suit your needs # prettier-ignore -cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca loglevel OIDC oidc otlphttp pprof prodevent prometheusremotewrite servicegraph spanevents spanmetrics upsert zpages struct +cSpell:ignore: cfssl cfssljson fluentforward gencert genkey hostmetrics initca loglevel OIDC oidc otlphttp pprof prodevent prometheusremotewrite servicegraph spanevents spanmetrics struct upsert zpages --- From fa5c7d67711c59cd20b02a96f65a0cdcc9030bbc Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Wed, 22 Nov 2023 09:49:58 +0100 Subject: [PATCH 37/37] Edits --- content/en/docs/collector/configuration.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/configuration.md b/content/en/docs/collector/configuration.md index cd3b837514bd..509f00ae92a3 100644 --- a/content/en/docs/collector/configuration.md +++ b/content/en/docs/collector/configuration.md @@ -255,14 +255,16 @@ receivers: Processors take the data collected by receivers and modify or transform it before sending it to the exporters. Data processing happens according to rules or settings defined for each processor, which might include filtering, dropping, -renaming, or recalculating telemetry, among other operations. +renaming, or recalculating telemetry, among other operations. The order of the +processors in a pipeline determines the order of the processing operations that +the Collector applies to the signal. Processors are optional, although some [are recommended](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor#recommended-processors). You can configure processors using the `processors` section of the Collector -configuration file. Some processors may require security settings. Any setting -you specify overrides the default values, if present. +configuration file. Any setting you specify overrides the default values, if +present. > Configuring a processor does not enable it. Processors are enabled by adding > them to the appropriate pipelines within the [service](#service) section.