From c1ec31f78147deaebe6ed8bf509675e9409a02e7 Mon Sep 17 00:00:00 2001 From: Alex Boten <223565+codeboten@users.noreply.github.com> Date: Mon, 9 Dec 2024 11:29:25 -0800 Subject: [PATCH 1/4] remove suggestion to process internal telemetry through collector The following PR removes the suggestion in the documentation to send the Collector's internal telemetry through the Collector's pipelines. As the warning suggests, doing this is risky and we likely don't want to recommend users do this. Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com> --- .../en/docs/collector/internal-telemetry.md | 79 ++++--------------- 1 file changed, 16 insertions(+), 63 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index f125988bdbe8..473de3f2ea06 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -43,6 +43,22 @@ service: port: 8888 ``` +Alternatively, the Collector can push its internal metrics via +to an OTLP backend via the following configuration: + +```yaml + +service: + telemetry: + metrics: + readers: + - periodic: + exporter: + otlp: + protocol: grpc/protobuf + endpoint: http://localhost:14317 +``` + You can adjust the verbosity of the Collector metrics output by setting the `level` field to one of the following values: @@ -151,69 +167,6 @@ Note that the `tracer_provider` section there corresponds to `traces` here. [kitchen-sink-config]: https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml -### Self-monitoring - -The Collector can be configured to push its own telemetry to an -[OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver) -and send the data through configured pipelines. In the following example, the -Collector is configured to push metrics, traces, and logs every 10s using OTLP -gRPC to `localhost:14317`: - -```yaml -receivers: - otlp/internal: - protocols: - grpc: - endpoint: localhost:14317 -exporters: - debug: -service: - pipelines: - metrics: - receivers: [otlp/internal] - exporters: [debug] - traces: - receivers: [otlp/internal] - exporters: [debug] - telemetry: - metrics: - readers: - - periodic: - interval: 10000 - exporter: - otlp: - protocol: grpc/protobuf - endpoint: http://localhost:14317 - traces: - processors: - - batch: - exporter: - otlp: - protocol: grpc/protobuf - endpoint: http://localhost:14317 - logs: - processors: - - batch: - exporter: - otlp: - protocol: grpc/protobuf - endpoint: http://localhost:14317 -``` - -{{% alert title="Caution" color="warning" %}} - -When self-monitoring, the Collector collects its own telemetry and sends it to -the desired backend for analysis. This can be a risky practice. If the Collector -is underperforming, its self-monitoring capability could be impacted. As a -result, the self-monitored telemetry might not reach the backend in time for -critical analysis. - -Moreover, sending internal telemetry through the Collector's own pipelines can -create a continuous loop of spans, metric points, or logs, putting undue strain -on the Collector's performance. This setup should not be used in production. - -{{% /alert %}} - ## Types of internal telemetry The OpenTelemetry Collector aims to be a model of observable service by clearly From 18a8afc5ca92370f92ac93f2ea30f7c679c34481 Mon Sep 17 00:00:00 2001 From: Alex Boten <223565+codeboten@users.noreply.github.com> Date: Mon, 9 Dec 2024 11:36:15 -0800 Subject: [PATCH 2/4] fix lint Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com> --- content/en/docs/collector/internal-telemetry.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 473de3f2ea06..41a994e0c2c8 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -43,11 +43,10 @@ service: port: 8888 ``` -Alternatively, the Collector can push its internal metrics via -to an OTLP backend via the following configuration: +Alternatively, the Collector can push its internal metrics via to an OTLP +backend via the following configuration: ```yaml - service: telemetry: metrics: From bc48f45e731f40418a1581402b5f4adb97164b10 Mon Sep 17 00:00:00 2001 From: Alex Boten <223565+codeboten@users.noreply.github.com> Date: Tue, 10 Dec 2024 11:10:26 -0800 Subject: [PATCH 3/4] change the order to prioritize otlp over prometheus Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com> --- .../en/docs/collector/internal-telemetry.md | 31 ++++++++++--------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 41a994e0c2c8..5f6b6962c9d9 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -25,39 +25,42 @@ You can configure how internal metrics are generated and exposed by the Collector. By default, the Collector generates basic metrics about itself and exposes them using the OpenTelemetry Go [Prometheus exporter](https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus) -for scraping at `http://127.0.0.1:8888/metrics`. You can expose the endpoint to -one specific or all network interfaces when needed. For containerized -environments, you might want to expose this port on a public interface. +for scraping at `http://127.0.0.1:8888/metrics`. -Set the Prometheus config under `service::telemetry::metrics`: +The Collector can push its internal metrics to an OTLP +backend via the following configuration: ```yaml service: telemetry: metrics: readers: - - pull: + - periodic: exporter: - prometheus: - host: '0.0.0.0' - port: 8888 + otlp: + protocol: grpc/protobuf + endpoint: http://localhost:14317 ``` -Alternatively, the Collector can push its internal metrics via to an OTLP -backend via the following configuration: +Alternatively, you can expose the Prometheus endpoint to +one specific or all network interfaces when needed. For containerized +environments, you might want to expose this port on a public interface. + +Set the Prometheus config under `service::telemetry::metrics`: ```yaml service: telemetry: metrics: readers: - - periodic: + - pull: exporter: - otlp: - protocol: grpc/protobuf - endpoint: http://localhost:14317 + prometheus: + host: '0.0.0.0' + port: 8888 ``` + You can adjust the verbosity of the Collector metrics output by setting the `level` field to one of the following values: From 14fe781a40f3e90a7f026920cb0a64cc4360119a Mon Sep 17 00:00:00 2001 From: Alex Boten <223565+codeboten@users.noreply.github.com> Date: Tue, 10 Dec 2024 11:13:37 -0800 Subject: [PATCH 4/4] fix linter Signed-off-by: Alex Boten <223565+codeboten@users.noreply.github.com> --- content/en/docs/collector/internal-telemetry.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 5f6b6962c9d9..3ddbc5adf28e 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -27,8 +27,8 @@ exposes them using the OpenTelemetry Go [Prometheus exporter](https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus) for scraping at `http://127.0.0.1:8888/metrics`. -The Collector can push its internal metrics to an OTLP -backend via the following configuration: +The Collector can push its internal metrics to an OTLP backend via the following +configuration: ```yaml service: @@ -42,9 +42,9 @@ service: endpoint: http://localhost:14317 ``` -Alternatively, you can expose the Prometheus endpoint to -one specific or all network interfaces when needed. For containerized -environments, you might want to expose this port on a public interface. +Alternatively, you can expose the Prometheus endpoint to one specific or all +network interfaces when needed. For containerized environments, you might want +to expose this port on a public interface. Set the Prometheus config under `service::telemetry::metrics`: @@ -60,7 +60,6 @@ service: port: 8888 ``` - You can adjust the verbosity of the Collector metrics output by setting the `level` field to one of the following values: