Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove suggestion to process internal telemetry through collector #5749

Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 20 additions & 66 deletions content/en/docs/collector/internal-telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,26 @@ You can configure how internal metrics are generated and exposed by the
Collector. By default, the Collector generates basic metrics about itself and
exposes them using the OpenTelemetry Go
[Prometheus exporter](https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus)
for scraping at `http://127.0.0.1:8888/metrics`. You can expose the endpoint to
one specific or all network interfaces when needed. For containerized
environments, you might want to expose this port on a public interface.
for scraping at `http://127.0.0.1:8888/metrics`.

The Collector can push its internal metrics to an OTLP backend via the following
configuration:

```yaml
service:
telemetry:
metrics:
readers:
- periodic:
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
```

Alternatively, you can expose the Prometheus endpoint to one specific or all
network interfaces when needed. For containerized environments, you might want
to expose this port on a public interface.

Set the Prometheus config under `service::telemetry::metrics`:

Expand Down Expand Up @@ -151,69 +168,6 @@ Note that the `tracer_provider` section there corresponds to `traces` here.
[kitchen-sink-config]:
https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml

### Self-monitoring

The Collector can be configured to push its own telemetry to an
[OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver)
and send the data through configured pipelines. In the following example, the
Collector is configured to push metrics, traces, and logs every 10s using OTLP
gRPC to `localhost:14317`:

```yaml
receivers:
otlp/internal:
protocols:
grpc:
endpoint: localhost:14317
exporters:
debug:
service:
pipelines:
metrics:
receivers: [otlp/internal]
exporters: [debug]
traces:
receivers: [otlp/internal]
exporters: [debug]
telemetry:
metrics:
readers:
- periodic:
interval: 10000
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
traces:
processors:
- batch:
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
logs:
processors:
- batch:
exporter:
otlp:
protocol: grpc/protobuf
endpoint: http://localhost:14317
```

{{% alert title="Caution" color="warning" %}}

When self-monitoring, the Collector collects its own telemetry and sends it to
the desired backend for analysis. This can be a risky practice. If the Collector
is underperforming, its self-monitoring capability could be impacted. As a
result, the self-monitored telemetry might not reach the backend in time for
critical analysis.

Moreover, sending internal telemetry through the Collector's own pipelines can
create a continuous loop of spans, metric points, or logs, putting undue strain
on the Collector's performance. This setup should not be used in production.

{{% /alert %}}

## Types of internal telemetry

The OpenTelemetry Collector aims to be a model of observable service by clearly
Expand Down
Loading