From 5584ba70138116dd0f7ad03517da94225920f680 Mon Sep 17 00:00:00 2001 From: Dinesh Gurumurthy Date: Tue, 5 Nov 2024 05:08:19 -0500 Subject: [PATCH] Add Gateway deployment arch (#5452) Co-authored-by: Severin Neumann --- .../{gateway.md => gateway/index.md} | 35 ++++++++++++++++++- .../deployment/gateway/otel-gateway-arch.svg | 10 ++++++ 2 files changed, 44 insertions(+), 1 deletion(-) rename content/en/docs/collector/deployment/{gateway.md => gateway/index.md} (78%) create mode 100644 content/en/docs/collector/deployment/gateway/otel-gateway-arch.svg diff --git a/content/en/docs/collector/deployment/gateway.md b/content/en/docs/collector/deployment/gateway/index.md similarity index 78% rename from content/en/docs/collector/deployment/gateway.md rename to content/en/docs/collector/deployment/gateway/index.md index 8ef00132d819..6b3dae479b81 100644 --- a/content/en/docs/collector/deployment/gateway.md +++ b/content/en/docs/collector/deployment/gateway/index.md @@ -4,7 +4,8 @@ description: Why and how to send signals to a single OTLP end-point and from there to backends weight: 3 -cSpell:ignore: hostnames loadbalancer loadbalancing +# prettier-ignore +cSpell:ignore: filelogreceiver hostmetricsreceiver hostnames loadbalancer loadbalancing resourcedetectionprocessor --- The gateway collector deployment pattern consists of applications (or other @@ -198,6 +199,38 @@ The load-balancing exporter emits metrics including that you can use for health and performance monitoring of the OTLP endpoint collector. +## Combined deployment of Collectors as agents and gateways + +Often a deployment of multiple OpenTelemetry collectors involves running both +Collector as gateways and as [agents](/docs/collector/deployment/agent/). + +The following diagram shows an architecture for such a combined deployment: + +- We use the Collectors running in the agent deployment pattern (running on each + host, similar to Kubernetes daemonsets) to collect telemetry from services + running on the host and host telemetry, such as host metrics and scrap logs. +- We use Collectors running in the gateway deployment pattern to process data, + such as filtering, sampling, and exporting to backends etc. + +![gateway](otel-gateway-arch.svg) + +This combined deployment pattern is necessary, when you use components in your +Collector that either need to be unique per host or that consume information +that is only available on the same host as the application is running: + +- Receivers like the + [`hostmetricsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) + or + [`filelogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) + need to be unique per host instance. Running multiple instances of these + receivers will result in duplicated data. + +- Processors like the + [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) + are used to add information about the host, the collector and the application + are running on. Running them within a Collector on a remote machine will + result in incorrect data. + ## Tradeoffs Pros: diff --git a/content/en/docs/collector/deployment/gateway/otel-gateway-arch.svg b/content/en/docs/collector/deployment/gateway/otel-gateway-arch.svg new file mode 100644 index 000000000000..f2062bb879e6 --- /dev/null +++ b/content/en/docs/collector/deployment/gateway/otel-gateway-arch.svg @@ -0,0 +1,10 @@ + + + + + + + + LoadBalancerOTLPExporterOTLPReceiverhostmetricsreceiverhostmetricsreceiverOTLPExporterOTel Collector DaemonOTel Collector DaemonGateway Node in Cluster BNode in Cluster BBackend AexporterOTLPReceiverOTel Collector Gateway instance 1OTLPReceiverBackend Bexporter Backend AexporterBackend Bexporter Backend ABackend BOTel Collector Gateway instance 2Service BService Aresourcedetectionprocessor \ No newline at end of file