diff --git a/content/en/blog/2024/otel-collector-survey/areas-of-improvement.png b/content/en/blog/2024/otel-collector-survey/areas-of-improvement.png new file mode 100644 index 000000000000..80741de2dae9 Binary files /dev/null and b/content/en/blog/2024/otel-collector-survey/areas-of-improvement.png differ diff --git a/content/en/blog/2024/otel-collector-survey/deployment-environment.png b/content/en/blog/2024/otel-collector-survey/deployment-environment.png new file mode 100644 index 000000000000..81f9834e3689 Binary files /dev/null and b/content/en/blog/2024/otel-collector-survey/deployment-environment.png differ diff --git a/content/en/blog/2024/otel-collector-survey/deployment-scale.png b/content/en/blog/2024/otel-collector-survey/deployment-scale.png new file mode 100644 index 000000000000..31447ed77c58 Binary files /dev/null and b/content/en/blog/2024/otel-collector-survey/deployment-scale.png differ diff --git a/content/en/blog/2024/otel-collector-survey/index.md b/content/en/blog/2024/otel-collector-survey/index.md new file mode 100644 index 000000000000..77184375f8ec --- /dev/null +++ b/content/en/blog/2024/otel-collector-survey/index.md @@ -0,0 +1,163 @@ +--- +title: Insights from the OpenTelemetry Collector Survey +linkTitle: OpenTelemetry Collector Survey +date: 2024-05-08 +author: '[Hope Oluwalolope](https://github.com/iamebonyhope) (Microsoft)' +# prettier-ignore +cSpell:ignore: attributesprocessor basicauthextension batchprocessor bearertokenauthextension clientauthextension countconnector customizability datadogconnector debugexporter distros filelogreceiver filterprocessor healthcheckextension hostmetricsreceiver k8sattributesprocessor lokiexporter memorylimiterprocessor Oluwalolope otlpexporter otlpreceiver Pprof pprofextension prometheusexporter prometheusreceiver prometheusremotewriteexporter routingconnector sclusterreceiver Servicegraph servicegraphconnector spanmetricsconnector +--- + +The [OpenTelemetry (OTel) Collector](/docs/collector/) has become a cornerstone +tool for observing and monitoring modern software applications. Recently, the +End User SIG conducted a survey to gather feedback from users about their +experiences with the OTel Collector. While we acknowledge that the 186 responses +we received may not be statistically significant, they represent a great start +and provide valuable insights. These insights include details about users’ +deployment practices and implementation challenges, which are instrumental in +helping to drive the OTel Collector’s future direction. + +## Key Takeaways + +- Companies typically have medium to large collector deployments: + - \>5 Collectors: 125/186 + - \>10 Collectors: 100/186 +- Building custom binaries/distributions of the Collector is more popular than + expected (61/186), with most people using the + [OTel Collector Builder](https://github.com/open-telemetry/opentelemetry-collector-builder) + to do so (49/61) +- The vast majority deploy the Collector on Kubernetes (150/186) +- There's more desire for stability (59), self-observability (53), and config + management (59), than for new components (14) + +## Detailed Insights + +### Deployment Scale and Environment + +Our findings indicate robust use of the OTel Collector at scale, with +53.8%(100/186) of respondents deploying more than 10 collectors, 13.4%(25/186) +running between 5 and 10 collectors, and 22%(41/186) running between 2 and 5 +collectors. + +![Chart showing how many otel collectors people run in their organization](deployment-scale.png) + +Kubernetes is the leading platform for Collector deployment (80.6%), followed by +virtual machines (33.3%) and bare metal (10.8%). + +![Chart showing where people deploy their otel collectors](deployment-environment.png) + +### Usage Scenarios + +The OTel Collector is primarily used as a gateway (64.5%), demonstrating its +pivotal role in aggregating telemetry data from various sources. Daemonsets +(51.6%) and sidecars (23.7%) are also popular deployment models, showcasing the +flexibility of the OTel Collector in different operational contexts. + +![Chart showing the usage scenarios of the otel collector](usage-scenarios.png) + +### Customization and Configuration + +A surprising number of people build their own distributions of the Collector +(61/186), indicating that providing a composable Collector is important for the +community. Most people who do build their own Collector distros use the +[OTel Collector Builder (OCB)](https://github.com/open-telemetry/opentelemetry-collector-builder) +(49/61). Of the 49 respondents who leverage the OCB, most were able to figure it +out, with only 2 respondents stating that it’s too hard to use. + +![Chart showing how easy people find it to use the otel collector builder](ocb-usage.png) + +### Monitoring and Observability + +When it comes to monitoring the Collectors, a vast majority of respondents rely +on Collector metrics and logs (81.7%), and only a few don't monitor their +Collector(s) at all (16.6%). When we peered deeper into the data, we found that +of the 125 respondents with more than 5 Collectors, only 15 are not monitoring +their Collectors, and of the 100 respondents with more than 10 Collectors, only +9 are not monitoring their Collectors. This seems to indicate that users take +monitoring their Collectors seriously once they reach a certain maturity with +their Collector deployments. + +![Chart showing how people monitor their otel collector](monitoring.png) + +### OTel Components Usage + +The OTel Collector's flexibility is vividly illustrated by the wide array of +exporters, receivers, processors, connectors, and extensions that are used +across various environments. This highlights the Collector's capability to +integrate with a vast range of tools and systems. + +The top components according to our survey results are as follows: + +### Exporters + +1. [otlpexporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter) +2. [prometheusremotewriteexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/README.md) +3. [prometheusexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md) +4. [lokiexporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md) +5. [debugexporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter) + +### Receivers + +1. [otlpreceiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver) +2. [prometheusreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md) +3. [filelogreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) +4. [hostmetricsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/README.md) +5. [k8sclusterreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sclusterreceiver/README.md) + +### Processors + +1. [batchprocessor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor) +2. [attributesprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md) +3. [filterprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) +4. [memorylimiterprocessor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor) +5. [k8sattributesprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md) + +### Connectors + +1. [spanmetricsconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/spanmetricsprocessor/README.md) +2. [servicegraphconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/servicegraphprocessor/README.md) +3. [routingconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/routingprocessor/README.md) +4. [countconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/countprocessor/README.md) +5. [datadogconnector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/README.md) + +### Extensions + +1. [healthcheckextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/healthcheckextension/README.md) +2. [basicauthextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/basicauthextension/README.md) +3. [pprofextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/pprofextension/README.md) +4. [bearertokenauthextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/bearertokenauthextension/README.md) +5. [oauth2clientauthextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/oauth2clientauthextension/README.md) + +
+ +For a more detailed look at the specific exporters, receivers, processors, +connectors, and extensions in use, you can check out the +[raw results](https://github.com/open-telemetry/sig-end-user/blob/main/end-user-surveys/otel-collector/otel-collector-survey.csv). +The data provides a clear view of the popular choices within the community as +well as the niche configurations that exemplify the customizability of the OTel +Collector. + +### Areas for Improvement + +Respondents were clear about their desire for enhancements in stability (30.6%), +configuration management and resolution (30.1%), and self-observability (28%), +versus wanting new components (<8%) + +![Chart showing interested areas of improvement for the otel collector](areas-of-improvement.png) + +The +[OTel Collector survey results](https://github.com/open-telemetry/sig-end-user/blob/main/end-user-surveys/otel-collector/otel-collector-survey.csv) +offer a snapshot of the current state of Collector deployment and utilization. +It's evident that while the OTel Collector is widely adopted and highly +customizable, there are also opportunities to make it more user-friendly and +robust. + +## Keep in touch + +Thanks to everyone who participated in the survey! We rely on your feedback to +help guide the future development of OpenTelemetry and to ensure it continues to +meet your evolving needs. + +We will post upcoming surveys in the following avenues: +[#otel-sig-end-user Slack channel](https://cloud-native.slack.com/archives/C01RT3MSWGZ) +– you can also reach out to us here! +[End user resources page](/community/end-user/) diff --git a/content/en/blog/2024/otel-collector-survey/monitoring.png b/content/en/blog/2024/otel-collector-survey/monitoring.png new file mode 100644 index 000000000000..f96529100c9c Binary files /dev/null and b/content/en/blog/2024/otel-collector-survey/monitoring.png differ diff --git a/content/en/blog/2024/otel-collector-survey/ocb-usage.png b/content/en/blog/2024/otel-collector-survey/ocb-usage.png new file mode 100644 index 000000000000..08a35f95e374 Binary files /dev/null and b/content/en/blog/2024/otel-collector-survey/ocb-usage.png differ diff --git a/content/en/blog/2024/otel-collector-survey/usage-scenarios.png b/content/en/blog/2024/otel-collector-survey/usage-scenarios.png new file mode 100644 index 000000000000..a7d8a12eeb0d Binary files /dev/null and b/content/en/blog/2024/otel-collector-survey/usage-scenarios.png differ diff --git a/static/refcache.json b/static/refcache.json index 6ea30781ffec..c649381ecad8 100644 --- a/static/refcache.json +++ b/static/refcache.json @@ -2631,6 +2631,10 @@ "StatusCode": 200, "LastSeen": "2024-01-30T16:15:04.170576-05:00" }, + "https://github.com/iamebonyhope": { + "StatusCode": 200, + "LastSeen": "2024-05-08T10:07:06.998803+02:00" + }, "https://github.com/imandra-ai/ocaml-opentelemetry/": { "StatusCode": 200, "LastSeen": "2024-01-18T19:13:40.755331-05:00" @@ -2999,6 +3003,10 @@ "StatusCode": 200, "LastSeen": "2024-01-18T19:36:40.22377-05:00" }, + "https://github.com/open-telemetry/opentelemetry-collector-builder": { + "StatusCode": 200, + "LastSeen": "2024-05-08T10:07:07.582195+02:00" + }, "https://github.com/open-telemetry/opentelemetry-collector-contrib": { "StatusCode": 200, "LastSeen": "2024-01-18T20:05:30.222833-05:00"