-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Collector security documentation #5209
Changes from 10 commits
631930d
2dc2e17
0d88c24
135fc1d
6063b77
2de00e2
7cbbcf6
b4987c3
79688ec
be0d8ee
3e7c3c9
932a3d5
a1427f2
cc131a8
cbf67a9
02d0944
3cee030
500d331
0a05a95
b53353a
18283d5
bb8852f
0532e02
92eccee
36d55c1
c95ba0c
dbfd1b6
2a582e7
8605abb
44ae838
2e606e2
fb566d1
6d9150f
961ac47
af95741
73f4c38
3f856e9
faf27e4
f099457
3b86e29
ea36f37
a496603
3eceaba
cba2e7c
5f9dff7
ac9713f
74ce955
c13f118
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,28 @@ | ||
--- | ||
title: Security | ||
weight: 970 | ||
title: Security in OpenTelemetry | ||
description: Security in OpenTelemetry | ||
weight: 10 | ||
--- | ||
|
||
<!--- TODO: Add content to introduce CVE and security response docs ---> | ||
|
||
When setting up the OpenTelemetry (OTel) Collector, consider implementing | ||
security best practices in both your hosting infrastructure and your OTel | ||
Collector configuration. Running a secure Collector can help you | ||
|
||
- Protect telemetry that might contain sensitive information, such as personally | ||
identifiable information (PII), application-specific data, or network traffic | ||
patterns. | ||
- Prevent data tampering that makes telemetry unreliable and disrupts incident | ||
responses. | ||
- Comply with data privacy and security regulations. | ||
- Defend against denial of service (DoS) attacks. | ||
|
||
See [Hosting best practices](/security/hosting-best-practices) to learn how to | ||
secure your Collector's infrastructure. | ||
|
||
See [Configuration best practices](/security/config-best-practices) to learn how | ||
to securely configure your Collector. | ||
|
||
For Collector component developers, see | ||
[Security best practices](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
--- | ||
title: Collector configuration best practices | ||
linkTitle: Collector config | ||
description: Best practices to securely configure OpenTelemetry Collector. | ||
weight: 20 | ||
--- | ||
|
||
When configuring the OpenTelemetry (OTel) Collector, consider the following | ||
practices to better secure your Collector instance. | ||
|
||
## Receivers and exporters | ||
|
||
We recommend enabling only the minimum required components for a Collector | ||
configuration: one receiver and one exporter. Configuring only the minimum set | ||
reyang marked this conversation as resolved.
Show resolved
Hide resolved
|
||
of components exposes a minimum set of required ports. | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- Receivers and exporters can be either push- or pull-based. In either case, the | ||
connection established should be over a secure and authenticated channel. | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- Unused receivers and exporters should be disabled to minimize the attack | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
vector of the Collector. | ||
- Receivers and exporters might expose buffer, queue, payload, and worker | ||
settings using configuration parameters. If these settings are available, you | ||
should proceed with caution before modifying the default configuration values. | ||
Improperly setting these values might expose the OpenTelemetry Collector to | ||
additional attack vectors including resource exhaustion. | ||
reyang marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Safeguards against denial of service attacks | ||
|
||
Bind receivers' servers to addresses that limit connections to authorized users, | ||
so that your Collectors aren't exposed to the public internet or to wider | ||
networks than necessary. | ||
|
||
For example, if the OTLP receiver OTLP/gRPC server has only local clients, bind | ||
the `endpoint` setting to `localhost`: | ||
|
||
```yaml | ||
receivers: | ||
otlp: | ||
protocols: | ||
grpc: | ||
endpoint: localhost:4317 | ||
reyang marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
Use `localhost`-like addresses instead of `0.0.0.0`. For more information, see | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
[CWE-1327: Binding to an Unrestricted IP Address](https://cwe.mitre.org/data/definitions/1327.html). | ||
|
||
To change the default endpoint to be `localhost`-bound in all components, enable | ||
the `component.UseLocalHostAsDefaultHost` feature gate. This feature gate will | ||
be enabled by default in the Collector in a future release. | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Encryption and authentication | ||
|
||
Your OTel Collector configuration should include encryption and authentication. | ||
|
||
- For communication encryption, see | ||
[Configuring certificates](/collector/configuration/#setting-up-certificates). | ||
- For authentication, use the OTel Collector's authentication mechanism, as | ||
described in [Authentication](/collector/configuration/#authentication). | ||
|
||
## Processors | ||
|
||
[Processors](/collector/configuration/#processors) sit between receivers and | ||
exporters. They are responsible for processing telemetry before it's analyzed. | ||
From a security perspective, processors are useful in a few ways. | ||
|
||
### Recommended processors | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
<!--- TODO: SHOULD configure recommended processors. If so, what are they? | ||
|
||
redaction processor (never tried myself) | ||
transform processor (great for redacting PIIs and such) --> | ||
|
||
### Scrubbing sensitive data | ||
|
||
You can use the OpenTelemetry Collector to scrub sensitive data before exporting | ||
it to a backend. Configure the Collector to obfuscate or scrub sensitive data | ||
before exporting. | ||
|
||
<!--- TODO: SHOULD configure obfuscation/scrubbing of sensitive metadata. How? Give more details and/or link to an existing document --> | ||
|
||
Use OpenTelemetry Collector's `redaction` processor to scrub sensitive data. | ||
|
||
<!--- TODO: Give example config for the redaction processor or remove this line. ---> | ||
|
||
### Safeguarding resource utilization | ||
|
||
Processors also offer safeguards for resource utilization. After implementing | ||
safeguards for resource utilization in your | ||
[hosting infrastructure](/security/hosting-best-practices/), make sure your | ||
OpenTelemetry Collector configuration uses these safeguards. | ||
|
||
<!-- start same page content in hosting-best-practices --> | ||
|
||
The `batch` and `memory_limiter` processors help ensure the OpenTelemetry | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Given the discussions we are having around the batch processor, we should probably stop advertising it. cc @mx-psi There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. After a second thought, perhaps we should stop talking about the memory limiter processor as well, mentioning the extension instead once it gets ready? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Based on @reyang's earlier comment that resource utilization is an important topic, it seems like we should say something about best practices here. If we don't recommend the batch and memory limiter processors, and the memory limiter extension is still in development, how should we advise users to safeguard resource utilization in their Collector configurations and hosting? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I took a shot at writing this section on my own (8605abb), but it could definitely use a developer's review. I mention batching and memory limiting, but I don't call out the processors or extensions specifically. Once the memory limiter extension is ready, we can update with an example. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think a general mention to batching and memory limiting should be more future-proof |
||
Collector is resource efficient and does not run out of memory when overloaded. | ||
These two processors should be enabled on every defined pipeline. | ||
|
||
For more information on recommended processors and how to order them in your | ||
configuration, see the | ||
[Collector processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor) | ||
documentation. | ||
|
||
<!-- /end same page content in hosting-best-practices --> | ||
|
||
## Extensions | ||
|
||
While receivers, processors, and exporters handle telemetry directly, extensions | ||
serve different needs. | ||
|
||
<!--- TODO: Extensions SHOULD NOT expose sensitive health or telemetry data. How? What can you do? --> | ||
|
||
### Health and telemetry | ||
|
||
Extensions are available for health check information, Collector metrics and | ||
traces, and generating and collecting profiling data. When enabled with their | ||
default settings, all of these extensions, except the health check extension, | ||
are accessible only locally to the OpenTelemetry Collector. Take care to protect | ||
sensitive information when configuring these extensions for remote access, as | ||
they might expose it accidentally. | ||
mx-psi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Forwarding | ||
|
||
A forwarding extension is used when you need to collect telemetry that's not | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
natively supported by the Collector. For example, the `http_forwarder` extension | ||
can receive and forward HTTP payloads. Forwarding extensions are similar to | ||
receivers and exporters so the same security considerations apply. | ||
|
||
### Collector's internal telemetry | ||
|
||
<!--- INSERT RECOMMENDATIONS HERE. For example: | ||
|
||
1. Remove zPages. | ||
1. Remove configuration endpoints. | ||
--> | ||
mx-psi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Observers | ||
|
||
An observer is a component that discovers services in endpoints. Other | ||
components of the OpenTelemetry Collector, such as receivers, can subscribe to | ||
these extensions to be notified of endpoints coming or going. | ||
|
||
Observers might require certain permissions in order to discover services. For | ||
example, the `k8s_observer` requires certain RBAC permissions in Kubernetes, | ||
while the `host_observer` requires the OpenTelemetry Collector to run in | ||
privileged mode. | ||
|
||
<!--- But what about Juraci's comment here: https://github.com/open-telemetry/opentelemetry.io/pull/3652/files?diff=unified&w=0#r1417409370 ---> | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Subprocesses | ||
|
||
Extensions can also be used to run subprocesses when the Collector can't | ||
natively run the collection mechanisms (for example, FluentBit). Subprocesses | ||
mx-psi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
expose a completely separate attack vector that depends on the subprocess | ||
itself. In general, take care before running any subprocesses alongside the | ||
Collector. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
--- | ||
title: Collector hosting best practices | ||
linkTitle: Collector hosting | ||
description: Follow best practices to securely host OpenTelemetry Collector. | ||
weight: 15 | ||
--- | ||
|
||
When setting up hosting for OpenTelemetry (OTel) Collector, consider the | ||
following practices to better secure your hosting instance. | ||
|
||
## Storing configuration information securely | ||
|
||
<!--- TODO: SHOULD ensure sensitive configuration information is stored securely. How? --> | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Permissions | ||
|
||
<!--- TODO: SHOULD not run the OpenTelemetry Collector as root/admin user. Why? (Give the reader motivation.) How do you do that? | ||
- NOTE: MAY require privileged access for some components | ||
|
||
The Collector SHOULD NOT require privileged access, except where the data it's obtaining is in a privileged location. For instance, in order to get pod logs by mounting a node volume, the Collector daemonset needs enough privileges to get that data. | ||
|
||
The rule of least privilege applies here. ---> | ||
|
||
## Receivers and exporters | ||
|
||
To limit the exposure of servers to authorized users: | ||
|
||
- Enable authentication, using bearer token authentication extensions and basic | ||
tiffany76 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
authentication extensions. | ||
- Restrict the IPs that OTel Collector runs on. | ||
|
||
## Processors | ||
|
||
[Processors](/collector/configuration/#processors) sit between receivers and | ||
exporters. They are responsible for processing telemetry before it's analyzed. | ||
From a security perspective, processors are useful in a few ways. | ||
|
||
### Safeguarding resource utilization | ||
|
||
In addition, processors offer safeguards around resource utilization. | ||
|
||
The `batch` and `memory_limiter` processors help ensure that the OpenTelemetry | ||
Collector is resource efficient and does not run out memory when overloaded. | ||
These two processors should be enabled on every defined pipeline. | ||
|
||
For more information on recommended processors and how to order them in your | ||
configuration, see the | ||
[Collector processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor) | ||
documentation. | ||
|
||
After installing resource utilization safeguards in your hosting, make sure your | ||
Collector configuration uses those | ||
[safeguards in its configuration](/security/config-best-practices/). | ||
|
||
### Another example | ||
|
||
<!--- TODO: INSERT ADDITIONAL EXAMPLES HERE. --> | ||
|
||
## Extensions | ||
|
||
<!--- TODO: Extensions SHOULD NOT expose sensitive health or telemetry data. How? What can you do? --> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty sure we have a guidance somewhere stating that people should not store PII in telemetry data. Perhaps we should reinforce it here, and add a link to that place? Something like: "help you protect telemetry that shouldn't, but might contain sensitive information, such ..."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still searching for the existing guidance, but I've updated the wording as suggested. I'll keep looking for the PII reference.