Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Collector security documentation #5209

Merged
merged 48 commits into from
Nov 2, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
631930d
Edit security index.md
tiffany76 Sep 17, 2024
2dc2e17
Copy content into config-best-practices.md
tiffany76 Sep 17, 2024
0d88c24
Merge branch 'main' into collector-security
tiffany76 Sep 19, 2024
135fc1d
Fix spelling issues
tiffany76 Sep 19, 2024
6063b77
Make linter fixes
tiffany76 Sep 19, 2024
2de00e2
Update links on index.md
tiffany76 Sep 19, 2024
7cbbcf6
Copy content into hosting-best-practices.md
tiffany76 Sep 19, 2024
b4987c3
Add TODO
tiffany76 Sep 19, 2024
79688ec
Edits to config and hosting best practices
tiffany76 Sep 19, 2024
be0d8ee
Make Prettier fix
tiffany76 Sep 19, 2024
3e7c3c9
Apply suggestions from Juraci
tiffany76 Sep 23, 2024
932a3d5
Merge branch 'main' into collector-security
tiffany76 Sep 23, 2024
a1427f2
Update index.md with PII no-no
tiffany76 Sep 23, 2024
cc131a8
Add headings for child pages to index.md
tiffany76 Sep 23, 2024
cbf67a9
Update config receivers and exporters section
tiffany76 Sep 24, 2024
02d0944
Make link and linter fixes
tiffany76 Sep 24, 2024
3cee030
Merge branch 'main' into collector-security
tiffany76 Sep 24, 2024
500d331
Update DOS safeguard section
tiffany76 Sep 24, 2024
0a05a95
Adjust info architecture of all pages
tiffany76 Sep 24, 2024
b53353a
Merge branch 'main' into collector-security
tiffany76 Sep 26, 2024
18283d5
Edit protocol section
tiffany76 Sep 27, 2024
bb8852f
Change headings and info arch
tiffany76 Sep 27, 2024
0532e02
Edit scrubbing sensitive data section
tiffany76 Sep 27, 2024
92eccee
Merge branch 'main' into collector-security
tiffany76 Oct 6, 2024
36d55c1
Create new top level section and rework content
tiffany76 Oct 6, 2024
c95ba0c
Create 'specific risk' section
tiffany76 Oct 6, 2024
dbfd1b6
Remove forwarding section
tiffany76 Oct 6, 2024
2a582e7
Merge branch 'main' into collector-security
tiffany76 Oct 14, 2024
8605abb
Edit resource utlization section
tiffany76 Oct 14, 2024
44ae838
Minor edits
tiffany76 Oct 15, 2024
2e606e2
Make cSpell happy
tiffany76 Oct 15, 2024
fb566d1
Apply suggestions from Pablo's review
tiffany76 Oct 18, 2024
6d9150f
Merge branch 'main' into collector-security
tiffany76 Oct 18, 2024
961ac47
Fix formatting and remove sections
tiffany76 Oct 18, 2024
af95741
Rewrite and move observers section
tiffany76 Oct 18, 2024
73f4c38
Rewrite hosting best practices
tiffany76 Oct 18, 2024
3f856e9
Apply suggestions from Pablo's second review
tiffany76 Oct 19, 2024
faf27e4
Merge branch 'main' into collector-security
tiffany76 Oct 28, 2024
f099457
Rewrite index page opening
tiffany76 Oct 28, 2024
3b86e29
Rewrite least privilege section
tiffany76 Oct 28, 2024
ea36f37
Results from /fix:refcache
opentelemetrybot Oct 28, 2024
a496603
Merge branch 'main' into collector-security
reyang Oct 29, 2024
3eceaba
Update content/en/docs/security/hosting-best-practices.md
tiffany76 Oct 29, 2024
cba2e7c
Merge branch 'main' into collector-security
tiffany76 Oct 31, 2024
5f9dff7
Merge branch 'main' into collector-security
tiffany76 Nov 1, 2024
ac9713f
Apply fixes from Patrice
tiffany76 Nov 2, 2024
74ce955
Results from /fix:format
opentelemetrybot Nov 2, 2024
c13f118
Merge branch 'main' into collector-security
tiffany76 Nov 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions content/en/docs/security/_index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,28 @@
---
title: Security
weight: 970
title: Security in OpenTelemetry
description: Security in OpenTelemetry
weight: 10
---

<!--- TODO: Add content to introduce CVE and security response docs --->

When setting up the OpenTelemetry (OTel) Collector, consider implementing
security best practices in both your hosting infrastructure and your OTel
Collector configuration. Running a secure Collector can help you

- Protect telemetry that might contain sensitive information, such as personally
identifiable information (PII), application-specific data, or network traffic
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure we have a guidance somewhere stating that people should not store PII in telemetry data. Perhaps we should reinforce it here, and add a link to that place? Something like: "help you protect telemetry that shouldn't, but might contain sensitive information, such ..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still searching for the existing guidance, but I've updated the wording as suggested. I'll keep looking for the PII reference.

patterns.
- Prevent data tampering that makes telemetry unreliable and disrupts incident
responses.
- Comply with data privacy and security regulations.
- Defend against denial of service (DoS) attacks.

See [Hosting best practices](/security/hosting-best-practices) to learn how to
secure your Collector's infrastructure.

See [Configuration best practices](/security/config-best-practices) to learn how
to securely configure your Collector.

For Collector component developers, see
[Security best practices](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md).
155 changes: 155 additions & 0 deletions content/en/docs/security/config-best-practices.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
---
title: Collector configuration best practices
linkTitle: Collector config
description: Best practices to securely configure OpenTelemetry Collector.
weight: 20
---

When configuring the OpenTelemetry (OTel) Collector, consider the following
practices to better secure your Collector instance.

## Receivers and exporters

We recommend enabling only the minimum required components for a Collector
configuration: one receiver and one exporter. Configuring only the minimum set
reyang marked this conversation as resolved.
Show resolved Hide resolved
of components exposes a minimum set of required ports.
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved

- Receivers and exporters can be either push- or pull-based. In either case, the
connection established should be over a secure and authenticated channel.
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved
- Unused receivers and exporters should be disabled to minimize the attack
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved
vector of the Collector.
- Receivers and exporters might expose buffer, queue, payload, and worker
settings using configuration parameters. If these settings are available, you
should proceed with caution before modifying the default configuration values.
Improperly setting these values might expose the OpenTelemetry Collector to
additional attack vectors including resource exhaustion.
reyang marked this conversation as resolved.
Show resolved Hide resolved

### Safeguards against denial of service attacks

Bind receivers' servers to addresses that limit connections to authorized users,
so that your Collectors aren't exposed to the public internet or to wider
networks than necessary.

For example, if the OTLP receiver OTLP/gRPC server has only local clients, bind
the `endpoint` setting to `localhost`:

```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317
reyang marked this conversation as resolved.
Show resolved Hide resolved
```

Use `localhost`-like addresses instead of `0.0.0.0`. For more information, see
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved
[CWE-1327: Binding to an Unrestricted IP Address](https://cwe.mitre.org/data/definitions/1327.html).

To change the default endpoint to be `localhost`-bound in all components, enable
the `component.UseLocalHostAsDefaultHost` feature gate. This feature gate will
be enabled by default in the Collector in a future release.
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved

### Encryption and authentication

Your OTel Collector configuration should include encryption and authentication.

- For communication encryption, see
[Configuring certificates](/collector/configuration/#setting-up-certificates).
- For authentication, use the OTel Collector's authentication mechanism, as
described in [Authentication](/collector/configuration/#authentication).

## Processors

[Processors](/collector/configuration/#processors) sit between receivers and
exporters. They are responsible for processing telemetry before it's analyzed.
From a security perspective, processors are useful in a few ways.

### Recommended processors
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved

<!--- TODO: SHOULD configure recommended processors. If so, what are they?

redaction processor (never tried myself)
transform processor (great for redacting PIIs and such) -->

### Scrubbing sensitive data

You can use the OpenTelemetry Collector to scrub sensitive data before exporting
it to a backend. Configure the Collector to obfuscate or scrub sensitive data
before exporting.

<!--- TODO: SHOULD configure obfuscation/scrubbing of sensitive metadata. How? Give more details and/or link to an existing document -->

Use OpenTelemetry Collector's `redaction` processor to scrub sensitive data.

<!--- TODO: Give example config for the redaction processor or remove this line. --->

### Safeguarding resource utilization

Processors also offer safeguards for resource utilization. After implementing
safeguards for resource utilization in your
[hosting infrastructure](/security/hosting-best-practices/), make sure your
OpenTelemetry Collector configuration uses these safeguards.

<!-- start same page content in hosting-best-practices -->

The `batch` and `memory_limiter` processors help ensure the OpenTelemetry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the discussions we are having around the batch processor, we should probably stop advertising it.

cc @mx-psi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a second thought, perhaps we should stop talking about the memory limiter processor as well, mentioning the extension instead once it gets ready?

https://github.com/open-telemetry/opentelemetry-collector/tree/main/extension/memorylimiterextension

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on @reyang's earlier comment that resource utilization is an important topic, it seems like we should say something about best practices here. If we don't recommend the batch and memory limiter processors, and the memory limiter extension is still in development, how should we advise users to safeguard resource utilization in their Collector configurations and hosting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a shot at writing this section on my own (8605abb), but it could definitely use a developer's review.

I mention batching and memory limiting, but I don't call out the processors or extensions specifically. Once the memory limiter extension is ready, we can update with an example.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a general mention to batching and memory limiting should be more future-proof

Collector is resource efficient and does not run out of memory when overloaded.
These two processors should be enabled on every defined pipeline.

For more information on recommended processors and how to order them in your
configuration, see the
[Collector processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor)
documentation.

<!-- /end same page content in hosting-best-practices -->

## Extensions

While receivers, processors, and exporters handle telemetry directly, extensions
serve different needs.

<!--- TODO: Extensions SHOULD NOT expose sensitive health or telemetry data. How? What can you do? -->

### Health and telemetry

Extensions are available for health check information, Collector metrics and
traces, and generating and collecting profiling data. When enabled with their
default settings, all of these extensions, except the health check extension,
are accessible only locally to the OpenTelemetry Collector. Take care to protect
sensitive information when configuring these extensions for remote access, as
they might expose it accidentally.
mx-psi marked this conversation as resolved.
Show resolved Hide resolved

### Forwarding

A forwarding extension is used when you need to collect telemetry that's not
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved
natively supported by the Collector. For example, the `http_forwarder` extension
can receive and forward HTTP payloads. Forwarding extensions are similar to
receivers and exporters so the same security considerations apply.

### Collector's internal telemetry

<!--- INSERT RECOMMENDATIONS HERE. For example:

1. Remove zPages.
1. Remove configuration endpoints.
-->
mx-psi marked this conversation as resolved.
Show resolved Hide resolved

### Observers

An observer is a component that discovers services in endpoints. Other
components of the OpenTelemetry Collector, such as receivers, can subscribe to
these extensions to be notified of endpoints coming or going.

Observers might require certain permissions in order to discover services. For
example, the `k8s_observer` requires certain RBAC permissions in Kubernetes,
while the `host_observer` requires the OpenTelemetry Collector to run in
privileged mode.

<!--- But what about Juraci's comment here: https://github.com/open-telemetry/opentelemetry.io/pull/3652/files?diff=unified&w=0#r1417409370 --->
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved

### Subprocesses

Extensions can also be used to run subprocesses when the Collector can't
natively run the collection mechanisms (for example, FluentBit). Subprocesses
mx-psi marked this conversation as resolved.
Show resolved Hide resolved
expose a completely separate attack vector that depends on the subprocess
itself. In general, take care before running any subprocesses alongside the
Collector.
61 changes: 61 additions & 0 deletions content/en/docs/security/hosting-best-practices.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: Collector hosting best practices
linkTitle: Collector hosting
description: Follow best practices to securely host OpenTelemetry Collector.
weight: 15
---

When setting up hosting for OpenTelemetry (OTel) Collector, consider the
following practices to better secure your hosting instance.

## Storing configuration information securely

<!--- TODO: SHOULD ensure sensitive configuration information is stored securely. How? -->
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved

## Permissions

<!--- TODO: SHOULD not run the OpenTelemetry Collector as root/admin user. Why? (Give the reader motivation.) How do you do that?
- NOTE: MAY require privileged access for some components

The Collector SHOULD NOT require privileged access, except where the data it's obtaining is in a privileged location. For instance, in order to get pod logs by mounting a node volume, the Collector daemonset needs enough privileges to get that data.

The rule of least privilege applies here. --->

## Receivers and exporters

To limit the exposure of servers to authorized users:

- Enable authentication, using bearer token authentication extensions and basic
tiffany76 marked this conversation as resolved.
Show resolved Hide resolved
authentication extensions.
- Restrict the IPs that OTel Collector runs on.

## Processors

[Processors](/collector/configuration/#processors) sit between receivers and
exporters. They are responsible for processing telemetry before it's analyzed.
From a security perspective, processors are useful in a few ways.

### Safeguarding resource utilization

In addition, processors offer safeguards around resource utilization.

The `batch` and `memory_limiter` processors help ensure that the OpenTelemetry
Collector is resource efficient and does not run out memory when overloaded.
These two processors should be enabled on every defined pipeline.

For more information on recommended processors and how to order them in your
configuration, see the
[Collector processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor)
documentation.

After installing resource utilization safeguards in your hosting, make sure your
Collector configuration uses those
[safeguards in its configuration](/security/config-best-practices/).

### Another example

<!--- TODO: INSERT ADDITIONAL EXAMPLES HERE. -->

## Extensions

<!--- TODO: Extensions SHOULD NOT expose sensitive health or telemetry data. How? What can you do? -->
Loading