wip: Establish Security section for OTel documentation

[mjingle]

OpenTelemetry (OTel) currently does not have public documentation on security recommendations. A markdown file in the OpenTelemetry Collector repository (linked below) does address some concerns, but is not easily findable by end users.

The goal of this PR is to establish hosting and configuration security recommendations for OTel Collector end users within the OpenTelemetry public documentation site. While the inspiration document contains both end-user and Component Developer security recommendations, this public documentation adapts the existing content focused on end users and links back to the inspiration document for Component Developers.

NOTE: The inspiration document also does not cover topics in detail and I need help filling in those blanks. Refer to any <!-- TODO: --> within the files that indicates knowledge gaps. From there, the page content can be re-organized, given the available information.

This PR adds:

• a Security Overview page
• an OpenTelemetry Collector hosting best practices page
• an OpenTelemetry Collector configuration best practices page

This proposed structure tries to be flexible for future Security content pages to be added. (Additional content doesn't have to focus on OTel Collector. Other Security-related areas in OTel can be addressed also, but that's not within the scope of this PR.)

Identified Areas of Potential Help

I already know this PR needs some extra help and this is my first time contributing a PR to this project. These include:

• Filling in content gaps. (Refer to NOTE: above.)
• Correct wiring for the menu. My goal is to have Security as a top level menu option.
• Links to related content within the OTel community documentation, if applicable.
• Amending the inspiration document in the OTel Collector repo to focus on Component Developers only.

Source Materials

https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md

Associated Issues

Initial issue: #3479
Related issue: #3227

---

Preview: https://deploy-preview-3652--opentelemetry.netlify.app/docs/security/

[mjingle]

Am not familiar with Observers. Is this content related to component developers? If so, I can remove it from this PR for end users.

[jpkrohling]

This is for end-users. Perhaps an example would be useful here, for users to understand what they can do with observers.

Here's one such example, where a redis receiver is created if a redis pod is found on a Kubernetes cluster:

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/observer/k8sobserver

cc @rmfitzpatrick , @dmitryax

[mjingle]

Amended the header here, since Observers are for end-users. 👍

An example here would be good to have. Still need to incorporate what you shared, @jpkrohling, but I appreciate the resource you shared!

[jpkrohling]

This has a mixed audience: the first sentence targets component developers, then it proceeds with advice for users. I would rewrite as:

Some receivers and exporters might allow you tweak internal queues, buffers, and other parameters. Experiment with these values, but remember that values that are too wide (such as a large queue number) might mean that the Collector will attempt to use more memory than a setting with a smaller value.

[mjingle]

Thank you, @jpkrohling! For values that are too wide, rather than:

remember that values that are too wide (such as a large queue number) might mean that the Collector will attempt to use more memory than a setting with a smaller value.

Would it be more accurate to describe as:

remember that if the values are too wide, such as a large queue number, then the OTel Collector may use more memory than necessary, compared to using a setting with a smaller value.

[mjingle]

@jpkrohling Circling back to ask your feedback on the above ^

[jpkrohling]

I would remove this if we don't have any such component today.

[mjingle]

Can we ask a specific person or team for confirmation about this assertion? (This sentence is located in the inspiration document here:https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md?plain=1#L124 )

[mjingle]

@jpkrohling Circling back to ask if there's an answer for the above-mentioned question?

[jpkrohling]

The transform processor is the best one for this use case nowadays: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor

[jpkrohling]

This is for end-users. Perhaps an example would be useful here, for users to understand what they can do with observers.

Here's one such example, where a redis receiver is created if a redis pod is found on a Kubernetes cluster:

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/observer/k8sobserver

cc @rmfitzpatrick , @dmitryax

[jpkrohling]

The Collector SHOULD NOT require privileged access, except where the data it's obtaining is in a privileged location. For instance, in order to get pod logs by mounting a node volume, the Collector daemonset needs enough privileges to get that data.

The rule of least privilege applies here.

[jpkrohling]

• Authentication (bearer token auth extension, basic auth extension)
• By restricting the IPs the collector is running

[chalin]

Initial drive-by comments.

[svrnm]

Thinking about the "where to put this" once again: We now have established /docs/security/ and have the CVE there and will also have more content like #3675 as well. This PR contains security guidelines for the collector, so we have technically 2 places to house this:

1. /docs/security/collector
2. /docs/collector/security

I think we should go for option (2) but at the same time add a "Practices" page under /docs/security/practices that will list collector (and later other components as well) and link back to the component specific security, wdyt?

[jpkrohling]

Thinking out "loud", there are two moments I'm concerned about security for my collector:

• when evaluating the collector
• when hardening it right before going into production

For the first situation, I'd prefer the security documentation to be close to the getting started, so that I can have an idea whether the collector satisfies my security needs. During the second situation, I think I'd prefer it close to the collector documentation again, as I probably don't care about the security aspects of the SDK or other parts.

In summary: I'm with you that placing it closer to the collector makes sense, as long as there's a reference to it from within the /security pages.

[cartersocha]

I vote put it in the collector docs and agree we can link out to it

[mjingle]

@svrnm My question is why not have the content in both locations? (This may not be possible with the functionality/architecture of the OTel docs site, but I still ask the question.)

I agree that when setting up the OTel Collector, information on securing your OTel Collector is very relevant and should exist within the OTel Collector documentation.

In the original vision by establishing this Security section, we're also:

1. trying to highlight to potential OTel adopters that you can be secure and open source.
2. encouraging security best practices related to OpenTelemetry...beyond the OTel Collector.

[svrnm]

@svrnm My question is why not have the content in both locations? (This may not be possible with the functionality/architecture of the OTel docs site, but I still ask the question.)

We can certainly do this (somehow), but this often comes with some additional questions: is this a redirect (so people go from /security to /collector), is it just the same content presented at 2 places, what is the point where this list outgrows that (I assume that we eventually will have security best practices across the docs).

To make it short: yes, we can do that, but we might reconsider it eventually.

I agree that when setting up the OTel Collector, information on securing your OTel Collector is very relevant and should exist within the OTel Collector documentation.

In the original vision by establishing this Security section, we're also:

1. trying to highlight to potential OTel adopters that you can be secure and open source.

2. encouraging security best practices related to OpenTelemetry...beyond the OTel Collector.

This seems to be untouched by the decision where this content lives. Even if we have it only in the collector section I would want to have a /security/practices page that gives a general overview and then links out to relevant pages like the collector security pages.

[theletterf]

Could we explain briefly in the intro what's at stake here? Why should someone care about securing the OpenTelemetry Collector? Is it a painful process? What knowledge is required?

[mjingle]

You ask great questions @theletterf! If you have the answers, I'm happy to put them in this document. I defer to the SMEs to answer these questions otherwise. At the very least, I will file issues if these questions aren't answered or incorporated by the time we merge this PR.

[theletterf]

I'd remove this line or provide an example in a subsection.

[theletterf]

Suggested change

	extension are only accessibly locally to the OpenTelemetry Collector. Care
	should be taken when configuring these extensions for remote access as sensitive
	information may be exposed as a result.
	extension are only accessibly locally to the OpenTelemetry Collector. Take
	care to protect access to sensitive information when configuring these
	extensions, as they might expose it accidentally.

[theletterf]

Suggested change

	An observer is capable of performing service discovery of endpoints. Other
	components of the OpenTelemetry Collector such as receivers MAY subscribe to
	these extensions to be notified of endpoints coming or going. Observers MAY
	require certain permissions in order to perform service discovery. For example,
	the `k8s_observer` requires certain RBAC permissions in Kubernetes, while the
	`host_observer` requires the OpenTelemetry Collector to run in privileged mode.
	An observer is a component that discovers services in endpoints. Other
	components of the OpenTelemetry Collector such as receivers can subscribe to
	these extensions to be notified of endpoints coming or going.
	Observers might require certain permissions in order to discover services.
	For example, the `k8s_observer` requires certain RBAC permissions in
	Kubernetes, while the `host_observer` observer requires to run the
	OpenTelemetry Collector in privileged mode.

[theletterf]

@mjingle Thanks SO much for starting this. This is necessary. Hope you'll find my suggestions useful.

[svrnm]

@mjingle any updates on this issue? Anything we can do to help you proceed?

[svrnm]

@mjingle any updates for this PR?

[chalin]

All: I've rebased & resolved conflicts so that we can get a fresher view of this addition in the context of the updated website -- a lot has changed since this was submitted! :)

[theletterf]

@mjingle LMK if you need to discuss the draft or need more direct support — my comments are not meant to be blockers. Would be happy to commit to your fork if you want.

[cartermp]

Closing old PR (and some unfortunate conflicts, too)