Support send data to remote Otel collector

[jvanz]

This epic describe the changes required to allow Kubewarden to integrate with other kinds of OpenTelemetry collector deployments modes. Right now, the only supported is to send data to OpenTelemetry collector running as sidecar. However sidecar does not allow fetch metadata from the Kubernetes cluster. Therefore, we decided to review how we integrate with Otel collectors. This is a spin off of this issue

This epic has two major goals:

• Keep supporting the current way of integration using Otel collector sidecar with the minimum changes
• Support Kubewarden stack send the metrics and tracing data to a remote Otel collector running in the cluster (e.g. deaemonset or deployment) using a secure GRPc communication (TLS)

To allow us to integrate better with Otel collectors we need to allow the Kubewarden controller and the Policy server created by it to send data to collectors running somewhere else in the cluster. This means that we need to configure the exporter in the controller and the policy server to send data to a remote endpoint. Gladly, most of the configuration of the used libraries (go and rust) allow the users to configure the endpoint using environment variables. Besides that, controller already have a CLI flag that allow Otel endpoint customization which the default values is localhost (the sidecar endpoint). Policy server code does not have this CLI flag. But the rust library allow customization using environment variables

For the Go code, both traces and metrics exporters in the OpenTelemetry packages have environment variables used to define endpoints, certificates and many more configuration to send data to the Otel collector. For example:

OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (default: "https://localhost:4317") - target to which the exporter sends telemetry. The target syntax is defined in https://github.com/grpc/grpc/blob/master/doc/naming.md. The value must contain a scheme ("http" or "https") and host. The value may additionally contain a port, and a path. The value should not contain a query string or fragment. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT takes precedence over OTEL_EXPORTER_OTLP_ENDPOINT. The configuration can be overridden by WithEndpoint, WithEndpointURL, WithInsecure, and WithGRPCConn options.

OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE (default: none) - the filepath to the trusted certificate to use when verifying a server's TLS credentials. OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE takes precedence over OTEL_EXPORTER_OTLP_CERTIFICATE. The configuration can be overridden by WithTLSCredentials, WithGRPCConn options.

Therefore, we can use this environment variables to configure the exporter with minimum need of change in the code base. But it's necessary to configure this values in the controller deployment and allow the mounting of the certificate files in the pods.

For the Rust code base, it's also possible to define the endpoint using environment variables. However, it's not possible to define the path of the certificate to be used as in the Go packages (there is an open issue for that). Therefore, the policy server code should be change to add the certificate defined by the user using a environment variable or a CLI flag.

Acceptance criteria

• Support send data to remote Otel collector #933
• Allow configuration to send metric/tracing data to remote Otel collector  helm-charts#591
  • This issue is a duplicate of this.
• Support send data to remote Otel collector policy-server#993
• Update documentation how to configure Kubewarden to send data to a remove Otel collector docs#475
• Add integration test for the new integration with remote Otel collector

[flavio]

This is an exhaustive card, thanks for the research work @jvanz 👏

I envision the user setting the otel configuration values (the endpoint address and the name of the Secret/ConfigMap holding the CA) with our helm chart. This will then be translated by the helm chart either using env variables or flags.

I think the tricky part is how to propagate that to the policy-server, especially the CA to be trusted. That's because the kubewarden-controller isn't aware of the origin of the CA (what is the name and namespace of the Secret/ConfigMap).

Another option could be to let the user configure this detail on the PolicyServer CRD. The kubewarden defaults helm chart could take care of that with the helm chart, but the user would be in charge of managing this detail for their custom PolicyServer instances.

Each approach has its pros and cons. I think we should discuss about them together

[viccuad]

Good spike!
I vote also for exposing the endpoint and CA config via our helm charts.

I think the tricky part is how to propagate that to the policy-server, especially the CA to be trusted. That's because the kubewarden-controller isn't aware of the origin of the CA (what is the name and namespace of the Secret/ConfigMap).

We have:

• name, namespace of the Secret/Configmap containing the CA, or a way to mount it.
• endpoint of the collector.

I suppose we either hardcode things:

• hardcoded name, and namespace of the Secret/Configmap is the same namespace of the PolicyServer (as we are doing other secrets right now, and simplifies ServiceAccounts)
• endpoint expected in a specific Service for example

Or we pass them into the PolicyServer CRD. At this point I vote to bite the bullet and add it to the CRD, simplifies custom PolicyServer instances.

[jvanz]

This is an exhaustive card, thanks for the research work @jvanz 👏

Thanks!

Ah, sorry, I should be more precise what I think that we should do. The acceptance criteria is too generic.

I envision the user setting the otel configuration values (the endpoint address and the name of the Secret/ConfigMap holding the CA) with our helm chart. This will then be translated by the helm chart either using env variables or flags.

Yes, that's what I have in mind as well.

I think the tricky part is how to propagate that to the policy-server, especially the CA to be trusted. That's because the kubewarden-controller isn't aware of the origin of the CA (what is the name and namespace of the Secret/ConfigMap).

Another option could be to let the user configure this detail on the PolicyServer CRD. The kubewarden defaults helm chart could take care of that with the helm chart, but the user would be in charge of managing this detail for their custom PolicyServer instances.

Each approach has its pros and cons. I think we should discuss about them together

What I have in mind is that we should add another field in the CRD to allow user to mount secret with the certificate in the policy server deployment. Following the same patner we do for the environment variable.

[fabriziosestito]

Thanks for the spike!

It looks like we could help OpenTelemetry by adding the missing environment variables. I checked the codebase, and OTEL_EXPORTER_OTLP_HEADERS is already supported, so the issue seems to be a bit outdated.

[viccuad]

At this point I vote to bite the bullet and add it to the CRD, simplifies custom PolicyServer instances.

On second thought, we have PolicyServer's spec.Env, we can configure the env variables there via the Controller, without adding to the CRD, if we feel it's too early for that.

[jvanz]

@kubewarden/kubewarden-developers I've created three issue to keep track of the changes in the controller, policy server and helm chart. Can you please review it as well? Let me know if I forgot to add something from our conversation earlier Today.

[jvanz]

This epic is done and released on Kubewarden v1.20. Closing this issue