Logging OTLP support

[a-thaler]

Motivation
The telemetry module was initially designed to be based fully on the OpenTelemetry project. As the logs domain of the project was not stable yet and also the adoption is simply not there yet, the module was released having the application logs based on Fluentbit using the proprietary HTTP output. The SAP Cloud Logging backend was not supporting OTLP as well.

These criterias changed and the adoption slowly kicks-in:

• On the backends:
  • SAP Cloud Logging supports OTLP based ingestion with much better performance and much better correlation capabilities with traces and metrics.
  • Most 3party service ofeerings are supporting it like dynatrace, signalFX or honeycomb
  • More and more backends are supporting it, also Loki is in an active implementation for adding support.
• On the clients:
  • The otel-sdk is getting more and more used for instrumentation of traces and metrics and supports logs as well, so people will start trying it out.
  • Istio access logging is supporting OTLP, avoiding the mix up of the access logs with the envoy application logs via stdout.
  • SAP CAP starts using the otel-sdk

The client adoption brings the benefit of streamlining the setup by introducing a gateway like done for traces and metrics already, allowing clients to directly push logs without the indirection via stdout, enabling more opportunities like enabling easy collection of kubernetes event logs,

Making logs based on OTLP will allow a streamlined approach to telemetry data across traces and metrics. All data can have streamlined attributes following streamlined semantics.

Also the proprietary protocol used with fluentbit is strongly aligned with the SAP Cloud Logging API and is not common for ingestion to other systems, so the usage is very limited.

Fluentbit as technology is different to the otel-collector framework and not that flexible. A lot of synergies can be seen in the telemetry-manager between the traces and metrics domain and could be applied to logging as well, simplifying the code base and maintenance.

Goal and requirements

Goal is to make logs based on the OTLP protocol (for backend ingestion to support more providers, for client ingestion to avoid indirection via stdout if not desired), leveraging the otel-collector framework to have a streamlined technology stack, still supporting collection of logs via stdout.

The requirements can be split into three parts:

• backend OTLP ingestion support
  • support of OTLP outputs similar to traces and metrics
  • data will get enriched in OTLP fashion following same enrichment logic as already done for traces and metrics, covering mainly the cluster name, the k8s resource attributes and the service name
• tail based log collection enabled by dedicated input
  • similar to metrics input, have a way to enable log collection via runtime log tailing
  • support namespace filtering
  • try a meaningful mapping of log entries to the OTLP protocol in case of JSON logs
  • optional support dedicated mapping instructions per workload
• client OTLP support (not needed for feature parity)
  • introduce an endpoint to push logs in OTLP protocol similar to metrics/traces

The new API must be available in parallel to the old API which will get deprecated but will stay till there is no usage anymore.
With kyma-project/kyma#15932 fluentbit introduced support for OTLP and a first iteration might introduce the OTLP output based on fluenbit. However, first investigations revealed that fluentbit does neglect OTLP resource attributes, especially the enrichment of relevant k8s metadata will be not possible. With that a direct jump to the otel-collector seems to be needed (which is desired anyway).

Targetted Architecture

Actions

• Concept phase
  • Build up knowledge of the OTLP log semantics and tooling and come up with a mapping concept when transforming JSON logs to OTLP (OTLP Log Mapping Concept #691)
  • Prove the derived mapping concept (PoC for Log parsing with Otel Collector #720)
  • Prove that shipping tailed logs via the gateway can be reliable enough, performant enough, scalable enough (PoC for log tailing shipment via log gateway #769)
  • Understand and define access log integration based on Istio OTLP support (Understand and define access log integration based on Istio OTLP support #821)
• Implementation phase
  • Do performance tests on the future gateway setup to understand if auto-scaling will be a mandatory requirement to run bigger cluster setups ([OTLP Log] Gateway performance evaluation #1289)
  • Add a feature flag and have a modified LogPipeline CRD which gets active on experimental channel (Prepare LogPipeline reconciler for OTLP output #1415)
  • Add a new OTLP output which reveals into a log gateway deployment (Have a first OTLP output available in the development release of a LogPipeline which activates a Log gateway #872)
  • Come up with a concrete log agent config and have another perf test (Validate new OTLP log setup #1631)
  • OTLP Logs: Deploy a log agent which tails logs and maps them to OTLP #1683
  • Assure that namespace filtering works proper in a multi-pipeline setup
  • Add parser and multiline parser support
  • Assure to set timestamp to the log timestamp with fallback to runtime timestamp
  • Assure to always set the observedTimestamp if not set yet
  • Add self-monitoring capabilities
  • Add k8s label support (and maybe deprecate annotation support)
  • Add configurable persistent buffer support
  • Add new end-user documentation and deprecate the old one (but keep it)
  • Deprecate fluentbit API
• Follow-Up
  • New input for k8s events ([Telemetry Operator] Concept on how to enable kubelet logs as input #560)
  • Istio access logs via OTLP and related noise filter

[a-thaler]

The testing of the OTLP output revealed serious problems. Some got fixed with 2.0.9 but still problems are present.

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale