Improve Kubernetes Logs Collection Experience #25251
Comments
TylerHelmuth
added
needs triage
|
/cc @dmitryax @djaglowski |
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
There is no way to apply label selectors with filelog receiver because that information is not exposed on the file paths. The only way to do that is by fetching logs from k8s API, which adds significant load on the API and likely will significantly degrade the performance. This should not be part of filelog receiver. We have a proposal with another component for this purpose: #24439. Feel free to take a look |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
We have also ended the same situation and would like to export the logs based on the namespace level. |
|
@dmitryax @TylerHelmuth +1 for this! It would be nice if we can skip/exclude Pods' logfiles based on Pod's labels/annotations. This is super useful when a user would like to use https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/#otel_logs_exporter to collect the logs for specific instrumented apps while for the rest of the apps/Pods filelog receiver should handle the collection. Without the option to skip/exclude specific Pod's logfiles based on metadata we end up having duplicate records. |
|
@TylerHelmuth I wonder if for this we could instead use the Sth like that: receivers:
receiver_creator:
watch_observers: [ k8s_observer ]
receivers:
filelog:
rule: type == "pod" && labels["otel.logs.exporter"] == "otlp"
config:
...Note: I'm trying to make it work but without success yet but I'm posting the question already just to verify if that approach would be a no-go for any reasons. |
|
@ChrsMark I think that is an option for handling the multi-tenency solution, but I don't think it address these issues:
|
Maybe I miss sth here but
Won't a labelSelectors:
- component=kube-apiserver`? I'm just trying to understand what extra value the Let me know what you think @TylerHelmuth. I'm also cc-ing @h0cheung who works on the I'm also sharing my working example for reference: daemonset.yamlapiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: daemonset
spec:
mode: daemonset
serviceAccount:
hostNetwork: true
volumeMounts:
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
volumes:
- name: varlogpods
hostPath:
path: /var/log/pods
config: |
exporters:
debug: {}
logging: {}
otlp/elastic:
compression: none
endpoint: http://fleet-server:8200
tls:
insecure: true
insecure_skip_verify: true
extensions:
k8s_observer:
auth_type: serviceAccount
node: ${env:K8S_NODE_NAME}
observe_pods: true
health_check: {}
memory_ballast:
size_in_percentage: 40
processors:
batch: {}
resource/k8s:
attributes:
- key: service.name
from_attribute: app.label.component
action: insert
k8sattributes:
extract:
metadata:
- k8s.namespace.name
- k8s.deployment.name
- k8s.statefulset.name
- k8s.daemonset.name
- k8s.cronjob.name
- k8s.job.name
- k8s.node.name
- k8s.pod.name
- k8s.pod.uid
- k8s.pod.start_time
- container.id
labels:
- tag_name: app.label.component
key: app.kubernetes.io/component
from: pod
- tag_name: logs.exporter
key: otel.logs.exporter
from: pod
filter:
node_from_env_var: K8S_NODE_NAME
passthrough: false
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.ip
- sources:
- from: resource_attribute
name: k8s.pod.uid
- sources:
- from: connection
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
receivers:
receiver_creator:
watch_observers: [ k8s_observer ]
receivers:
filelog:
rule: type == "pod" && labels["otel.logs.exporter"] = "otlp"
config:
exclude:
- /var/log/pods/default_daemonset-opentelemetry-collector*_*/opentelemetry-collector/*.log
include:
- /var/log/pods/`namespace`_`name`*/*/*.log
include_file_name: false
include_file_path: true
operators:
- id: get-format
routes:
- expr: body matches "^{"
output: parser-docker
- expr: body matches '^[^ Z]+ '
output: parser-crio
- expr: body matches '^[^ Z]+Z'
output: parser-containerd
type: router
- id: parser-crio
regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: 2006-01-02T15:04:05.999999999Z07:00
layout_type: gotime
parse_from: attributes.time
type: regex_parser
- combine_field: attributes.log
combine_with: ""
id: crio-recombine
is_last_entry: attributes.logtag == 'F'
max_log_size: 102400
output: json_parser
source_identifier: attributes['log.file.path']
type: recombine
- id: parser-containerd
regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: regex_parser
- combine_field: attributes.log
combine_with: ""
id: containerd-recombine
is_last_entry: attributes.logtag == 'F'
max_log_size: 102400
output: json_parser
source_identifier: attributes["log.file.path"]
type: recombine
- id: parser-docker
output: json_parser
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: json_parser
- type: json_parser
if: 'body matches "^{.*}$"'
severity:
parse_from: attributes.level
start_at: end
service:
extensions:
- health_check
- k8s_observer
pipelines:
logs:
exporters:
- otlp/elastic
processors:
- k8sattributes
- batch
- resource/k8s
receivers:
- receiver_creator |
I think it would need configured multiple times as well, but it would hide all the big, long, complex fileconsumer config behind the scenes. Even with the receivercreator I believe the config in the description would need duplicated.
Oh I didn't realize it could do that, that's pretty cool. Yes I think that's equivalent. I expect there'd end up being some other k8s-specific options in a k8slogreceiver, but I'm curious if there would be any more overlap. I think obfuscating the complexity of the fileconsumer configuration is the primary benefit of the k8slogreceiver. It lets users focus on k8s-stuff an not worry about how the receiver gets the logs, trusting that it knows how to take advantage of standard k8s formatting and expectations. |
Agree @TylerHelmuth , I think a "wrapper" over the current filelog receiver which will be k8s specific hiding details would benefit the users. |
|
If in the end we're just looking for a way to gloss over the complexity of the filelog receiver, I think we should be looking to solve it with a "template". See open-telemetry/opentelemetry-collector#8372. That said, I believe #24439 was intended to add additional functionality which is not possible with filelog receiver. |
|
@djaglowski ya we're looking for both things:
It may be possible to use a combination of templates and the receiver creator to achieve this, but I'm not sure if that would come together in a solution that is simpler than a targeted receiver. |
That would be great. I guess we would need to "hide" (make it an implementation detail) the operator part that handles the At the moment the routing as well as the special handling of the logs per runtime looks weird/scary to someone that is not fully familiar with the Collector's features. Also I wonder if this functionality is somehow tested. Are there any tests that ensure that the Collector can handle In addition, any specific configuration details should be well documented. We can wait and cover all those as part of the new |
#23339 (comment) is still the right approach in my opinion. Basically a dedicated receiver which shares much of the same code but can additionally add k8s specific features as needed. If there's agreement there, we might consider consolidating this issue with #23339 |
|
I'm removing the filelog receiver label since there does not appear to be anything actionable in relation that that receiver (though there may be changes to shared packages). |
@djaglowski agreed |
|
Closing in favor of #23339. Please continue conversation over there. |
Component(s)
receiver/filelog
Describe the issue you're reporting
Problem Statement
The Collector's solution for collecting logs from Kubernetes is the Filelog Receiver and it can handle collection of Kubernetes Logs for most scenarios. But Filelog Receiver was created to be a generic solution and therefore does not take advantage of useful Kubernetes assumptions out-of-the-box.
At the moment to collector logs with the Filelog receiver the recommended configuration is:
To a new user that is a lot of scary configuration that will take time to comprehend, and they also probably don't want to comprehend it, yet it has to live in their configuration. In the Collector Helm chart we hide this complexity behind a preset, but it can't handle all situations.
Here are a couple experiences I'd like to improve:
includesection as needed.Some of the solution might be in the helm chart and some might be in the file log receiver itself. It is also possible this spawns a new k8s-specific receiver that is using stanza behind the scenes.
Ultimately, I am looking to improve the "easy path" solution for most users in Kubernetes. I want to make it easier for user to collect logs for a specific subset of all the logs in the cluster and for it to be easier to configure multiple instances of the receiver to support different destinations. Packaging up all the Kubernetes assumptions into something like:
or
would be great.
The text was updated successfully, but these errors were encountered: