Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FluentBit not filtering logs from /var/log/containers/*.log #6540

Closed
prabuprasad opened this issue Dec 13, 2022 · 3 comments
Closed

FluentBit not filtering logs from /var/log/containers/*.log #6540

prabuprasad opened this issue Dec 13, 2022 · 3 comments
Labels
Stale waiting-for-user Waiting for more information, tests or requested changes

Comments

@prabuprasad
Copy link

prabuprasad commented Dec 13, 2022

I have installed logging operator following the link https://docs.ranchermanager.rancher.io/v2.5/pages-for-subheaders/logging in my Rancher platform

Now,

I have two DaemonSet's

  1. rancher-logging-rke-aggregator
  2. rancher-logging-root-fluentbit
    One deployment
  3. rancher-logging
    One Stateful set
  4. rancher-logging-root-fluentd

And all the workloads are up and running.

Expected behavior
Daemonset's " rancher-logging-rke-aggregator , rancher-logging-root-fluentbit" has to grep the logs from container and need to push to fluentd.

But these two Daemonset's are not fetching any logs. It's starting successfully and then no output's was generated.

POD: rancher-logging-rke-aggregator-h2p2s
Fluent Bit v1.9.3

  • Copyright (C) 2015-2022 The Fluent Bit Authors
  • Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  • https://fluentbit.io
    [2022/12/13 07:13:53] [ info] [fluent bit] version=1.9.3, commit=9eb4996b7d, pid=1
    [2022/12/13 07:13:53] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
    [2022/12/13 07:13:53] [ info] [cmetrics] version=0.3.1
    [2022/12/13 07:13:53] [ info] [sp] stream processor started
    [2022/12/13 07:13:53] [ info] [output:forward:forward.0] worker #0 started
    [2022/12/13 07:13:53] [ info] [output:forward:forward.0] worker Fix include paths #1 started
    .
    .
    .

POD : rancher-logging-root-fluentbit-nfzp9
Fluent Bit v1.9.3

  • Copyright (C) 2015-2022 The Fluent Bit Authors
  • Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  • https://fluentbit.io
    [2022/12/13 07:14:07] [ info] [fluent bit] version=1.9.3, commit=9eb4996b7d, pid=1
    [2022/12/13 07:14:07] [ info] [storage] version=1.2.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128
    [2022/12/13 07:14:07] [ info] [storage] backlog input plugin: storage_backlog.1
    [2022/12/13 07:14:07] [ info] [cmetrics] version=0.3.1
    [2022/12/13 07:14:07] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
    [2022/12/13 07:14:07] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
    [2022/12/13 07:14:07] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
    [2022/12/13 07:14:07] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
    [2022/12/13 07:14:07] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
    [2022/12/13 07:14:07] [ info] [sp] stream processor started
    [2022/12/13 07:14:07] [ info] [output:forward:forward.0] worker #0 started
    [2022/12/13 07:14:07] [ info] [output:forward:forward.0] worker Fix include paths #1 started
    .
    .
    .

Environment Details
rke version : v2.6.8
K8s version : v1.21.12
Node version : REDHAT 8.6

@patrick-stephens
Copy link
Contributor

Does it work with the official helm chart? If so suggests an issue with the Rancher/Banzai operator so better to raise there.
https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit

Can you increase to log_level debug so we can see output from those files/directories? Might be permissions or dangling symlinks - if your /var/log/containers is actually a set of symlinks to log files elsewhere then you need to mount the targets as well.

I've no idea what the Rancher operator does so I think you will need to provide more details on the specific configuration of the Fluent Bit pods - what images are they using, what is their pod spec, what Fluent Bit configuration file are they using?

@patrick-stephens patrick-stephens added waiting-for-user Waiting for more information, tests or requested changes and removed status: waiting-for-triage labels Dec 15, 2022
@github-actions
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

@github-actions github-actions bot added the Stale label Mar 16, 2023
@github-actions
Copy link
Contributor

This issue was closed because it has been stalled for 5 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Stale waiting-for-user Waiting for more information, tests or requested changes
Projects
None yet
Development

No branches or pull requests

2 participants