send kube-events to grafana cloud

[OS-nikhileshdoddi]

I need to send the kube-events to Grafana cloud via fluent bit, I have usedd this page as configuration but I see the connection request is success and I don't see any logs for output.
input:
[INPUT]
Name kubernetes_events
Tag kube-events
kube_url https://kubernetes.default.svc
Interval_Sec 60
output:
[OUTPUT]
Name loki
Match kube-events.*
Log_Level info
Host ${GRAFANA_CLOUD_HOST}
Port 443
tls on
tls.verify on
http_user ${GRAFANA_CLOUD_USER}
http_passwd ${GRAFANA_CLOUD_PASSWD}
labels job=fluentbit, ring=${RING}, stamp=${STAMP}, type=event
label_map_path /fluent-bit/etc/conf/labelmap.conf
storage.total_limit_size 1G
logs:
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.0] token updated
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.1] token updated
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2024/12/10 12:38:23] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

[patrick-stephens]

Add a stdout output to confirm you are getting some.

[OS-nikhileshdoddi]

i have tried adding the stdout but no result
[OUTPUT] Name stdout Match kube-events.*
these are the complete logs
`Fluent Bit v3.1.9

• Copyright (C) 2015-2024 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

---

| | | | | | ___ () | |__ |/ |
| |_ | |_ _ ___ _ __ | |_ | |/ /| |_ __ __ / /| |
| | | | | | |/ _ \ ' | | | ___ \ | | \ \ / / \ \ | |
| | | | || | _/ | | | | | |/ / | |_ \ V /./ /| |
_| ||_,_|_|| ||_| _/||_| _/ _()__/

[2024/12/12 12:17:39] [ info] [fluent bit] version=3.1.9, commit=431fa79ae2, pid=1
[2024/12/12 12:17:39] [ info] [storage] created root path /var/data/storage
[2024/12/12 12:17:39] [ info] [storage] ver=1.5.2, type=memory+filesystem, sync=full, checksum=off, max_chunks_up=128
[2024/12/12 12:17:39] [ info] [storage] backlog input plugin: storage_backlog.6
[2024/12/12 12:17:39] [ info] [cmetrics] version=0.9.6
[2024/12/12 12:17:39] [ info] [ctraces ] version=0.5.6
[2024/12/12 12:17:39] [ info] [input:http:http.0] initializing
[2024/12/12 12:17:39] [ info] [input:http:http.0] storage_strategy='memory' (memory only)
[2024/12/12 12:17:39] [ info] [input:http:http.1] initializing
[2024/12/12 12:17:39] [ info] [input:http:http.1] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] initializing
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] multiline core started
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] db: delete unmonitored stale inodes from the database: count=0
[2024/12/12 12:17:39] [ info] [input:tail:tail.3] initializing
[2024/12/12 12:17:39] [ info] [input:tail:tail.3] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [input:tail:tail.3] multiline core started
[2024/12/12 12:17:39] [ info] [input:tail:tail.3] db: delete unmonitored stale inodes from the database: count=0
[2024/12/12 12:17:39] [ info] [input:tail:tail.4] initializing
[2024/12/12 12:17:39] [ info] [input:tail:tail.4] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [error] [input:tail:tail.4] parser 'syslog' is not registered
[2024/12/12 12:17:39] [ info] [input:tail:tail.4] db: delete unmonitored stale inodes from the database: count=0
[2024/12/12 12:17:39] [ info] [input:dummy:dummy.5] initializing
[2024/12/12 12:17:39] [ info] [input:dummy:dummy.5] storage_strategy='memory' (memory only)
[2024/12/12 12:17:39] [ info] [input:storage_backlog:storage_backlog.6] initializing
[2024/12/12 12:17:39] [ info] [input:storage_backlog:storage_backlog.6] storage_strategy='memory' (memory only)
[2024/12/12 12:17:39] [ info] [input:storage_backlog:storage_backlog.6] queue memory limit: 4.8M
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.0] token updated
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.1] token updated
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2024/12/12 12:17:39] [ info] [filter:kubernetes:kubernetes.1] connectivity OK
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.3] initializing
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.3] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.4] initializing
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.4] storage_strategy='memory' (memory only)
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.8] initializing
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.8] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.9] initializing
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.9] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.10] initializing
[2024/12/12 12:17:39] [ info] [input:emitter:emitter_for_rewrite_tag.10] storage_strategy='filesystem' (memory + filesystem)
[2024/12/12 12:17:39] [ info] [filter:grep:grep.17] AND mode
[2024/12/12 12:17:39] [ info] [output:forward:forward.0] worker #0 started
[2024/12/12 12:17:39] [ info] [output:forward:forward.0] worker #1 started
[2024/12/12 12:17:39] [ info] [output:forward:forward.1] worker #0 started
[2024/12/12 12:17:39] [ info] [output:forward:forward.1] worker #1 started
[2024/12/12 12:17:39] [ info] [output:http:http.2] worker #0 started
[2024/12/12 12:17:39] [ info] [output:http:http.2] worker #1 started
[2024/12/12 12:17:39] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/12/12 12:17:39] [ info] [sp] stream processor started
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1058040 watch_fd=1 name=/var/log/containers/coredns-576bfc4dc7-tbb9t_kube-system_coredns-7c7773cfb3d84e56ef5ae0289bb5883c313bdf3742924536fd96cb6ad7c4fc36.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1196843 watch_fd=2 name=/var/log/containers/eventrouter-c586d557-rcwr7_kube-prometheus_kube-eventrouter-60097c1cefc4cd04fca80b750eab5be89d0d747f469d1d1ef7b44e7a571c6f9e.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1075798 watch_fd=3 name=/var/log/containers/grafana-stack-9fc55dd76-l996b_grafana-system_grafana-1d869f093902ade228b81eee2d9df4047df0450cbb2061345ec717211e5b3525.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1070110 watch_fd=4 name=/var/log/containers/grafana-stack-9fc55dd76-l996b_grafana-system_grafana-sc-datasources-818d77cfe532420860e89661fd41c515b664a4846b5ffc301ac73fd32307eebd.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1075849 watch_fd=5 name=/var/log/containers/grafana-stack-loki-0_grafana-system_loki-8be5425621b722d85904730fa480bb7bc9a235ae5625bd399e277069860e9b9c.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1083828 watch_fd=6 name=/var/log/containers/grafana-stack-promtail-9lk9z_grafana-system_promtail-7da2ef2d6c77c922931ce9f6e51cd132c4b30618b06b1811643838d59c472520.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1061072 watch_fd=7 name=/var/log/containers/helm-install-cluster-fluxcd-x2gpl_flux-system_helm-3c63ec686a0b6c69c9cf2d2a0c780bf83ac6b41cd6415b22772355334b3e8c8d.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1061069 watch_fd=8 name=/var/log/containers/helm-install-grafana-stack-z4cln_grafana-system_helm-413188375acc0416ae1cc1531883fbcccd2c41b7d92368525f55c75f7fad82d6.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1193283 watch_fd=9 name=/var/log/containers/image-reflector-controller-59b9fc49b7-5gbd8_flux-system_manager-b2f806d9aa515c81baecc21a6bc8ed52e63e9f2488d7bd1e2fa8e5a0e754a09d.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1193315 watch_fd=10 name=/var/log/containers/notification-controller-86999cd88d-zpxtm_flux-system_manager-5b86534eeab0b2a3e671853fdd42522a4a003c55602b98893058a30db1d1e825.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1066733 watch_fd=11 name=/var/log/containers/svclb-grafana-stack-12b4cb13-wfrpd_kube-system_lb-tcp-3100-f5d1bcf7e0141e3919bfb57635db4800c9b0de4fa1f47606765974cd8170d8e6.log
[2024/12/12 12:17:39] [ info] [input:tail:tail.3] inotify_fs_add(): inode=1201148 watch_fd=1 name=/var/log/containers/fluent-fluent-bit-mdtxr_kube-fluentbit_fluent-bit-c8b586f839d0c694a75271536926666462c42170381b96a11b585c2a53ddb7fb.log
[2024/12/12 12:17:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1205365 watch_fd=12 name=/var/log/containers/open-telemetry-opentelemetry-collector-agent-7d7mt_kube-otel_opentelemetry-collector-d7a2c00c1801eb8362491b2735c88f841a798721a84c8d9ef8679012201d59a5.log
[2024/12/12 12:17:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1206792 watch_fd=13 name=/var/log/containers/prometheus-adapter-6ccbc87df8-dr4fj_kube-prometheus_prometheus-adapter-d86cc06f31b420d92116f163d0def8149d82c9137880f82004c95fae36a99441.log
[2024/12/12 12:17:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1206768 watch_fd=14 name=/var/log/containers/prometheus-stack-admission-create-29rwq_kube-prometheus_create-1eb5ee11c71ddde9336c7dd373d041d9162fe764f2692c8a68a96b8f33c24419.log
[2024/12/12 12:17:59] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1316741 watch_fd=15 name=/var/log/containers/prometheus-pushgateway-684686d559-5g48p_kube-prometheus_pushgateway-22b5d7fde78022bd1af23793f9e4b24ae1c92486cdfc923cd99bb70359c000b0.log
[2024/12/12 12:17:59] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1316776 watch_fd=16 name=/var/log/containers/prometheus-stack-prometheus-node-exporter-pjtb4_kube-prometheus_node-exporter-6f36e8b35ed35eb01f9cec478df357b4b6505233a6e75acfd919312618c09475.log
[2024/12/12 12:18:09] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1317116 watch_fd=17 name=/var/log/containers/prometheus-prometheus-stack-prometheus-shard-1-0_kube-prometheus_init-config-reloader-452172a0bb45f3484318b7499683775c9bf6e24790aa96d8722e28b727a0cc20.log
[2024/12/12 12:18:09] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1316995 watch_fd=18 name=/var/log/containers/prometheus-stack-kube-state-metrics-66c8f9bdc-7rvkr_kube-prometheus_kube-state-metrics-03ec00557fe413684395cccf76010c3d28ab10492601daafefae9b367dc9f888.log
[2024/12/12 12:18:09] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1316791 watch_fd=19 name=/var/log/containers/prometheus-stack-operator-c8dbcc65-wlpwp_kube-prometheus_prom-4972ea5f371c69fbe782dd1ad2b7edb61a72d01d8524bf059b582db6193d1ca7.log
[2024/12/12 12:18:19] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1317396 watch_fd=20 name=/var/log/containers/prometheus-stack-admission-patch-2zngb_kube-prometheus_patch-21632483a0aaf4c9bc372c9c568acca3ef07510284cc85a8fff55ed4af7167f4.log
[2024/12/12 12:18:30] [ info] [input:tail:tail.2] inotify_fs_remove(): inode=1206768 watch_fd=14
[2024/12/12 12:18:30] [ info] [input:tail:tail.2] inotify_fs_remove(): inode=1317396 watch_fd=20
[2024/12/12 12:18:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1320533 watch_fd=21 name=/var/log/containers/prometheus-prometheus-stack-prometheus-shard-1-0_kube-prometheus_config-reloader-59c800dc2757f3daff733129fc34039747c1605fb1ec7d2e2087909a19590f34.log
[2024/12/12 12:18:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1320496 watch_fd=22 name=/var/log/containers/prometheus-prometheus-stack-prometheus-shard-1-0_kube-prometheus_prometheus-a7741d714cac309586872866fcafb67d3f9f451dbf454444fb2c5b9d27f4b4a3.log
[2024/12/12 12:18:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1322768 watch_fd=23 name=/var/log/containers/prometheus-prometheus-stack-prometheus-shard-1-0_kube-prometheus_thanos-sidecar-baeb2a3ae8c468457021864cc79918da67699406de7626be6a467346736d4527.log
[2024/12/12 12:18:49] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1320582 watch_fd=24 name=/var/log/containers/thanos-query-frontend-69f88c488c-r892c_kube-prometheus_query-frontend-18754d347bd99d4b8f4e99ed91798c6ed64815a4c964ee4707d8bc861956a7a4.log
[2024/12/12 12:19:09] [ info] [input:tail:tail.2] inotify_fs_add(): inode=1337935 watch_fd=25 name=/var/log/containers/open-telemetry-collector-opentelemetry-collector-8dffb9d6999skb_kube-otel_opentelemetry-collector-b88a4ff93d7431f9074049a83d7aec48f07b7e2b5047cba323b423374717e398.log`

[patrick-stephens]

Sounds like no events are coming through, I'd check the RBAC too or any errors on the api server side.

[patrick-stephens]

Use match * for everything too.

[patrick-stephens]

Ah no I see the issue, your tag is not what you are matching.

Your tag is kube-events, however you are saying I want only records that match kube-events.*, i.e. an extra dot. It's a typo in your config.

[OS-nikhileshdoddi]

so you mean to say match should you like this ?
Match kube-events*

[OS-nikhileshdoddi]

but as far I see i don't see events under policy rules under the service-acccount
ikhilesh.doddi@OS-NPF944WDTP flux-stamp-config-monitoring-stack % kubectl describe clusterrole monitoring-team-reconciler-fluentbit

Name: monitoring-team-reconciler-fluentbit
Labels: kustomize.toolkit.fluxcd.io/name=monitoring-stack
kustomize.toolkit.fluxcd.io/namespace=flux-system
toolkit.fluxcd.io/tenant=monitoring-team
Annotations:
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs

---

clusterrolebindings.rbac.authorization.k8s.io [] [] [get list watch create patch update delete]
clusterroles.rbac.authorization.k8s.io [] [] [get list watch create patch update delete]
rolebindings.rbac.authorization.k8s.io [] [] [get list watch create patch update delete]
namespaces [] [] [get list watch]
nodes/proxy [] [] [get list watch]
nodes [] [] [get list watch]
pods [] [] [get list watch]

[patrick-stephens]

Yeah so two things:

1. Your match is wrong
2. stdout should show the events, if it doesn't then check rbac or similar as to why

[OS-nikhileshdoddi]

input:
[INPUT]
Name kubernetes_events
Tag kube-events
kube_url https://kubernetes.default.svc
Interval_Sec 60
output:
[OUTPUT]
Name stdout
Match kube-events*
is this the matching part correct ?
\

[patrick-stephens]

sure or just use match *