Floating point exception after upgrading to flunetBit v1.8.3

[hegderohit89]

Bug Report

Few of our customers are observing frequent crashes and restarts (675 times) with fluentBit after upgrading to v1.8.3 from v1.7.4
There was no configuration change/update, except for changing the version to v1.8.3.
This was observed immediately after the upgrade and the older version was healthy and performant for months.

It fails with this message 7 Floating point exception(core dumped)

Here is the debug log:

Fluent Bit v1.8.3
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io[2021/10/04 17:23:16] [ info] [engine] started (pid=7)
[2021/10/04 17:23:16] [ info] [storage] version=1.1.1, initializing...
[2021/10/04 17:23:16] [ info] [storage] in-memory
[2021/10/04 17:23:16] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/04 17:23:16] [ info] [cmetrics] version=0.1.6
[2021/10/04 17:23:16] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2021/10/04 17:23:16] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2021/10/04 17:23:16] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2021/10/04 17:23:16] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2021/10/04 17:23:16] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/10/04 17:23:16] [ info] [sp] stream processor started
[2021/10/04 17:23:16] [ info] [input:tail:tail.0] inotify_fs_add(): inode=543518586 watch_fd=1 name=/var/log/containers/
...
...
influxdb-5f65dbc48d-db9ps_monitoring_influxdb-53965187969cca895e09c67b3d8e418c4badc84291e09d83aadb6648f1e021b9.log
[2021/10/04 17:25:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=272356016 watch_fd=29 name=/var/log/containers/cluster-status-1633368300-45b8z_rtf_job-0354b2b704ae0b36041b1ee720a9d8bf9ce52be3621396a0ab65992ae192d8d6.log
/docker-entrypoint.sh: line 19: 7 Floating point exception(core dumped) /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf

Your Environment

• Version used: v1.8.3
• Configuration:

[SERVICE]
    Flush         5
    Config_Watch  On
    Log_Level     info
    Daemon        Off
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_PORT     2020
    Parsers_File  parsers.conf

@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE config/*

[INPUT]
    Name               tail
    Tag                kube.*
    Path               /var/log/containers/*.log
    Parser             docker
    DB                 ${DB_FILE_PATH}
    Mem_Buf_Limit      5MB
    Skip_Long_Lines    On
    Refresh_Interval   10
    Rotate_Wait        10 
    Docker_Mode        On
    Docker_Mode_Flush  5
    Docker_Mode_Parser test

• Environment name and version : Kubernetes v13
• Server type and version:
• Operating System and version:
• Filters and plugins: Kubernetes filter and connecting to ELK

[edsiper]

Please attach full filters configuration and let us know if the issue is reproducible or not with v1.8.7

[nokute78]

@hegderohit89 Are you using out_es ?
It may cause SIGFPE #3905
It was fixed from v1.8.4.

[hegderohit89]

@edsiper Here is the filter:

[FILTER]
		Name                kubernetes
		Match               kube.*
		Kube_URL            ${KUBERNETES_SVC_URL}
		Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
		Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
		Kube_Tag_Prefix     kube.var.log.containers.
		K8S-Logging.Parser  On
		K8S-Logging.Exclude On
		Annotations         Off
		Merge_Log           On
		Labels              On

Is there a way to get the coredump of fluentBit process? (other than the "kill -CONT pidof fluent-bit" command)?

@nokute78 : No, the upstream server is ELK

[hegderohit89]

@edsiper any update on this ?

[edsiper]

@hegderohit89 do you have simple steps to reproduce the issue ?

[hegderohit89]

@hegderohit89 do you have simple steps to reproduce the issue ?

Unfortunately not, this was reported by couple of our customers and we are trying to reproduce this in house. I will share if we find anything.
Thats where i was trying to see where exactly does the fBit process writes the core dump file ?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[hegderohit89]

Are these issues fixed in the version 1.8.11 ?