FluentBit stops working few minutes after startup

[themahanto]

Bug Report

Describe the bug
[seen in 1.7.6 version also]
We are using fluent bit version 1.0.5 on a centos7 VM, where after fluent-bit service startup, only few log chunks are sent at the beginning.
After few minutes no logs are getting transferred to output server.
I have tested with 1.7.6 version, the issue exists in this version also.

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

[2021/05/25 21:03:38] [ info] [storage] created root path /var/log/flb-storage/
[2021/05/25 21:03:38] [ info] [storage] initializing...
[2021/05/25 21:03:38] [ info] [storage] root path '/var/log/flb-storage/'
[2021/05/25 21:03:38] [ info] [storage] normal synchronization mode, checksum disabled
[2021/05/25 21:03:38] [ info] [storage] backlog input plugin: storage_backlog.0
[2021/05/25 21:03:38] [ info] [engine] started (pid=1696)
[2021/05/25 21:03:46] [ info] [storage backlog] queue memory limit: 47.7M

• Steps to reproduce the problem:

Expected behavior
FluentBit should continue working.

Screenshots

Your Environment

• Version used: 1.0.5 and 1.7.6
• Configuration:

[SERVICE]
    # Flush
    # =====
    # Set an interval of seconds before to flush records to a destination
    Flush        5
    storage.path              /var/log/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.backlog.mem_limit 50M

    # Daemon
    # ======
    # Instruct Fluent Bit to run in foreground or background mode.
    Daemon       Off

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # By default 'info' is set, that means it includes 'error' and 'warning'.
    Log_Level    info
    Log_File    /var/log/fluentbit.log

    # Parsers_File
    # ============
    # Specify an optional 'Parsers' configuration file
    Parsers_File sdlaParsers.conf
    Plugins_File sdlaPlugins.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name tail
    Path /var/log/*.log
    Path_Key class
    DB /var/log/fluentbit.db
    Exclude_Path /var/log/*.db
    Multiline         On
    Parser_Firstline  multiline_pattern
    Buffer_Chunk_Size 128k
    Buffer_max_size 256k
    Skip_Long_Lines On


[INPUT]
    Name tail
    Path /var/log/**/*.log
    Path_Key class
    DB /var/log/fluentbit.db 
    Exclude_Path /var/log/*.db
    Multiline         On
    Parser_Firstline  multiline_pattern
    Buffer_Chunk_Size 128k
    Buffer_max_size 256k
    Skip_Long_Lines On


[INPUT]
    Name tail
    Path /var/log/**/**/*.log
    Path_Key class
    DB /var/log/fluentbit.db
    Exclude_Path /var/log/*.db
    Multiline         On
    Parser_Firstline  multiline_pattern
    Buffer_Chunk_Size 128k
    Buffer_max_size 256k
    Skip_Long_Lines On


[FILTER]
    Name record_modifier
    Match *
    Record vnf_id ${A_NAME}
    Record instance_id ${B_NAME}
    Record component_name ${C_TYPE}
    Record app_name ${D_TYPE}

[OUTPUT]
    Name  gstdout
    Match *

• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version: Centos7 deployed on openstack
• Operating System and version: Centos7
• Filters and plugins: Tail

Additional context

[themahanto]

#3518 (comment)
seems like a similar issue, also exists in different environment.

[themahanto]

Adding @pierluigilenoci @edsiper for information.

[themahanto]

Additional issue observed.
#3565

[dewaldmienie]

@anuragmahanto, I have a similar issue as specified here https://stackoverflow.com/questions/67695229/fluent-bit-tail-input-only-reads-the-first-few-lines-per-log-file-until-it-is-re

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

@edsiper could you please take a look?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

@edsiper could you please take a look?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[pierluigilenoci]

@edsiper @leonardo-albertovich @nokute78 could you please take a look?

[nokute78]

How many number of elements do you send ?
If the elements of map is greater than 128, filter_record_modifier will read invalid memory.

Note : Similar issue #3968, #4013. I sent a patch #3982 (is not merged yet)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.