Tail input with multi-line and exit_on_eof broken in 1.6.x

[LionelCons]

The newly released 1.6.0 does not work like before when using multi-line and exit_on_eof.

Consider the following input:

line 1                                                                                                                                                        
line                                                                                                                                                          
  2                                                                                                                                                           
line 3                                                                                                                                                        

And the following configuration:

[PARSER]                                                                                                                                                      
    Name         test1                                                                                                                                        
    Format       regex                                                                                                                                        
    Regex        ^(?<tag>\w+)                                                                                                                                 
[INPUT]                                                                                                                                                       
    Name              tail                                                                                                                                    
    Path              test.log                                                                                                                                
    Multiline         On                                                                                                                                      
    Multiline_Flush   1                                                                                                                                       
    Parser_Firstline  test1                                                                                                                                   
    Tag               test                                                                                                                                    
    Exit_On_EOF       On                                                                                                                                      
    Read_From_Head    On                                                                                                                                     

When using 1.6.0 (and Read_From_Head to have the same behaviour as before) we get:

[2020/10/12 08:12:34] [ info] [input:tail:tail.0] inotify_fs_add(): inode=67815790 watch_fd=1 name=test.log
{"date":1602483154.629946,"tag":"line"}
{"date":1602483154.629992,"tag":"line\n  2"}
[2020/10/12 08:12:34] [ warn] [engine] service will stop in 5 seconds
[2020/10/12 08:12:38] [ info] [engine] service stopped
[2020/10/12 08:12:38] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=67815790 watch_fd=1

So we miss the third line.

Without Exit_On_EOF, we do get the third line.

In contrast, with 1.5.6, with Exit_On_EOF, we get:

[2020/10/12 08:14:26] [ info] inotify_fs_add(): inode=67815790 watch_fd=1 name=test.log
{"date":1602483266.717917,"tag":"line"}
{"date":1602483266.717965,"tag":"line\n  2"}
[2020/10/12 08:14:26] [ warn] [engine] service will stop in 5 seconds
[2020/10/12 08:14:26] [ info] [input] tail.0 resume (mem buf overlimit)
{"date":1602483266.717981,"tag":"line"}
[2020/10/12 08:14:30] [ info] [engine] service stopped
[2020/10/12 08:14:30] [ info] [input] pausing tail.0
[2020/10/12 08:14:30] [ info] inotify_fs_remove(): inode=67815790 watch_fd=1

So we do get the third line.

[adisong]

I also have problems with Read_From_Head it seems that if log file has only one line it is not read at all. /var/log/sample1.log has 2 lines and /var/log/sample2.log has only one and there is no output from sample2.log at all.

Those two lines that are read are also reported as one event I have Format json in [OUTPUT].

[OUTPUT]
  Name stdout
  Match *
  Format json
[INPUT]
  Name tail
  Read_From_Head On
  Path /var/log/*
  Refresh_Interval 5
  Tag test

[2020/10/12 11:45:11] [debug] [input:tail:tail.1] inode=33554523 file=/var/log/sample1.log promote to TAIL_EVENT
[2020/10/12 11:45:11] [ info] [input:tail:tail.1] inotify_fs_add(): inode=33554523 watch_fd=1 name=/var/log/sample1.log
[2020/10/12 11:45:11] [debug] [input:tail:tail.1] inode=100668982 file=/var/log/sample2.log promote to TAIL_EVENT
[2020/10/12 11:45:11] [ info] [input:tail:tail.1] inotify_fs_add(): inode=100668982 watch_fd=2 name=/var/log/sample2.log
[{"date":1602503111.279498,"log":"This is a sample1.log."},{"date":1602503111.279511,"log":"Log line 2 of sample1.log"}]

[LionelCons]

I've tried again with 1.6.2: the problem is the same...

[LionelCons]

I've tried again with 1.6.4: the problem is the same...

[edsiper]

@LionelCons

the following fixes have been applied

• f26cea5 in_tail: dockermode: flush pending data on static files at exit (Tail input with multi-line and exit_on_eof broken in 1.6.x #2668)
• 8d78417 in_tail: multiline: flush pending data on static files at exit (Tail input with multi-line and exit_on_eof broken in 1.6.x #2668)

the main problem was that on hard exit (exit on eof) the pending buffers of in_tail were not flushed

[edsiper]

@adisong

I tried to reproduce your problem but was not possible, @LionelCons issue was different.

the following is my config:

[SERVICE]
  flush            1
  log_level        info

[INPUT]
  name             tail
  tag              test
  path             *.log
  read_from_head   on
  refresh_interval 5
  
[OUTPUT]
  name             stdout
  Match            *
  format           json_lines

file sample_1.log content

a
b

file sample_2.log content

c

Fluent Bit Output

Fluent Bit v1.7.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/11/23 17:46:11] [ info] [engine] started (pid=956445)
[2020/11/23 17:46:11] [ info] [storage] version=1.0.6, initializing...
[2020/11/23 17:46:11] [ info] [storage] in-memory
[2020/11/23 17:46:11] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/11/23 17:46:11] [ info] [sp] stream processor started
[2020/11/23 17:46:11] [ info] [input:tail:tail.0] inotify_fs_add(): inode=17044050 watch_fd=1 name=sample_1.log
[2020/11/23 17:46:11] [ info] [input:tail:tail.0] inotify_fs_add(): inode=17044051 watch_fd=2 name=sample_2.log
{"date":1606175171.04159,"log":"a"}
{"date":1606175171.041602,"log":"b"}
{"date":1606175171.041706,"log":"c"}

if you are still able to reproduce the issue, please open a new Github issue and reference this one so we can troubleshoot.

I am closing this ticket since the primary issue is fixed.