frequent crash on tail plugin with multiline on and file rotation #255
Comments
|
Upstream issue: fluent/fluent-bit#4177 |
|
I tried using valgrind, and found that msgpack_sbuffer_destroy was called twice: |
|
@wubn2000 please post your valgrind output logs |
|
@wubn2000 please also attach your config files and example log files as separate files in the issue, or please use markdown formatting to make them clear like: Otherwise it's just a mess of text which is hard to read. Also I think I copied your config wrong, since I am getting: |
|
[input:tail:tail.0] purge: monitored file has been deleted: /data/testdata/v000/stats-0-10-11-21-21-24-03.csv (deleted) |
|
parser.example.conf conf.conf inputs.example.conf |
|
one sample line of message (it has multiline fields, but to crash the fluentbit, you can use single line here): |
|
@wubn2000 Thanks, I was able to reproduce it with a simple script: log.txt is just a file with the log lines in your comment above. Valgrind: Looks like its double free-ing the msgpack in some case. |
|
@hossain-rayhan Can you own fixing this bug? |
|
I was able to reproduce similar behavior using Wesley's script and @wubn2000's config file. I will take a look for details. |
|
FYI: #100 (comment) |
|
This has been fixed in: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.21.1 |
Describe the bug
When turn on the multiline parser, ( the one I used is
[PARSER]
Name multiline_stats
Format regex
Regex /^(?\d{10,}.*)/
), if monitoring on a rotate log file, it always crashed due to the file rotation (after the rotated file got deleted), with info like:
*** Error in `./fluent-bit': double free or corruption (out): 0x00007f66c2f65e80 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f66fea7e329]
./fluent-bit[0x4b73ea]
./fluent-bit(flb_tail_file_remove+0x13e)[0x4ba469]
./fluent-bit(flb_tail_file_purge+0x293)[0x4bc068]
./fluent-bit(flb_input_collector_fd+0x3b2)[0x4614bd]
./fluent-bit(flb_engine_start+0x707)[0x472660]
./fluent-bit[0x456cf5]
Without turning on multiline parser, it won't crash. But with it on, it always crash during the file deletion.
To Reproduce
Rubular link if applicable:
Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
Note here is one message, with the last field as a multiline field:
1633801021538,"customer2788",13,2788,1623239523,"java.lang.RuntimeException: Something has gone wrong, aborting!
at com.myproject.module.MyProject.badMethod(MyProject.java:22)
at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
at com.myproject.module.MyProject.someMethod(MyProject.java:10)
at com.myproject.module.MyProject.main(MyProject.java:6)","Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam. Ut enim ad minim ven."
Steps to reproduce the problem:
FLuentbit is monitoring only stats-0.csv. I keep appending this message to a file stats-0.csv, when the size reaches a threshold (can be 500k), do file rotation by "mv stats-0.csv stats-0-timestamp1.csv", then empty stats-0.csv and keep appending the message again. After a while, some oldest stats-0-timestamp1.csv will be deleted from the directory. The crash always happens if the old file stats-0-timestamp1.csv got deleted.
Expected behavior
The fluentbit should not crash.
Screenshots
Your Environment
Version used: 1.8.8
Configuration:
[INPUT]
Name tail
Path /data/test/stats-0.csv
Path_Key file_name
Tag stats
Buffer_Chunk_Size 32KB
Buffer_Max_Size 128KB
Skip_Long_Lines On
Multiline On
Parser_Firstline multiline_stats
Multiline_Flush 5
Rotate_Wait 35
Refresh_Interval 1
Read_from_head true
Mem_Buf_Limit 500MB
storage.type filesystem
DB /data/tail-0.db
DB.locking true
[SERVICE]
Flush 1
Daemon off
Log_Level info
Parsers_File parser.example.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
storage.path /data/storage
storage.backlog.mem_limit 500M
storage.max_chunks_up 200
@include inputs.example.conf
[OUTPUT]
Name stdout
Environment name and version (e.g. Kubernetes? What version?): ec2 machine
Server type and version:
Operating System and version:
Filters and plugins:
Additional context
The text was updated successfully, but these errors were encountered: