[1.7.x] Fluent-bit crashes with multiple inputs/outputs (SIGSEGV error) on high load #3412

jevgenimarenkov · 2021-04-23T19:57:55Z

Bug Report

Describe the bug

To Reproduce

Steps to reproduce the problem:
Generate backpressure by writing 5000 log lines per second to the file. I was using BQ output plugin though not related to the output plugin. The problem is not reproducible with low volumes of logs.

2021-04-23_19:52:07.32103 [2021/04/23 12:52:07] [engine] caught signal (SIGSEGV)
2021-04-23_19:52:07.35854 #0  0x7f7d6fcd10ce      in  ???() at s/x86_64/multiarch/memcpy-ssse3-back.S:1348
2021-04-23_19:52:07.35866 #1  0x710af0            in  cio_file_write() at lib/chunkio/src/cio_file.c:824
2021-04-23_19:52:07.35872 #2  0x70d325            in  cio_chunk_write() at lib/chunkio/src/cio_chunk.c:173
2021-04-23_19:52:07.35887 #3  0x474fe0            in  flb_input_chunk_write() at src/flb_input_chunk.c:90
2021-04-23_19:52:07.35888 #4  0x476759            in  flb_input_chunk_append_raw() at src/flb_input_chunk.c:892
2021-04-23_19:52:07.35911 #5  0x48f1ff            in  process_content() at plugins/in_tail/tail_file.c:358
2021-04-23_19:52:07.35911 #6  0x490e93            in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:981
2021-04-23_19:52:07.35920 #7  0x48b6fd            in  in_tail_collect_event() at plugins/in_tail/tail.c:261
2021-04-23_19:52:07.35929 #8  0x495f7f            in  tail_fs_event() at plugins/in_tail/tail_fs_inotify.c:268
2021-04-23_19:52:07.35944 #9  0x44a2d4            in  flb_input_collector_fd() at src/flb_input.c:1004
2021-04-23_19:52:07.35957 #10 0x459c25            in  flb_engine_handle_event() at src/flb_engine.c:352
2021-04-23_19:52:07.35957 #11 0x459c25            in  flb_engine_start() at src/flb_engine.c:613
2021-04-23_19:52:07.35967 #12 0x43fe8a            in  flb_lib_worker() at src/flb_lib.c:493
2021-04-23_19:52:07.35988 #13 0x7f7d70874e74      in  start_thread() at debug/glibc-2.17-c758a686/nptl/pthread_create.c:307
2021-04-23_19:52:07.35989 #14 0x7f7d6fc798fc      in  ???() at s/unix/sysv/linux/x86_64/clone.S:111
2021-04-23_19:52:07.35989 #15 0xffffffffffffffff  in  ???() at ???:0
2021-04-23_19:52:07.52994 run returned -1 6

Expected behavior
Fluentbit does not crash

Screenshots

Your Environment

Version used: 1.7.4
Configuration:
Environment name and version (e.g. Kubernetes? What version?): Amazon Linux
Server type and version:
Operating System and version:
Filters and plugins:

Additional context
The problem is reproducible only in 1.7.* Version 1.6.* does not have that problem.

Configuration

[SERVICE]
    # Flush
    # =====
    # Set an interval of seconds before to flush records to a destination
    Flush                           5

    # Storage (buffers)
    # ======
    storage.path                    /etc/fluent-bit/buffers
    storage.metrics                 On

    # Daemon
    # ======
    # Instruct Fluent Bit to run in foreground or background (Daemon) mode.
    Daemon                          Off

    Log_File                        /mnt/logs/comp/comp-fluent-bit.log

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    # [ error | warning | info | debug | trace ]
    #
    # By default 'info' is set, that means it includes 'error' and 'warning'.
    Log_Level                       info

    # Parsers_File
    # ============
    # Specify an optional 'Parsers' configuration file
    Parsers_File                    config/parsers.conf
    Parsers_File                    config/parsers-static.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    HTTP_Server                     On
    HTTP_Listen                     0.0.0.0
    HTTP_Port                       2020

 [INPUT]
     Name                            tail
     Tag                             service.tag5.test
     Path                            path5
     Parser                          parser5
     DB                              /etc/fluent-bit/flb.db
     storage.type                    filesystem
     Read_from_Head                  On

 [INPUT]
     Name                            tail
     Tag                             service.tag4.test
     Path                            path4
     Parser                          parser4
     DB                              /etc/fluent-bit/flb.db
     storage.type                    filesystem
     Read_from_Head                  On

 [INPUT]
     Name                            tail
     Tag                             service.tag3.test
     Path                            path3
     Parser                          parser3
     DB                              /etc/fluent-bit/flb.db
     storage.type                    filesystem
     Read_from_Head                  On

[INPUT]
    Name                            tail
    Alias                           tail_service.haproxy
    Tag                             service.tag2.test
    Path                            /path2
    Parser                          parser2
    DB                              /etc/fluent-bit/flb.db
    storage.type                    filesystem
    Read_from_Head                  On

[INPUT]
    Name                            tail
    Tag                             service.tag1.test
    Path                            /path1
    Parser                          parser1
    DB                              /etc/fluent-bit/flb.db
    storage.type                    filesystem
    Read_from_Head                  On

[OUTPUT]
    Name                            bigquery
    Alias                           bigquery_service.nginx.access_json
    Match                           service.tag1.json
    google_service_credentials      /secrets/secret.json
    project_id                      some_project_id
    dataset_id                      some_dataset_id
    table_id                        some_table_id

[OUTPUT]
    Name                            bigquery
    Alias                           bigquery_service.nginx.access_json
    Match                           service.tag5.json
    google_service_credentials      /secrets/secret.json
    project_id                      some_project_id
    dataset_id                      some_dataset_id
    table_id                        some_table_id

 [OUTPUT]
     Name                            bigquery
     Match                           service.tag4.json
     google_service_credentials      /secrets/secret.json
     project_id                      some_project_id
     dataset_id                      some_dataset_id
     table_id                        some_table_id

 [OUTPUT]
     Name                            bigquery
     Match                           service.tag3.json
     google_service_credentials      /secrets/secret.json
     project_id                      some_project_id
     dataset_id                      some_dataset_id
     table_id                        some_table_id

 [OUTPUT]
     Name                            bigquery
     Match                           service.tag2.json
     google_service_credentials      /secrets/secret.json
     project_id                      some_project_id
     dataset_id                      some_dataset_id
     table_id                        some_table_id

The text was updated successfully, but these errors were encountered:

agup006 · 2021-04-23T21:13:10Z

@jevgenimarenkov would you be able to add your configuration?

jevgenimarenkov · 2021-04-24T11:53:17Z

@agup006 configuration added

jevgenimarenkov · 2021-04-24T11:55:02Z

I detected the pattern that the interval of error occurrence depends on the number of input/output. With one input and output, the problem occurs every 30 minutes with a high load but with 5 - 6 inputs/outputs, the problem occurs every 3-4 minutes.

jevgenimarenkov · 2021-04-24T11:56:30Z

I was also running the same tests with 1.6 and they pass successfully but with 1.7.* fluent bit crashed all the time.

jevgenimarenkov · 2021-04-28T06:19:49Z

Reproduced the same problem also in master.

jevgenimarenkov · 2021-04-28T19:56:09Z

Added AddressSanitizer report

Fluent Bit v1.7.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
=================================================================
==4297==ERROR: AddressSanitizer: heap-use-after-free on address 0x7efe030965d3 at pc 0x7efe0710aaf0 bp 0x7efe02afe320 sp 0x7efe02afdad0
WRITE of size 33384 at 0x7efe030965d3 thread T1 (flb-pipeline)
    #0 0x7efe0710aaef in __interceptor_memcpy /jenkins/gcc-9.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790
    #1 0x8a9424 in cio_file_write /home/root/fluent-bit/lib/chunkio/src/cio_file.c:817
    #2 0x8a5f33 in cio_chunk_write /home/root/fluent-bit/lib/chunkio/src/cio_chunk.c:164
    #3 0x4c6b2b in flb_input_chunk_write /home/root/fluent-bit/src/flb_input_chunk.c:67
    #4 0x4c9d47 in flb_input_chunk_append_raw /home/root/fluent-bit/src/flb_input_chunk.c:815
    #5 0x50ced1 in process_content /home/root/fluent-bit/plugins/in_tail/tail_file.c:358
    #6 0x511dff in flb_tail_file_chunk /home/root/fluent-bit/plugins/in_tail/tail_file.c:981
    #7 0x501497 in in_tail_collect_pending /home/root/fluent-bit/plugins/in_tail/tail.c:91
    #8 0x4619d9 in flb_input_collector_fd /home/root/fluent-bit/src/flb_input.c:1004
    #9 0x482f04 in flb_engine_handle_event /home/root/fluent-bit/src/flb_engine.c:352
    #10 0x482f04 in flb_engine_start /home/root/fluent-bit/src/flb_engine.c:613
    #11 0x446d4b in flb_lib_worker /home/root/fluent-bit/src/flb_lib.c:493
    #12 0x7efe06e5fe74 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307
    #13 0x7efe062638fc in clone (/lib64/libc.so.6+0xfe8fc)
0x7efe030965d3 is located 44499 bytes inside of 131072-byte region [0x7efe0308b800,0x7efe030ab800)
freed by thread T1 (flb-pipeline) here:
    #0 0x7efe0717ab57 in __interceptor_free /jenkins/gcc-9.2.0/libsanitizer/asan/asan_malloc_linux.cc:122
    #1 0x461ec0 in flb_free /home/root/fluent-bit/include/fluent-bit/flb_mem.h:122
    #2 0x462a5d in flb_filter_do /home/root/fluent-bit/src/flb_filter.c:183
    #3 0x4c9f25 in flb_input_chunk_append_raw /home/root/fluent-bit/src/flb_input_chunk.c:832
    #4 0x50ced1 in process_content /home/root/fluent-bit/plugins/in_tail/tail_file.c:358
    #5 0x511dff in flb_tail_file_chunk /home/root/fluent-bit/plugins/in_tail/tail_file.c:981
    #6 0x501777 in in_tail_collect_static /home/root/fluent-bit/plugins/in_tail/tail.c:140
    #7 0x4619d9 in flb_input_collector_fd /home/root/fluent-bit/src/flb_input.c:1004
    #8 0x482f04 in flb_engine_handle_event /home/root/fluent-bit/src/flb_engine.c:352
    #9 0x482f04 in flb_engine_start /home/root/fluent-bit/src/flb_engine.c:613
    #10 0x446d4b in flb_lib_worker /home/root/fluent-bit/src/flb_lib.c:493
    #11 0x7efe06e5fe74 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307
previously allocated by thread T1 (flb-pipeline) here:
    #0 0x7efe0717b30e in __interceptor_realloc /jenkins/gcc-9.2.0/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x66b798 in msgpack_sbuffer_write /home/root/fluent-bit/lib/msgpack-c/include/msgpack/sbuffer.h:81
    #2 0x8ac11d in msgpack_pack_str /home/root/fluent-bit/lib/msgpack-c/include/msgpack/pack_template.h:766
    #3 0x8ac652 in msgpack_pack_object /home/root/fluent-bit/lib/msgpack-c/src/objectc.c:66
    #4 0x8ac809 in msgpack_pack_object /home/root/fluent-bit/lib/msgpack-c/src/objectc.c:113
    #5 0x67589e in apply_modifying_rules /home/root/fluent-bit/plugins/filter_modify/modify.c:1225
    #6 0x675ce9 in cb_modify_filter /home/root/fluent-bit/plugins/filter_modify/modify.c:1297
    #7 0x4625b2 in flb_filter_do /home/root/fluent-bit/src/flb_filter.c:118
    #8 0x4c9f25 in flb_input_chunk_append_raw /home/root/fluent-bit/src/flb_input_chunk.c:832
    #9 0x50ced1 in process_content /home/root/fluent-bit/plugins/in_tail/tail_file.c:358
    #10 0x511dff in flb_tail_file_chunk /home/root/fluent-bit/plugins/in_tail/tail_file.c:981
    #11 0x501777 in in_tail_collect_static /home/root/fluent-bit/plugins/in_tail/tail.c:140
    #12 0x4619d9 in flb_input_collector_fd /home/root/fluent-bit/src/flb_input.c:1004
    #13 0x482f04 in flb_engine_handle_event /home/root/fluent-bit/src/flb_engine.c:352
    #14 0x482f04 in flb_engine_start /home/root/fluent-bit/src/flb_engine.c:613
    #15 0x446d4b in flb_lib_worker /home/root/fluent-bit/src/flb_lib.c:493
    #16 0x7efe06e5fe74 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307
Thread T1 (flb-pipeline) created by T0 here:
    #0 0x7efe070ad332 in __interceptor_pthread_create /jenkins/gcc-9.2.0/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x8cc164 in mk_utils_worker_spawn /home/root/fluent-bit/lib/monkey/mk_core/mk_utils.c:251
    #2 0x446f65 in flb_start /home/root/fluent-bit/src/flb_lib.c:524
    #3 0x439ee1 in flb_main /home/root/fluent-bit/src/fluent-bit.c:1153
    #4 0x439f45 in main /home/root/fluent-bit/src/fluent-bit.c:1170
    #5 0x7efe06187574 in __libc_start_main ../csu/libc-start.c:266
SUMMARY: AddressSanitizer: heap-use-after-free /jenkins/gcc-9.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0fe04060ac60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060ac70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060ac80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060ac90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060aca0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0fe04060acb0: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x0fe04060acc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060acd0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060ace0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060acf0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fe04060ad00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==4297==ABORTING

jevgenimarenkov · 2021-04-28T19:57:44Z

Seems that there is a memory leak in memcpy(cf->st_content + cf->data_size, buf, count) as buf is already freed for some reason when that method is called.

agup006 · 2021-05-06T02:02:00Z

Thanks for providing the Stack Trace for 1.7.0, would you be able to provide the same with 1.7.4/latest?

jevgenimarenkov · 2021-05-06T05:26:34Z

@agup006, added stack trace for the 1.7.4

2021-05-06_05:25:22.60883 [2021/05/05 22:25:22] [engine] caught signal (SIGSEGV)
2021-05-06_05:25:22.76339 #0  0x7fc7906dbd78      in  ???() at s/x86_64/multiarch/memcpy-ssse3-back.S:279
2021-05-06_05:25:22.76547 #1  0x78a880            in  cio_file_write() at lib/chunkio/src/cio_file.c:824
2021-05-06_05:25:22.76555 #2  0x7870b5            in  cio_chunk_write() at lib/chunkio/src/cio_chunk.c:173
2021-05-06_05:25:22.76770 #3  0x4ecff7            in  flb_input_chunk_write() at src/flb_input_chunk.c:90
2021-05-06_05:25:22.76771 #4  0x4ee770            in  flb_input_chunk_append_raw() at src/flb_input_chunk.c:892
2021-05-06_05:25:22.76795 #5  0x507237            in  process_content() at plugins/in_tail/tail_file.c:358
2021-05-06_05:25:22.76796 #6  0x508ecb            in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:981
2021-05-06_05:25:22.76806 #7  0x50319a            in  in_tail_collect_pending() at plugins/in_tail/tail.c:91
2021-05-06_05:25:22.76980 #8  0x4c2294            in  flb_input_collector_fd() at src/flb_input.c:1004
2021-05-06_05:25:22.76993 #9  0x4d1be5            in  flb_engine_handle_event() at src/flb_engine.c:352
2021-05-06_05:25:22.76994 #10 0x4d1be5            in  flb_engine_start() at src/flb_engine.c:613
2021-05-06_05:25:22.77158 #11 0x4b7e4a            in  flb_lib_worker() at src/flb_lib.c:493
2021-05-06_05:25:22.77179 #12 0x7fc791280e74      in  start_thread() at debug/glibc-2.17-c758a686/nptl/pthread_create.c:307
2021-05-06_05:25:22.77367 #13 0x7fc7906858fc      in  ???() at s/unix/sysv/linux/x86_64/clone.S:111
2021-05-06_05:25:22.77368 #14 0xffffffffffffffff  in  ???() at ???:0
2021-05-06_05:25:22.91351 run returned -1 6
2021-05-06_05:25:25.91722

jevgenimarenkov · 2021-05-06T06:01:49Z

@agup006
Added AddressSanitizer report for 1.7.4

=================================================================
==12345==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f5fd0dd32d4 at pc 0x7f5fd7129af0 bp 0x7f5fd2afe2c0 sp 0x7f5fd2afda70
WRITE of size 11872 at 0x7f5fd0dd32d4 thread T1 (flb-pipeline)
    #0 0x7f5fd7129aef in __interceptor_memcpy /jenkins/gcc-9.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790
    #1 0x8965f0 in cio_file_write /root/fluent-bit/lib/chunkio/src/cio_file.c:824
    #2 0x892e25 in cio_chunk_write /root/fluent-bit/lib/chunkio/src/cio_chunk.c:173
    #3 0x4c69e8 in flb_input_chunk_write /root/fluent-bit/src/flb_input_chunk.c:90
    #4 0x4ca11f in flb_input_chunk_append_raw /root/fluent-bit/src/flb_input_chunk.c:892
    #5 0x50d43c in process_content /root/fluent-bit/plugins/in_tail/tail_file.c:358
    #6 0x51236a in flb_tail_file_chunk /root/fluent-bit/plugins/in_tail/tail_file.c:981
    #7 0x501a02 in in_tail_collect_pending /root/fluent-bit/plugins/in_tail/tail.c:91
    #8 0x46094c in flb_input_collector_fd /root/fluent-bit/src/flb_input.c:1004
    #9 0x481f58 in flb_engine_handle_event /root/fluent-bit/src/flb_engine.c:352
    #10 0x481f58 in flb_engine_start /root/fluent-bit/src/flb_engine.c:613
    #11 0x445c7b in flb_lib_worker /root/fluent-bit/src/flb_lib.c:493
    #12 0x7f5fd6e7ee74 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307
    #13 0x7f5fd62828fc in clone (/lib64/libc.so.6+0xfe8fc)

0x7f5fd0dd32d4 is located 2772 bytes inside of 131072-byte region [0x7f5fd0dd2800,0x7f5fd0df2800)
freed by thread T1 (flb-pipeline) here:
    #0 0x7f5fd7199b57 in __interceptor_free /jenkins/gcc-9.2.0/libsanitizer/asan/asan_malloc_linux.cc:122
    #1 0x460e33 in flb_free /root/fluent-bit/include/fluent-bit/flb_mem.h:122
    #2 0x4619d0 in flb_filter_do /root/fluent-bit/src/flb_filter.c:183
    #3 0x4ca327 in flb_input_chunk_append_raw /root/fluent-bit/src/flb_input_chunk.c:909
    #4 0x50d43c in process_content /root/fluent-bit/plugins/in_tail/tail_file.c:358
    #5 0x51236a in flb_tail_file_chunk /root/fluent-bit/plugins/in_tail/tail_file.c:981
    #6 0x50287c in in_tail_collect_event /root/fluent-bit/plugins/in_tail/tail.c:261
    #7 0x520791 in tail_fs_event /root/fluent-bit/plugins/in_tail/tail_fs_inotify.c:268
    #8 0x46094c in flb_input_collector_fd /root/fluent-bit/src/flb_input.c:1004
    #9 0x481f58 in flb_engine_handle_event /root/fluent-bit/src/flb_engine.c:352
    #10 0x481f58 in flb_engine_start /root/fluent-bit/src/flb_engine.c:613
    #11 0x445c7b in flb_lib_worker /root/fluent-bit/src/flb_lib.c:493
    #12 0x7f5fd6e7ee74 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307

previously allocated by thread T1 (flb-pipeline) here:
    #0 0x7f5fd719a30e in __interceptor_realloc /jenkins/gcc-9.2.0/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x663ad9 in msgpack_sbuffer_write /root/fluent-bit/lib/msgpack-c/include/msgpack/sbuffer.h:81
    #2 0x8993e9 in msgpack_pack_str_body /root/fluent-bit/lib/msgpack-c/include/msgpack/pack_template.h:784
    #3 0x89984d in msgpack_pack_object /root/fluent-bit/lib/msgpack-c/src/objectc.c:68
    #4 0x899a0a in msgpack_pack_object /root/fluent-bit/lib/msgpack-c/src/objectc.c:115
    #5 0x66dbdf in apply_modifying_rules /root/fluent-bit/plugins/filter_modify/modify.c:1225
    #6 0x66e02a in cb_modify_filter /root/fluent-bit/plugins/filter_modify/modify.c:1297
    #7 0x461525 in flb_filter_do /root/fluent-bit/src/flb_filter.c:118
    #8 0x4ca327 in flb_input_chunk_append_raw /root/fluent-bit/src/flb_input_chunk.c:909
    #9 0x50d43c in process_content /root/fluent-bit/plugins/in_tail/tail_file.c:358
    #10 0x51236a in flb_tail_file_chunk /root/fluent-bit/plugins/in_tail/tail_file.c:981
    #11 0x50287c in in_tail_collect_event /root/fluent-bit/plugins/in_tail/tail.c:261
    #12 0x520791 in tail_fs_event /root/fluent-bit/plugins/in_tail/tail_fs_inotify.c:268
    #13 0x46094c in flb_input_collector_fd /root/fluent-bit/src/flb_input.c:1004
    #14 0x481f58 in flb_engine_handle_event /root/fluent-bit/src/flb_engine.c:352
    #15 0x481f58 in flb_engine_start /root/fluent-bit/src/flb_engine.c:613
    #16 0x445c7b in flb_lib_worker /root/fluent-bit/src/flb_lib.c:493
    #17 0x7f5fd6e7ee74 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307

Thread T1 (flb-pipeline) created by T0 here:
    #0 0x7f5fd70cc332 in __interceptor_pthread_create /jenkins/gcc-9.2.0/libsanitizer/asan/asan_interceptors.cc:208
    #1 0x8b9370 in mk_utils_worker_spawn /root/fluent-bit/lib/monkey/mk_core/mk_utils.c:251
    #2 0x445e95 in flb_start /root/fluent-bit/src/flb_lib.c:524
    #3 0x438e11 in flb_main /root/fluent-bit/src/fluent-bit.c:1153
    #4 0x438e75 in main /root/fluent-bit/src/fluent-bit.c:1170
    #5 0x7f5fd61a6574 in __libc_start_main ../csu/libc-start.c:266

SUMMARY: AddressSanitizer: heap-use-after-free /jenkins/gcc-9.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:790 in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0fec7a1b2600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2610: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2620: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2630: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2640: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0fec7a1b2650: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x0fec7a1b2660: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2670: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b2690: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fec7a1b26a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==12345==ABORTING

github-actions · 2021-06-06T02:11:26Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions · 2021-06-12T01:47:51Z

This issue was closed because it has been stalled for 5 days with no activity.

jevgenimarenkov changed the title ~~SIGSEGV error when sending 5000 log lines per second.~~ Fluent-bit crashes with multiple (5-6 inputs/outputs) every 3 - 5 minutes (SIGSEGV error) Apr 24, 2021

jevgenimarenkov changed the title ~~Fluent-bit crashes with multiple (5-6 inputs/outputs) every 3 - 5 minutes (SIGSEGV error)~~ Fluent-bit crashes with multiple (5-6 inputs/outputs) every 3 - 5 minutes (SIGSEGV error) on high load Apr 24, 2021

jevgenimarenkov mentioned this issue Apr 24, 2021

Bad file descriptor (errno 9) on very high load #3326

Closed

jevgenimarenkov changed the title ~~Fluent-bit crashes with multiple (5-6 inputs/outputs) every 3 - 5 minutes (SIGSEGV error) on high load~~ [1.7.x] Fluent-bit crashes with multiple (5-6 inputs/outputs) every 3 - 5 minutes (SIGSEGV error) on high load Apr 24, 2021

jevgenimarenkov changed the title ~~[1.7.x] Fluent-bit crashes with multiple (5-6 inputs/outputs) every 3 - 5 minutes (SIGSEGV error) on high load~~ [1.7.x] Fluent-bit crashes with multiple inputs/outputs every 3 - 5 minutes (SIGSEGV error) on high load Apr 24, 2021

jevgenimarenkov changed the title ~~[1.7.x] Fluent-bit crashes with multiple inputs/outputs every 3 - 5 minutes (SIGSEGV error) on high load~~ [1.7.x] Fluent-bit crashes with multiple inputs/outputs (SIGSEGV error) on high load Apr 27, 2021

github-actions bot added the Stale label Jun 6, 2021

github-actions bot closed this as completed Jun 12, 2021

nokute78 mentioned this issue Dec 24, 2021

Frequent SIGSEGV with Fluent Bit 1.8.7 #4164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[1.7.x] Fluent-bit crashes with multiple inputs/outputs (SIGSEGV error) on high load #3412

[1.7.x] Fluent-bit crashes with multiple inputs/outputs (SIGSEGV error) on high load #3412

jevgenimarenkov commented Apr 23, 2021 •

edited

Loading

agup006 commented Apr 23, 2021

jevgenimarenkov commented Apr 24, 2021

jevgenimarenkov commented Apr 24, 2021 •

edited

Loading

jevgenimarenkov commented Apr 24, 2021 •

edited

Loading

jevgenimarenkov commented Apr 28, 2021

jevgenimarenkov commented Apr 28, 2021

jevgenimarenkov commented Apr 28, 2021

agup006 commented May 6, 2021

jevgenimarenkov commented May 6, 2021 •

edited

Loading

jevgenimarenkov commented May 6, 2021

github-actions bot commented Jun 6, 2021

github-actions bot commented Jun 12, 2021

[1.7.x] Fluent-bit crashes with multiple inputs/outputs (SIGSEGV error) on high load #3412

[1.7.x] Fluent-bit crashes with multiple inputs/outputs (SIGSEGV error) on high load #3412

Comments

jevgenimarenkov commented Apr 23, 2021 • edited Loading

Bug Report

agup006 commented Apr 23, 2021

jevgenimarenkov commented Apr 24, 2021

jevgenimarenkov commented Apr 24, 2021 • edited Loading

jevgenimarenkov commented Apr 24, 2021 • edited Loading

jevgenimarenkov commented Apr 28, 2021

jevgenimarenkov commented Apr 28, 2021

jevgenimarenkov commented Apr 28, 2021

agup006 commented May 6, 2021

jevgenimarenkov commented May 6, 2021 • edited Loading

jevgenimarenkov commented May 6, 2021

github-actions bot commented Jun 6, 2021

github-actions bot commented Jun 12, 2021

jevgenimarenkov commented Apr 23, 2021 •

edited

Loading

jevgenimarenkov commented Apr 24, 2021 •

edited

Loading

jevgenimarenkov commented Apr 24, 2021 •

edited

Loading

jevgenimarenkov commented May 6, 2021 •

edited

Loading