With threaded on, SIGSEGV occurs during a dry run

[soukichi]

Bug Report

Describe the bug
With threaded on, SIGSEGV occurs during a dry run

To Reproduce

• /etc/fluent-bit/fluent-bit.conf

[SERVICE]
    flush        1
    daemon      Off
    log_level    info

    parsers_file parsers.conf
    plugins_file plugins.conf
    http_server  On
    http_listen  0.0.0.0
    http_port    2020

[INPUT]
    name cpu
    tag  cpu.local
    interval_sec 1
    threaded on
[OUTPUT]
    name  stdout
    match *

$ sudo /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
Fluent Bit v3.1.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

configuration test is successful
[2024/09/05 17:53:57] [engine] caught signal (SIGSEGV)
#0  0x64644873e28e      in  flb_input_exit_all() at src/flb_input.c:1341
#1  0x646448759c6c      in  flb_engine_shutdown() at src/flb_engine.c:1121
#2  0x646448732694      in  flb_destroy() at src/flb_lib.c:240
#3  0x64644869afeb      in  flb_main() at src/fluent-bit.c:1360
#4  0x7c24ad22a1c9      in  __libc_start_call_main() at libc_start_call_main.h:58
#5  0x7c24ad22a28a      in  __libc_start_main_impl() at t.c:360
#6  0x646448698bd4      in  ???() at ???:0
#7  0xffffffffffffffff  in  ???() at ???:0
Aborted

remove "threaded on", it works.

$ sudo /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
Fluent Bit v3.1.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

configuration test is successful

Expected behavior
--dry-run should work with threaded on

Your Environment

• Version used: v3.1.7
• Operating System and version: "Ubuntu 24.04 LTS" 6.8.0-38-generic

[patrick-stephens]

This also fails with the container image when using --dry-run but works fine when using the actual config, i.e. not a dry run.

$ docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:3.1.7 -c /fluent-bit/etc/fluent-bit.conf --dry-run 
Fluent Bit v3.1.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

configuration test is successful
[2024/09/06 10:01:36] [engine] caught signal (SIGSEGV)
#0  0xaaaae293bd9b      in  flb_input_exit_all() at src/flb_input.c:1341
#1  0xaaaae2951fbb      in  flb_engine_shutdown() at src/flb_engine.c:1121
#2  0xaaaae2930c4f      in  flb_destroy() at src/flb_lib.c:240
#3  0xaaaae28c693f      in  flb_main() at src/fluent-bit.c:1360
#4  0xffff7f47777f      in  ???() at ???:0
#5  0xffff7f477857      in  ???() at ???:0
#6  0xaaaae28c466f      in  ???() at ???:0
#7  0xffffffffffffffff  in  ???() at ???:0

$ cat fluent-bit.conf 
[SERVICE]
    flush        1
    daemon      Off
    log_level    info

    parsers_file parsers.conf
    plugins_file plugins.conf
    http_server  On
    http_listen  0.0.0.0
    http_port    2020

[INPUT]
    name cpu
    tag  cpu.local
    interval_sec 1
    threaded on
[OUTPUT]
    name  stdout
    match *

[patrick-stephens]

Removing the threaded on config option works fine with and without dry run then.

[patrick-stephens]

Using the debug image also fails:

docker run --rm -it -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:3.1.7-debug /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf --dry-run
Fluent Bit v3.1.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

configuration test is successful
[2024/09/06 10:04:21] [engine] caught signal (SIGSEGV)
#0  0xaaaaabecbd9b      in  flb_input_exit_all() at src/flb_input.c:1341
#1  0xaaaaabee1fbb      in  flb_engine_shutdown() at src/flb_engine.c:1121
#2  0xaaaaabec0c4f      in  flb_destroy() at src/flb_lib.c:240
#3  0xaaaaabe5693f      in  flb_main() at src/fluent-bit.c:1360
#4  0xffff9887773f      in  __libc_start_call_main() at libc_start_call_main.h:58
#5  0xffff98877817      in  __libc_start_main_impl() at t.c:360
#6  0xaaaaabe5466f      in  ???() at ???:0
#7  0xffffffffffffffff  in  ???() at ???:0

[AkshayDubey29]

I am also getting this issue but the message while tracing it is different

Thread 11 "flb-pipeline" received signal SIGSEGV, Segmentation fault.
0x000055d0bc071df0 in flb_log_event_encoder_dynamic_field_scope_leave ()

(gdb) continue
Continuing.

Thread 11 "flb-pipeline" received signal SIGABRT, Aborted.
0x00007f88a6cf3a84 in __pthread_kill_implementation () from /lib/libc.so.6
(gdb) bt
#0 0x00007f88a6cf3a84 in __pthread_kill_implementation () from /lib/libc.so.6
#1 0x00007f88a6c9c27e in raise () from /lib/libc.so.6
#2 0x00007f88a6c84902 in abort () from /lib/libc.so.6
#3 0x000055d0bbf80a18 in flb_signal_handler ()
#4
#5 0x000055d0bc071df0 in flb_log_event_encoder_dynamic_field_scope_leave ()
#6 0x000055d0bc07213a in flb_log_event_encoder_dynamic_field_reset ()
#7 0x000055d0bc06fef9 in flb_log_event_encoder_reset ()
#8 0x000055d0bc0a3260 in ml_flush_callback ()
#9 0x000055d0bc051f7c in flb_ml_flush_stream_group ()
#10 0x000055d0bc051f86 in flb_ml_flush_parser_instance ()
#11 0x000055d0bc0730c6 in flb_ml_stream_id_destroy_all ()
#12 0x000055d0bc0a3a95 in flb_tail_file_remove ()
#13 0x000055d0bc09f54a in tail_fs_event ()
#14 0x000055d0bc039d74 in flb_input_collector_fd ()
#15 0x000055d0bc014471 in flb_engine_start ()
#16 0x00007f88a6cf14b5 in start_thread () from /lib/libc.so.6
#17 0x00007f88a6d073ac in __clone () from /lib/libc.so.6

(gdb) info locals
No symbol table info available.
(gdb) info registers
rax 0x0 0
rbx 0x1f 31
rcx 0x7f88a63ff6c0 140248908785968
rdx 0x6 6