SIGSEGV on apple m1 with loki appender

[alexmherrmann]

Bug Report

Describe the bug
A segmentation fault happens when using the loki appender. I am running on an M1 mac and with the OS X or Docker container the loki plugin causes me some grief after running for about 20 seconds.

To Reproduce

• I have the following loki configuration (using a TCP input)

[Output]
    Name loki
    Match logs*
    host logs-prod3.grafana.net
    port 443
    http_user ...
    http_passwd ...
    # (30KiB)
    Label_Keys $level, $loggerName, $dmjv, $appDomain
    Line_Format key_value

    tls on
    tls.verify on

• Here is the failure from the official docker container

[2023/02/07 19:23:03] [engine] caught signal (SIGSEGV)
#0  0xaaaac2222870      in  edata_arena_ind_get() at lib/jemalloc-5.3.0/include/jemalloc/internal/edata.h:258
#1  0xaaaac2222870      in  tcache_bin_flush_impl() at lib/jemalloc-5.3.0/src/tcache.c:350
#2  0xaaaac2222870      in  tcache_bin_flush_bottom() at lib/jemalloc-5.3.0/src/tcache.c:519
#3  0xaaaac2222870      in  je_tcache_bin_flush_small() at lib/jemalloc-5.3.0/src/tcache.c:529
#4  0xaaaac2223b27      in  tcache_gc_small() at lib/jemalloc-5.3.0/src/tcache.c:148
#5  0xaaaac2225d0f      in  tcache_event() at lib/jemalloc-5.3.0/src/tcache.c:193
#6  0xaaaac2225d0f      in  je_tcache_gc_event_handler() at lib/jemalloc-5.3.0/src/tcache.c:219
#7  0xaaaac222825b      in  je_te_event_trigger() at lib/jemalloc-5.3.0/src/thread_event.c:299
#8  0xaaaac21d482b      in  te_event_advance() at lib/jemalloc-5.3.0/include/jemalloc/internal/thread_event.h:287
#9  0xaaaac21d482b      in  thread_alloc_event() at lib/jemalloc-5.3.0/include/jemalloc/internal/thread_event.h:298
#10 0xaaaac21d482b      in  do_rallocx() at lib/jemalloc-5.3.0/src/jemalloc.c:3545
#11 0xaaaac22526e7      in  flb_realloc() at include/fluent-bit/flb_mem.h:95
#12 0xaaaac2252abf      in  flb_sds_increase() at src/flb_sds.c:107
#13 0xaaaac2252b6f      in  flb_sds_cat() at src/flb_sds.c:127
#14 0xaaaac258764f      in  safe_sds_cat() at plugins/out_loki/loki.c:44
#15 0xaaaac258a46b      in  pack_format_line_value() at plugins/out_loki/loki.c:985
#16 0xaaaac258a813      in  pack_format_line_value() at plugins/out_loki/loki.c:1044
#17 0xaaaac258a6cb      in  pack_format_line_value() at plugins/out_loki/loki.c:1023
#18 0xaaaac258a813      in  pack_format_line_value() at plugins/out_loki/loki.c:1044
#19 0xaaaac258b0eb      in  pack_record() at plugins/out_loki/loki.c:1228
#20 0xaaaac258b52b      in  loki_compose_payload() at plugins/out_loki/loki.c:1381
#21 0xaaaac258b607      in  cb_loki_flush() at plugins/out_loki/loki.c:1408
#22 0xaaaac2270f43      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:528
#23 0xaaaac2ce4867      in  co_switch() at lib/monkey/deps/flb_libco/aarch64.c:133
#24 0xffffffffffffffff  in  ???() at ???:0

Your Environment

• Version used: 2.0.9
• Environment name and version: latest docker desktop & OS X 16

Additional context

[patrick-stephens]

Presumably you are using the ARM64 container image here rather than anything specific to MacOS (which you would have to compile yourself for ARM64 packages)?

I just wanted to clarify the specific text here: I am running on an M1 mac and with the OS X or Docker container
Does that mean you are using a MacOS packages you've made yourself and the container? They both see it?

[alexmherrmann]

Sorry, that means I've installed from homebrew (an unofficial package it looks like) and run the same configuration on a docker container. I believe the container is an arm64 container. In both cases it segfaults in the loki plugin. The OS X build has a different error I should be able to reproduce later.

It seems to only happen when I have the loki output on

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.