Fluentbit TLS SIGSEGV error

[dtwork2022]

Bugs should be filed for issues encountered whilst operating logging-operator.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors. #logging-operator
Please provide as much detail as possible.

Describe the bug:
When tls is enabled for Fluentbit using latest banzaicloud chart of 3.17.6 I'm getting CrashLoopBackOff on Fluentbit pods with log errors

[2022/05/31 17:40:06] [engine] caught signal (SIGSEGV)
[2022/05/31 17:40:06] [engine] caught signal (SIGSEGV)
#0  0x55d4a79abaa0      in  flb_tls_session_create() at src/tls/flb_tls.c:334
#1  0x55d4a79abaa0      in  flb_tls_session_create() at src/tls/flb_tls.c:334
#2  0x55d4a79b7699      in  flb_io_net_connect() at src/flb_io.c:109
#3  0x55d4a79b7699      in  flb_io_net_connect() at src/flb_io.c:109
#4  0x55d4a7995d13      in  create_conn() at src/flb_upstream.c:560
#5  0x55d4a799620f      in  flb_upstream_conn_get() at src/flb_upstream.c:705
#6  0x55d4a7995d13      in  create_conn() at src/flb_upstream.c:560
#7  0x55d4a799620f      in  flb_upstream_conn_get() at src/flb_upstream.c:705
#8  0x55d4a7a29c71      in  cb_forward_flush() at plugins/out_forward/forward.c:1182
#9  0x55d4a7a29c71      in  cb_forward_flush() at plugins/out_forward/forward.c:1182
#10 0x55d4a797fbe8      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:517
#10 0x55d4a797fbe8      in  output_pre_cb_flush() at include/fluent-bit/flb_output.h:517
#12 0x55d4a7ec0d66      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#13 0xffffffffffffffff  in  ???() at ???:0
#14 0x55d4a7ec0d66      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#15 0xffffffffffffffff  in  ???() at ???:0

Expected behaviour:
TLS should work if enabled and not crash

Steps to reproduce the bug:
Deploy 3.17.6 banzaicloud logging-operator-logging with tls enabled, in this config I disabled tls verify

fluentd:
  fluentLogDestination: "forward\n  <server>\n  name localhost\n  host 127.0.0.1\n  port 24240\n</server>"
  scaling:
    replicas: 3
  resources:
    limits:
      cpu: 2000m
      memory: 2000Mi
    requests:
      cpu: 2000m
      memory: 2000Mi
  metrics:
    serviceMonitor: true
fluentbit:
  inputTail:
    Mem_Buf_Limit: "10MB"
  filterKubernetes:
    Cache_Use_Docker_Id: "On"
    Buffer_Size: "64k"
  enableUpstream: true
  positiondb:
    hostPath:
      path: /var/lib/fluent-bit
  resources:
    limits:
      cpu: 400m
      memory: 500Mi
    requests:
      cpu: 400m
      memory: 500Mi
  metrics:
    serviceMonitor: true
disablePvc: false
enableHostPath: false
tls:
  enabled: true
  verify: false

Additional context:
Add any other context about the problem here.

Environment details:

• Kubernetes version (e.g. v1.15.2): 1.18.6
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): Rancher / RKE On-premise
• logging-operator version (e.g. 2.1.1): 3.17.6
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs): pasted above in problem
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data: pasted above in reproduction section

/kind bug

[dtwork2022]

bump, any response on this ?

[oshelot]

Going to bump this instead of creating a new bug. I tested the two following OUTPUT configs:

I forwarded to another local syslog server and everything worked as expected.

[OUTPUT]
    name                 syslog
    match                sys_out
    host                 localhost
    port                 13000
    mode                 tcp
    syslog_format        rfc3164
    syslog_message_key   message
    syslog_severity_key  pri
    syslog_facility_key  facility
    syslog_hostname_key  host
    syslog_appname_key   ident

The TLS config:

[OUTPUT]
    name                 syslog
    match                sys_out
    host                 *******
    port                 443
    mode                 tls
    syslog_format        rfc3164
    syslog_message_key   message
    syslog_severity_key  pri
    syslog_facility_key  facility
    syslog_hostname_key  host
    syslog_appname_key   ident
    tls.debug		 4
    tls.verify           off
    tls.ca_file          /tmp/chain.crt
    tls.crt_file         /tmp/domain.crt
    tls.key_file         /tmp/domain.key
    tls.key_passwd	 password

This errors almost immediately with:

[2022/06/21 17:15:08] [engine] caught signal (SIGSEGV)
#0 0x557ad3cce110 in flb_tls_session_create() at src/tls/flb_tls.c:334
fluent/fluent-bit#1 0x557ad3cdc985 in flb_io_net_connect() at src/flb_io.c:109
fluent/fluent-bit#2 0x557ad3cb4586 in create_conn() at src/flb_upstream.c:560
fluent/fluent-bit#3 0x557ad3cb4b28 in flb_upstream_conn_get() at src/flb_upstream.c:705
fluent/fluent-bit#4 0x557ad3da2c57 in cb_syslog_flush() at plugins/out_syslog/syslog.c:768
fluent/fluent-bit#5 0x557ad3c97e1b in output_pre_cb_flush() at include/fluent-bit/flb_output.h:517
fluent/fluent-bit#6 0x557ad42395ea in co_init() at lib/monkey/deps/flb_libco/amd64.c:117
fluent/fluent-bit#7 0xffffffffffffffff in ???() at ???:0

FB version 1.9.4
OS - Ubuntu 21 (esx VM)

[oshelot]

Upgraded to 1.9.5 and issue persists.

[edsiper]

@oshelot can you create a specific Fluent Bit issue in our repo https://github.com/fluent/fluent-bit ? , we will need FLB version, minimized repro case

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!