1.9.9: exec input not available or crashes because of parser not found

[PettitWesley]

Bug Report

I'm having weird issues on 1.9.9 with the exec input that honestly do not make sense.

fluent-bit.conf:

[SERVICE]
    Flush        5
    Grace        30
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_PORT    2020
    Log_Level    info

[INPUT]
    Name exec
    Command curl -s http://127.0.0.1:2020/api/v1/metrics/prometheus
    Interval_Sec 30
    Tag fb_metrics
    Alias FBPrometheus

[OUTPUT]
    Name stdout
    Match *

147dda285fd4:1_9_9_parser-startup-crash wppttt$ docker run -it -v $(pwd):/fluent-bit/etc fluent/fluent-bit:1.9.9
Fluent Bit v1.9.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/11/22 00:19:47] [error] [config] section 'exec' tried to instance a plugin name that don't exists
[2022/11/22 00:19:47] [error] configuration file contains errors, aborting.

147dda285fd4:1_9_9_parser-startup-crash wppttt$ docker run -it -v $(pwd):/fluent-bit/etc fluent/fluent-bit:1.9.9-debug
Fluent Bit v1.9.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/11/22 00:20:41] [error] [config] section 'exec' tried to instance a plugin name that don't exists
[2022/11/22 00:20:41] [error] configuration file contains errors, aborting.

On a separate dev instance compiling Fluent Bit and running it directly I get:

[ec2-user@ip-10-192-11-106 build]$ ./bin/fluent-bit -c ~/issues/fluent-bit-exec.conf
Fluent Bit v1.9.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/11/22 00:22:02] [ info] [fluent bit] version=1.9.9, commit=5c03b2e555, pid=20210
[2022/11/22 00:22:02] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/11/22 00:22:02] [ info] [cmetrics] version=0.3.7
[2022/11/22 00:22:02] [error] [input:exec:FBPrometheus] requested parser '��' not found
[2022/11/22 00:22:02] [ info] [output:stdout:stdout.0] worker #0 started
[2022/11/22 00:22:02] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/11/22 00:22:02] [ info] [sp] stream processor started
^C[2022/11/22 00:22:05] [engine] caught signal (SIGINT)

I am not sure what is going wrong here... am I doing some obviously wrong?

Also not sure how I am getting different results on different machines/deployments of 1.9.9

[PettitWesley]

Oh I see, in the Docker image, exec input is not enabled- why?

https://github.com/fluent/fluent-bit/blob/master/dockerfiles/Dockerfile#L71

[PettitWesley]

This issue does not occur on the AWS release of 1.9.9:

[ec2-user@ip-10-192-11-106 image]$ docker run -it -v $(pwd):/fluent-bit/etc public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
AWS for Fluent Bit Container Image Version 2.28.4
Fluent Bit v1.9.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/11/22 00:26:16] [ info] [fluent bit] version=1.9.9, commit=5fcfe330e5, pid=1
[2022/11/22 00:26:16] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/11/22 00:26:16] [ info] [cmetrics] version=0.3.7
[2022/11/22 00:26:16] [ info] [output:stdout:stdout.0] worker #0 started
[2022/11/22 00:26:16] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/11/22 00:26:16] [ info] [sp] stream processor started
^C[2022/11/22 00:26:17] [engine] caught signal (SIGINT)
[2022/11/22 00:26:17] [ warn] [engine] service will shutdown in max 30 seconds
[2022/11/22 00:26:18] [ info] [engine] service has stopped (0 pending tasks)
[2022/11/22 00:26:18] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/11/22 00:26:18] [ info] [output:stdout:stdout.0] thread worker #0 stopped

[PettitWesley]

I also got a segfault when building FLB from source: https://gist.github.com/PettitWesley/4c141146837b2ac962edc498b41eb711

So this and the weird parser load is the only unresolved issue:

[ec2-user@ip-10-192-11-106 build]$ ./bin/fluent-bit -c ~/issues/fluent-bit-exec.conf
Fluent Bit v1.9.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/11/22 00:22:02] [ info] [fluent bit] version=1.9.9, commit=5c03b2e555, pid=20210
[2022/11/22 00:22:02] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/11/22 00:22:02] [ info] [cmetrics] version=0.3.7
[2022/11/22 00:22:02] [error] [input:exec:FBPrometheus] requested parser '��' not found
[2022/11/22 00:22:02] [ info] [output:stdout:stdout.0] worker #0 started
[2022/11/22 00:22:02] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/11/22 00:22:02] [ info] [sp] stream processor started
^C[2022/11/22 00:22:05] [engine] caught signal (SIGINT)

I just compiled latest v1.9.9 tag with CMake flags -DFLB_DEV=On -DFLB_DEBUG=On

[patrick-stephens]

Exec is disabled as it does not function in distroless containers - it requires the existence of /bin/sh. Previously it just silently failed if you specified it so you ended up getting invalid/strange output with no idea why.

There is an issue on it all somewhere, let me dig it out.

[patrick-stephens]

#4719 is the change, it links out to the issue.

The docs do cover it: https://docs.fluentbit.io/manual/pipeline/inputs/exec#container-support

[patrick-stephens]

Is this your issue as well @PettitWesley ? #5848

[nokute78]

@PettitWesley Could you check a patch #5720 ?
It is not merged for v1.9 branch.

[PettitWesley]

@nokute78 Thanks! I think that possibly explains what I was seeing

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.