dockerfiles: disable exec plugin for distroless containers

[patrick-stephens]

Signed-off-by: Patrick Stephens [email protected]

Addresses #1758 by triggering an error when exec is used rather than just silently failing:

$ docker run --rm -it ghcr.io/fluent/fluent-bit/pr-4719:x86_64-1.8.12 /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Error: Invalid input type. Aborting

Note this is for linux/amd64 platform only currently as it is the only distroless one until #4691 . If you run with the ARM containers it will function and is not disabled therefore.

Previously it would just silently fail - note the time differences here when it should start outputting the information in ~10 seconds:

$ docker run --rm -it fluent/fluent-bit:1.8.12 /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Fluent Bit v1.8.12
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/01/31 14:47:45] [ info] [engine] started (pid=1)
[2022/01/31 14:47:45] [ info] [storage] version=1.1.5, initializing...
[2022/01/31 14:47:45] [ info] [storage] in-memory
[2022/01/31 14:47:45] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/01/31 14:47:45] [ info] [cmetrics] version=0.2.2
[2022/01/31 14:47:45] [ info] [sp] stream processor started
^C[2022/01/31 14:49:10] [engine] caught signal (SIGINT)
[2022/01/31 14:49:10] [ warn] [engine] service will shutdown in max 5 seconds
[2022/01/31 14:49:11] [ info] [engine] service has stopped (0 pending tasks)

Versus the working ARM64 version:

$ docker run --platform=linux/arm64 --rm -it fluent/fluent-bit:1.8.12 /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Fluent Bit v1.8.12
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/01/31 14:50:50] [ info] [engine] started (pid=1)
[2022/01/31 14:50:50] [ info] [storage] version=1.1.5, initializing...
[2022/01/31 14:50:50] [ info] [storage] in-memory
[2022/01/31 14:50:50] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/01/31 14:50:50] [ info] [cmetrics] version=0.2.2
[2022/01/31 14:50:50] [ info] [sp] stream processor started
[0] exec.0: [1643640651.411744756, {"exec"=>"apt"}]
[1] exec.0: [1643640651.415636313, {"exec"=>"btmp"}]
[2] exec.0: [1643640651.415825691, {"exec"=>"faillog"}]
[3] exec.0: [1643640651.415833482, {"exec"=>"lastlog"}]
[4] exec.0: [1643640651.415839680, {"exec"=>"wtmp"}]
[5] exec.0: [1643640652.410733014, {"exec"=>"apt"}]
[6] exec.0: [1643640652.410788045, {"exec"=>"btmp"}]
[7] exec.0: [1643640652.410795660, {"exec"=>"faillog"}]
[8] exec.0: [1643640652.410801442, {"exec"=>"lastlog"}]
[9] exec.0: [1643640652.410807049, {"exec"=>"wtmp"}]
[10] exec.0: [1643640653.411706527, {"exec"=>"apt"}]
[11] exec.0: [1643640653.411775556, {"exec"=>"btmp"}]
[12] exec.0: [1643640653.411783106, {"exec"=>"faillog"}]
[13] exec.0: [1643640653.411789436, {"exec"=>"lastlog"}]
[14] exec.0: [1643640653.411795617, {"exec"=>"wtmp"}]
[15] exec.0: [1643640654.424729594, {"exec"=>"apt"}]
[16] exec.0: [1643640654.424802553, {"exec"=>"btmp"}]
[17] exec.0: [1643640654.424814347, {"exec"=>"faillog"}]
[18] exec.0: [1643640654.424825500, {"exec"=>"lastlog"}]
[19] exec.0: [1643640654.424847787, {"exec"=>"wtmp"}]
^C[2022/01/31 14:50:56] [engine] caught signal (SIGINT)
[0] exec.0: [1643640655.415746480, {"exec"=>"apt"}]
[1] exec.0: [1643640655.415949095, {"exec"=>"btmp"}]
[2] exec.0: [1643640655.415957242, {"exec"=>"faillog"}]
[3] exec.0: [1643640655.415963296, {"exec"=>"lastlog"}]
[4] exec.0: [1643640655.415969036, {"exec"=>"wtmp"}]
[5] exec.0: [1643640656.410408899, {"exec"=>"apt"}]
[6] exec.0: [1643640656.410461574, {"exec"=>"btmp"}]
[7] exec.0: [1643640656.410468982, {"exec"=>"faillog"}]
[8] exec.0: [1643640656.410474616, {"exec"=>"lastlog"}]
[9] exec.0: [1643640656.410480158, {"exec"=>"wtmp"}]
[2022/01/31 14:50:56] [ warn] [engine] service will shutdown in max 5 seconds
[2022/01/31 14:50:57] [ info] [engine] service has stopped (0 pending tasks)

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#692 indicates it is not supported now.

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[patrick-stephens]

@niedbalski all gravy now if you're happy to merge?

[sudhanvac]

Hi @patrick-stephens
I'm using fluent-bit:1.9.5 image and trying to use exec input plugin, it fails saying
`
Fluent Bit v1.9.5

• Copyright (C) 2015-2022 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

[2022/07/20 09:25:21] [error] [config] section 'exec' tried to instance a plugin name that don't exists
[2022/07/20 09:25:21] [error] configuration file contains errors, aborting.
`

Tried using fluent-bit:1.9.5-debug image, even there it fails with the above error.
Any idea? I'm I missing anything?

[patrick-stephens]

Exec is not supported in containers, it silently failed then triggered weird errors before so we made it explicit.

[sudhanvac]

@patrick-stephens , Thank you for the update.
Earlier we used exec to cleanup the old logs on certain intervals(fluent-bit version 1.8.6) as mentioned below -

Name exec
Tag exec_log_cleanup
Command find /data/logs -mmin +480 -regex '..[0-9].log' -delete
Interval_Sec 28800

Now we are planning to upgrade to the latest i.e fluent-bit 1.9.5 version.
Are there any alternative that I can use instead of exec?
Or any suggestion that I can follow?

[patrick-stephens]

Depends what your use case is and this PR is probably not the right place for that discussion, maybe try the Slack channel.

Exec will never have worked in distroless containers - it just silently failed so if you were using it with a distroless container before then it was not doing what you think it was. It requires a shell to function, and specifically the /bin/sh exact shell in that location.

Please have a look at the linked issue for details but this is the main reason for this specific PR: to highlight that a configuration using exec is invalid for the distroless containers rather than just hiding the error and something else breaking later.

Exec works fine outside of distroless containers, i.e. with native packages (RPM/DEB/etc.) or a container with a shell also allows exec to function, e.g. the UBI one we publish for RHCC here:

• https://catalog.redhat.com/software/containers/calyptia/fluent-bit/61d8a3609ea53314a92f8e79
• https://github.com/calyptia/fluent-bit

Also, your use case sounds exactly like something logrotate was designed to do. Why not just clean up via a standard mechanism like that rather than rely on side effects from Fluent Bit filters?