Exec plugin doesn't work in docker container

[darrug]

Bug Report

Describe the bug
Exec input plugin doesn't work using the docker image. Tested the same command in centos 7 downloading the package and works. I have an executable that send logs to stdout but just for testing purposes I executed the command described in the manual

To Reproduce

docker run fluent/fluent-bit:1.3.2-debug /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /etc' -o stdout

output:

Fluent Bit v1.3.2
Copyright (C) Treasure Data

[2019/11/19 20:43:08] [ info] [storage] initializing...
[2019/11/19 20:43:08] [ info] [storage] in-memory
[2019/11/19 20:43:08] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2019/11/19 20:43:08] [ info] [engine] started (pid=1)
[2019/11/19 20:43:08] [ info] [sp] stream processor started

Expected behavior
Show stdout of the command in the output like this

Fluent Bit v1.3.2
Copyright (C) Treasure Data

[2019/11/19 21:44:45] [ info] [storage] initializing...
[2019/11/19 21:44:45] [ info] [storage] in-memory
[2019/11/19 21:44:45] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2019/11/19 21:44:45] [ info] [engine] started (pid=23020)
[2019/11/19 21:44:45] [ info] [sp] stream processor started
[0] exec.0: [1574196286.008087197, {"exec"=>"adjtime"}]
[1] exec.0: [1574196286.008141435, {"exec"=>"aliases"}]

Screenshots

Your Environment

• Version used: fluent/fluent-bit:1.3.2-debug
• Configuration: command line from documentation examples fluent-bit -i exec -p 'command=ls /var/log' -o stdout
• Server type and version: docker 19.03.5
• Operating System and version: Macos 10.15.1 and Centos 7
• Filters and plugins: exec plugin

Additional context
I have to launch an executable and read its stdout

[nokute78]

@darrug
ls command is not installed in fluentbit container.
So in_exec plugin does not execute ls.

[darrug]

@nokute78 I was trying using the version 1.3.2-debug which has the ls installed. This is the output of the command inside the container

[kantica]

This works for me on master. Could you test it also? Build debug container using master repository?
Like:

git clone https://github.com/fluent/fluent-bit.git
cd fluent-bit
docker build -t fluent-bit:master .
cat >Dockerfile.debug.master<<EOF
FROM debian:stretch as builder
COPY --from=amd64/busybox:1.31.0 /bin/busybox /bin/busybox
RUN chmod 555 /bin/busybox \
 && /bin/busybox --install

FROM fluent-bit:master
COPY --from=builder /bin/ /bin/
EOF

docker build -f Dockerfile.debug.master -t fluent-bit:master-debug .

And afterwards run it:
docker run fluent-bit:master-debug /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /etc' -o stdout

[darrug]

Yes on master input plugin works. Executing your instructions I have this output

I wrote a simple program that prints messages every second

package main
import (
	"log"
	"time"
)

func main() {
	i := 0
	for {
		log.Printf("message %v\n", i)
		i++
		time.Sleep(1 * time.Second)
	}
}

this is the plain output

Then I built a new image adding my executable

cat >Dockerfile.logmsg<<EOF
FROM golang:latest AS builder
WORKDIR /app
# Copy go mod and sum files
COPY go.mod go.sum ./
# Download all dependencies. Dependencies will be cached if the go.mod and go.sum files are not changed
RUN go mod download
# Copy the source from the current directory to the Working Directory inside the container
COPY ./logmsg.go .
# Build the Go app
RUN GOOS=linux GOARCH=amd64 go build logmsg.go 

FROM fluent-bit:master-debug
COPY --from=builder /app/logmsg /usr/bin/
EOF

docker build -f Dockerfile.logmsg -t fluent-bit-logmsg:master-debug .

Executing

docker run fluent-bit-logmsg:master-debug /fluent-bit/bin/fluent-bit -i exec -p 'command=logmsg' -o stdout

I have the output

It seems that the output is not parsed by fluentbit like it happens with the "ls" command. Do I need to specify a parser? Is there any default not documented?

[nokute78]

@darrug oops, I tested wrong container..

The root cause is missing /bin/sh.
in_exec uses popen(3) to execute command and the command is passed to /bin/sh.

http://man7.org/linux/man-pages/man3/popen.3.html

This command is passed to /bin/sh using the -c flag; interpretation, if any, is performed by the shell.

in_exec supports only one time command.
The patch to support long-running command like top(1) or your golang command is #732, but it didn't merged .

[darrug]

Thank you for the explanation. Long running commands aside, if /bin/shis missing is not possible to use the in_exec with the docker image at the moment. Is there a plan to change the base docker image to fix this?

[nokute78]

It is easy to add symlink /bin/sh to image.
I will try to make a patch to modify Dockerfile.

It may be better to replace popen to make fluent-bit more portable,
but it will take time to fix and test.

[panqingfeng]

any update?

[edsiper]

@nokute78 ping

[nokute78]

We are using gcr.io/distroless/cc-debian10 as a base image.
https://github.com/GoogleContainerTools/distroless/blob/main/cc/README.md
https://github.com/GoogleContainerTools/distroless/blob/main/base/README.md

It contains only an openssl and it doesn't contain shell and other executable binaries.
I want to know use case...

Is fluent-bit:master-debug container suitable for production?

[edsiper]

the debug version of the container is based on busybox but has a shell, the point of distroless is the security on that aspect.

hmm we cannot recommend -debug version for production usage. so it might be a wontfix.

[patrick-stephens]

It seems like we should remove the exec plugin from the container if it's not usable to keep it small and secure - plus make it obvious it won't work before people try to debug it.

[nokute78]

@patrick-stephens I agree.

@edsiper Which Dockerfile is to fix ?

https://github.com/fluent/fluent-bit/tree/master/dockerfiles
or
https://github.com/fluent/fluent-bit-docker-image

[patrick-stephens]

The one on master is the primary one now (from 1.9 onwards). I can look to do this shortly, I want to also verify whether the debug image supports it now we no longer need busybox.

[patrick-stephens]

The distroless AMD64 image will not work but the ARM images will do as they are not distroless yet - see #4691 .

So yes, it was busybox messing things up by the looks of it as we had a proper shell but then did a busybox install. Now with 1.8.12+ the debug images correctly function:

$ docker run --rm -it fluent/fluent-bit:1.8.12-debug /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Fluent Bit v1.8.12
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/01/31 14:19:55] [ info] [engine] started (pid=1)
[2022/01/31 14:19:55] [ info] [storage] version=1.1.5, initializing...
[2022/01/31 14:19:55] [ info] [storage] in-memory
[2022/01/31 14:19:55] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/01/31 14:19:55] [ info] [cmetrics] version=0.2.2
[2022/01/31 14:19:55] [ info] [sp] stream processor started
[0] exec.0: [1643638796.378421341, {"exec"=>"alternatives.log"}]
[1] exec.0: [1643638796.378491633, {"exec"=>"apt"}]
[2] exec.0: [1643638796.378503958, {"exec"=>"btmp"}]
[3] exec.0: [1643638796.378509405, {"exec"=>"dpkg.log"}]
[4] exec.0: [1643638796.378514609, {"exec"=>"faillog"}]
[5] exec.0: [1643638796.378519881, {"exec"=>"lastlog"}]
[6] exec.0: [1643638796.378524946, {"exec"=>"wtmp"}]
[7] exec.0: [1643638797.377786709, {"exec"=>"alternatives.log"}]
[8] exec.0: [1643638797.377818323, {"exec"=>"apt"}]
[9] exec.0: [1643638797.377822443, {"exec"=>"btmp"}]
[10] exec.0: [1643638797.377826368, {"exec"=>"dpkg.log"}]
[11] exec.0: [1643638797.377830265, {"exec"=>"faillog"}]
[12] exec.0: [1643638797.377833919, {"exec"=>"lastlog"}]
[13] exec.0: [1643638797.377837304, {"exec"=>"wtmp"}]
[14] exec.0: [1643638798.377972992, {"exec"=>"alternatives.log"}]
[15] exec.0: [1643638798.378000986, {"exec"=>"apt"}]
[16] exec.0: [1643638798.378004914, {"exec"=>"btmp"}]
[17] exec.0: [1643638798.378008585, {"exec"=>"dpkg.log"}]
[18] exec.0: [1643638798.378012329, {"exec"=>"faillog"}]
[19] exec.0: [1643638798.378015994, {"exec"=>"lastlog"}]
[20] exec.0: [1643638798.378019675, {"exec"=>"wtmp"}]
[21] exec.0: [1643638799.377839073, {"exec"=>"alternatives.log"}]
[22] exec.0: [1643638799.377870744, {"exec"=>"apt"}]
[23] exec.0: [1643638799.377875993, {"exec"=>"btmp"}]
[24] exec.0: [1643638799.377880891, {"exec"=>"dpkg.log"}]
[25] exec.0: [1643638799.377885832, {"exec"=>"faillog"}]
[26] exec.0: [1643638799.377890703, {"exec"=>"lastlog"}]
[27] exec.0: [1643638799.377895378, {"exec"=>"wtmp"}]
^C[2022/01/31 14:20:01] [engine] caught signal (SIGINT)
[0] exec.0: [1643638800.377519675, {"exec"=>"alternatives.log"}]
[1] exec.0: [1643638800.377571752, {"exec"=>"apt"}]
[2] exec.0: [1643638800.377574582, {"exec"=>"btmp"}]
[3] exec.0: [1643638800.377576917, {"exec"=>"dpkg.log"}]
[4] exec.0: [1643638800.377579286, {"exec"=>"faillog"}]
[5] exec.0: [1643638800.377581633, {"exec"=>"lastlog"}]
[6] exec.0: [1643638800.377583963, {"exec"=>"wtmp"}]
[2022/01/31 14:20:01] [ warn] [engine] service will shutdown in max 5 seconds
[2022/01/31 14:20:01] [ info] [engine] service has stopped (0 pending tasks)

The old 1.8.11 debug images do not (they just sit there as the call fails so notice the delay in the timestamps):

$ docker run --rm -it fluent/fluent-bit:1.8.11-debug /fluent-bit/bin/fluent-bit -i exec -p 'command=ls /var/log' -o stdout
Fluent Bit v1.8.11
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/01/31 14:20:22] [ info] [engine] started (pid=1)
[2022/01/31 14:20:22] [ info] [storage] version=1.1.5, initializing...
[2022/01/31 14:20:22] [ info] [storage] in-memory
[2022/01/31 14:20:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2022/01/31 14:20:22] [ info] [cmetrics] version=0.2.2
[2022/01/31 14:20:22] [ info] [sp] stream processor started
^C[2022/01/31 14:20:42] [engine] caught signal (SIGINT)
[2022/01/31 14:20:42] [ warn] [engine] service will shutdown in max 5 seconds
[2022/01/31 14:20:43] [ info] [engine] service has stopped (0 pending tasks)

I'm not sure we want to add a shell in just to allow this plugin as it invalidates the point of using Distroless. We should document it though on the exec plugin page.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.