Plugin causes fluent-bit to stop running in background mode

[boddob]

When running fluent-bit in background mode (i.,e Daemon set to yes in the conf file) and with the output plugin set to cloudwatch, the fluent-bit process seems to be always sleeping. I didn't see this issue when the output plugin was set to, say, the file plugin.

Doing a strace on the process shows it's stuck at:

epoll_pwait(1,

Is this expected, or am I doing something wrong?

[PettitWesley]

@boddob Legitimately interested in hearing more about this- I've never understood the Daemon mode setting.

How are you running Fluent Bit? I'm guessing not in a container?

I tried our using background mode in a container and the container seems to quit immediately.

Config file:

[SERVICE]
    Daemon On
    Log_Level debug

[INPUT]
    Name cpu
    Tag cpu

[OUTPUT]
    Name cloudwatch
    Match   *
    region us-east-1
    log_group_name fluent-bit-cloudwatch
    log_stream_prefix from-fluent-bit-
    auto_create_group true

Command:

$ docker run -it -v $(pwd):/fluent-bit/etc -e "FLB_LOG_LEVEL=debug" amazon/aws-for-fluent-bit:latest
Fluent Bit v1.3.2
Copyright (C) Treasure Data

[2019/11/21 06:40:59] [ info] switching to background mode (PID=20)

Running docker ps shows nothing... and checking the container with docker inspect shows it exited with code 0.

Forgive my ignorance, what is the use case for using background mode? I've actually wondered this for a while. Is it the same thing as making a process run in the background in a shell?

I assume this is for invoking Fluent Bit directly in a shell. So then whats the difference between running ./fluent-bit -c myconfig.conf & and running ./fluent-bit -c myconfig.conf with myconfig.conf having "Daemon On" in it?

[boddob]

@PettitWesley It's not in a container. I don't have a strong requirement of running fluent-bit in daemon mode. The reason why I'd considered setting it "Yes" was because I was going to wrap fluent-bit as a systemd service. I thought I might need to do that for it to run with systemd, so I thought I'd try the same. I've since learnt that that's not required.

I'm not sure, but I think shell's background mode and fluent-bit's "Daemon On" achieve similar things. There might be some subtleties, though. For example, with Daemon On, the original fluent-bit cli process we ran on shell would spawn a new process and exit. If run with bash in background mode, it will still spawn a new process, but I think it won't kill the parent (i.e, shell), and possibly still remain a child of shell? Another difference would be that in background mode, the process will stilll throw output to stdout stderr etc. This won't happen with daemon mode.

It' not necessary for me to get daemon mode running as such. It would just be nice to mention in the plugin's documentation somewhere that it has issues when fluent-bit is run in daemon mode.

[boddob]

I read some more about daemons. The main purpose of daemonise-ing a process seems to be to disassociate it from the standard input/output and make it unavailable to any terminal. There seems to a standard technique to achieve this [1] [2]. fluent-bit does that here:

https://github.com/fluent/fluent-bit/blob/master/src/flb_utils.c#L132

However, nowadays it's recommended that you leave this responsibility [3] to the OS's service manager, which is systemd for most linux distributions. In other words, trying to run fluent-bit in daemon mode while it's already a systemd service is wrong. fluent-bit probably gave the "Daemon" option in the first place for the sake of flexibility, just in case you were using a very minimal distro which didn't have systemd, or something along those lines.

I suspect that the cloudwatch plugin is possibly trying to access some resource that isn't available if fluent-bit is daemonized. Perhaps the plugin has hardcoded the use of stdout file descriptor somewhere, which isn't accessible in daemon mode? At least that's what the strace command pointed to.

I don't know much about containers, but I was wondering if the fluent-bit daemonizing code above did something that didn't sit well with containers. Perhaps one of the systems calls like setsid, or umask?

[1] https://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon/16317668#16317668
[2] https://stackoverflow.com/questions/473620/how-do-you-create-a-daemon-in-python
[3] https://www.mikeperham.com/2014/09/22/dont-daemonize-your-daemons/

[PettitWesley]

@boddob Interesting, thanks for all of that.

I suspect that the cloudwatch plugin is possibly trying to access some resource that isn't available if fluent-bit is daemonized.

I can't figure out what it would be though... the plugin code just uses memory and makes http requests.

This needs further investigation. I don't have time to look dive into it right now unfortunately, but I'll add a note in the README of this and the other plugins noting that Daemon mode may not work.

[boddob]

@boddob Interesting, thanks for all of that.

I suspect that the cloudwatch plugin is possibly trying to access some resource that isn't available if fluent-bit is daemonized.

I can't figure out what it would be though... the plugin code just uses memory and makes http requests.

This needs further investigation. I don't have time to look dive into it right now unfortunately, but I'll add a note in the README of this and the other plugins noting that Daemon mode may not work.

Sounds good. Thanks.