Filebeat: stdin prospector doesn't stop at closed pipe #6313
Labels
Comments
|
@ruflin perhaps you can comment on whether this is expected or not. |
|
Filebeat is only expected to stop when the |
|
@ruflin I did try with |
|
@tsg I think then it's a bug. Could you post the log output for the case where you run it with |
|
There you go: It stopped when I pressed ^C, ~1m after starting. (see the "sigint/sigterm" line ) |
|
It actually stops the crawlers and inputs as expected. Unfortunately we don't have enough logging to see the exact point it gets stuck. Since I last looked at it there is some new code around auto discovery. Just an idea. |
|
We are having the same problem resulting in this log (using v6.1.2): |
Something wrong with the one pushing to this channel - https://github.com/elastic/beats/blob/master/filebeat/beater/signalwait.go#L28 .. because .. results in .. Not sure about it. Will debug more tomorrow. |
|
I was digging a bit deeper into this issue and I think I found the cause of it. The problem is that the Additional note: The registry still has a |
|
Yes, problems still exist today with current master, taking a look. |
|
Looking at the code, we are stuck in a defer waitPublished := fb.config.ShutdownTimeout > 0 || *once
if waitPublished {
// Wait for registrar to finish writing registry
waitEvents.Add(withLog(wgEvents.Wait,
"Continue shutdown: All enqueued events being published."))
// Wait for either timeout or all events having been ACKed by outputs.
if fb.config.ShutdownTimeout > 0 {
logp.Info("Shutdown output timer started. Waiting for max %v.", timeout)
waitEvents.Add(withLog(waitDuration(timeout),
"Continue shutdown: Time out waiting for events being published."))
} else {
waitEvents.AddChan(fb.done)
}
}
return nil
} |
|
The problem is simple, solution takes a bit more times, I will just do a small brain dump. Stdin sources doesn't have a state so So in that case Filebeat waits on events that will never be written to disk, because we don't write states in the registry when a harvester is configured for stdin. I am not familiar enough with the 5.6.x branch, but I guess we were writing states for everything or at least call the right callback. I am working on a fix for that. |
|
After doing a quick spike, I think the best solution to fix this problem with the current code base is to do the following:
This will take a bit more time to do but I believe this is the best solution. |
|
Looks like a fix is finally tied to the inputs separation project/cleanup. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
botelastic
bot
added
Stalled
needs_team
|
This issue doesn't have a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With this configuration file:
and running Filebeat like this:
Filebeat processes correctly the test event, the stdin harvester stops, but it seems that the prospectors keeps on going and Filebeat doesn't stop:
I would expect that Filebeat automatically stops when reading from stdin and the pipe is closed. That's also the behaviour that Logstash has in that situation.
The text was updated successfully, but these errors were encountered: