Filebeat TCP input panics after 2^31 events received #7202
Comments
|
After having a look at the code, the only possibility that seems plausible is that the internal counter from the WaitGroup is overflown (it uses an int32). Can be reproduced with this example: That means that |
adriansr
changed the title
adriansr
changed the title
|
Thanks for the report will verify that |
|
I believe TCP/UDP and redis are affected by this issues well. |
|
Just to leave some feedback on this issue. This is the current flow of events for the Redis, TCP and up.
The problem is the current implementation of the registry is global, even if we don't need it. I will check if I can use the same strategy as the stdin. |
ph
added a commit
to ph/beats
that referenced
this issue
May 30, 2018
This commit introduces a change in how filebat handle ACK by default, before the ACK was using the private field of the event to retrieve a state. The updated state was sent to the registrar, and the registrar was finalizing the ACK. But with the introduction of the TCP/UDP and the Redis input, the events don't have any state attached. So in that scenario, Filebeat was not correctly acking these events to some wait group. The ACKer was modified to handle both stateless (default) and stateful events, when stateful is required, the states are sent to the registry otherwise, the waiting groups are directly updated. Fixes: elastic#7202
tsg
pushed a commit
that referenced
this issue
Jun 4, 2018
#7214) * Filebeat: Allow stateless and stateful ACKer on the global ack handler This commit introduces a change in how filebat handle ACK by default, before the ACK was using the private field of the event to retrieve a state. The updated state was sent to the registrar, and the registrar was finalizing the ACK. But with the introduction of the TCP/UDP and the Redis input, the events don't have any state attached. So in that scenario, Filebeat was not correctly acking these events to some wait group. The ACKer was modified to handle both stateless (default) and stateful events, when stateful is required, the states are sent to the registry otherwise, the waiting groups are directly updated. Fixes: #7202 * changelog
tsg
pushed a commit
to tsg/beats
that referenced
this issue
Jun 5, 2018
elastic#7214) * Filebeat: Allow stateless and stateful ACKer on the global ack handler This commit introduces a change in how filebat handle ACK by default, before the ACK was using the private field of the event to retrieve a state. The updated state was sent to the registrar, and the registrar was finalizing the ACK. But with the introduction of the TCP/UDP and the Redis input, the events don't have any state attached. So in that scenario, Filebeat was not correctly acking these events to some wait group. The ACKer was modified to handle both stateless (default) and stateful events, when stateful is required, the states are sent to the registry otherwise, the waiting groups are directly updated. Fixes: elastic#7202 * changelog (cherry picked from commit b9d2150)
urso
pushed a commit
that referenced
this issue
Jun 5, 2018
#7214) (#7258) * Filebeat: Allow stateless and stateful ACKer on the global ack handler This commit introduces a change in how filebat handle ACK by default, before the ACK was using the private field of the event to retrieve a state. The updated state was sent to the registrar, and the registrar was finalizing the ACK. But with the introduction of the TCP/UDP and the Redis input, the events don't have any state attached. So in that scenario, Filebeat was not correctly acking these events to some wait group. The ACKer was modified to handle both stateless (default) and stateful events, when stateful is required, the states are sent to the registry otherwise, the waiting groups are directly updated. Fixes: #7202 * changelog (cherry picked from commit b9d2150)
leweafan
pushed a commit
to leweafan/beats
that referenced
this issue
Apr 28, 2023
elastic#7214) (elastic#7258) * Filebeat: Allow stateless and stateful ACKer on the global ack handler This commit introduces a change in how filebat handle ACK by default, before the ACK was using the private field of the event to retrieve a state. The updated state was sent to the registrar, and the registrar was finalizing the ACK. But with the introduction of the TCP/UDP and the Redis input, the events don't have any state attached. So in that scenario, Filebeat was not correctly acking these events to some wait group. The ACKer was modified to handle both stateless (default) and stateful events, when stateful is required, the states are sent to the registry otherwise, the waiting groups are directly updated. Fixes: elastic#7202 * changelog (cherry picked from commit 31c46c9)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
User reports a panic ("sync: negative WaitGroup counter") after a week of running filebeat. filebeat stays running but does not accept the connections.
Stacktrace:
Log (two consecutive panics, same stracktrace):
For confirmed bugs, please report:
The text was updated successfully, but these errors were encountered: