[WARN] serf: Query queue depth: 3216 messages in the logs
#649
Comments
|
@davidgengenbach I've never observed this on moderate workloads, how many jobs and with what frequency are you running? |
Jobs:
Happens in a single-server configuration. The number somehow stabilized at a queue depth of ~ 4100. As we are evaluating using Dkron for an important part of our infrastructure we are especially keen on having it run perfectly. We might test multi-server configurations but hoped that a single server suffices as we have a microservice interfacing Dkron and do not need that much failure tolerance of Dkron. |
|
That's not too much, check the test instances I have at test.dkron.io:8080 I try to maintain it with a good amount of workload, and doesn't get the queue issue you are experiencing. I think you are experiencing something network related. Could you paste here your |
Your Dkron test setup is not a single-server configuration - maybe it's related to that? Also, I think the messages only show when having
We have a single-server configuration and - I guess - only loopback networking in serf?
server: true
log-level: debug
bootstrap-expect: 1
webhook-headers: 'Content-Type: application/json'
webhook-url: 'http://REDACTED'
webhook-payload: >-
{
"name": "{{.JobName}}",
"success": {{.Success}},
"finishedAt": "{{.FinishedAt}}",
"startedAt": "{{.StartTime}}",
"reportingNode": "{{.ReportingNode}}",
"nodeName": "{{.NodeName}}"
}I will observe the issue further! |
Single server config is actually much simpler than the test instance and should give you better performance because of less network roundtrips. Test instance use debug log level https://github.com/distribworks/dkron/blob/master/scripts/ansible/site.yml#L33
What's the server serf IP address? you can check it in the UI The config is super simple it should work. |
|
Fixed in #749 |
Describe the bug
We get a lot of
serfwarnings, e.g.[WARN] serf: Query queue depth: 3216where the queue depth increases steadily (~ queue depth increases at a rate of 2 per minute).Logs
The complete (redacted) logs are here:
cleaned.txt
To Reproduce
Steps to reproduce the behavior:
Expected behavior
These warnings should not happen?
Specifications:
dkron/dkron:v2Docker imageAdditional context
I found an old StackOverflow post with the same question but not much else.
This is happening a single-server configuration with
--bootstrap-expect 1.Is this something to be worried about or is it normal?
Thousand thanks 👍
The text was updated successfully, but these errors were encountered: