Filebeat performance when sending to Logstash

[tsg]

There have been reports that the Filebeat -> Logstash communication doesn't seem to be as efficient as expected. For reference, the reported numbers were 3 K/s events by Filebeat, compared to the TCP input doing 39 K/s or the Logstash-Forwarder doing around 13 K/s (in a report from another user).

I did some tests and could get Filebeat up to 16 K/s when running together with Logstash on the same, relatively powerful, machine (8 CPU threads), by increasing the bulk_max_size option of the Logstash output in Filebeat to 3000.

The bulk_max_size option used to have a default of 10000 and was recently reduced to 200 due to memory issues in Packetbeat when Logstash is not available (it can allocate 1000 x 10k events). The change from 10k to 200 didn't affect the performance of the async publisher in Packetbeat. However, as seen above, it affects the performance of the sync publisher in Filebeat. One possible easy solution would be to have different defaults for the sync and async types, but it seems to me like we need a bit more investigations here to figure out exactly what happens. IMHO, 200 should be enough for good performance, as also confirmed by the async version.

For reference, I used this Logstash config:

input {
    beats {
        port => 5044
    }
}

filter {
    metrics {
        meter => "events"
        add_tag => "metric"
    }
}

output {
    if "metric" in [tags] {
        stdout {
            codec => line {
                format => "Rate: %{[events][rate_1m]}"
            }
        }
    } else {
        null {
            workers => 2
        }
    }
}

And this Filebeat config:

filebeat:
  prospectors:
    -
      paths:
        - /home/tsg/testbed/logs/*
      input_type: log

output:
  logstash:
    hosts: ["localhost:5044"]
    bulk_max_size: 3000

The logs used for testing were downloaded from here.

We should also look into what the bottleneck is besides the bulk_max_size option, because both Filebeat and Logstash didn't seem to fill their whole CPU potential (Filebeat was ~40% CPU usage, LS was at ~150% CPU usage, but looking at the thread view, the largest was ~60%).

[urso]

updated default queue sizes to increase throughput and reduce buffering in libbeat.