Add more outputs than Elasticsearch and Logstash

[monicasarbu]

Currently you can configure in the Beats the following as outputs (in chronological order):

• Elasticsearch
• Redis
• Logstash

Once we added support for Logstash, we deprecated Redis as you can use the redis output plugin to export data to Redis.
For someone that wants to integrate the Elastic Stack into its own system and make use of a queuing system before inserting data to Elasticsearch, the deployment looks like this:

Beats -> Logstash -> Redis/Kafka -> Logstash -> Elasticsearch

We received many requests from the community to add support for more outputs, besides Elasticsearch and Logstash to avoid having an additional Logstash instance to translate the data to the desired queuing system. Here are the links to the GitHub issues:

• Redis https://github.com/elastic/filebeat/issues/132
• Kafka https://github.com/elastic/libbeat/pull/21
• Rabbitmq: https://github.com/elastic/libbeat/issues/313
• ActiveMQ https://discuss.elastic.co/t/beat-support-for-message-queue-output/39567

After internal discussions, we decided to follow the community feedback, so we will un-deprecate the Redis output and add support for Kafka output.

With these improvements, the deployments are simplified:

Beats -> Redis/Kafka -> Logstash -> Elasticsearch

TODOs:

• Undeprecate Redis output
• Update Redis output to support the same option as the Logstash and Elasticsearch outputs Update redis client #1353
• Implement Kafka output: Proposal: kafka output #942

We will continue to be very selective in terms of what outputs we support. For the moment, we are only considering Redis and Kafka based on their popularity.

[adiworkoholic]

So... is the support for a rabbitMQ output off the table :(

[ruflin]

@adiworkoholic Our current focus/priority is on implementing Redis and Kafka. Best is to subscribe to the issue https://github.com/elastic/libbeat/issues/313 as we will post an update there in case we add further outputs.

[brandbintley]

Kafka support is awesome. How do we see handling security? If I got it right, libbeat supports TLS while Kafka supports SSL and SASL.

[froztbyte]

I'm also in favour of a RabbitMQ output.

My previous model for this was beaver[0](with a sincedb configured) connecting to rabbitmq, logstash reading off of logstash then outputting to elasticsearch and others. The benefits of this design is that it gives quite a strong ability to deal with various failures or operational necessities. On a future model of Filebeats and friends, I would ideally like to be able to reuse this same design, or a design with its strengths

• [0] The one downside with beaver was that it was synchronous by default. That wasn't great.

[monicasarbu]

Closing the issue as all the tasks were finished.