Additional Metricbeat Modules and Metricsets

[ruflin]

This issue is intended to discuss new modules and metricsets for Metricbeat to decide on the priorities of the next modules on metricsets. This list is only a list of ideas and does not mean they will be implemented. Please comment with new modules below, I will update the list here. As soon as the implementation of a module is started, a separate issue is created and linked here.

Suggestion for additonal modules:

• Docker Docker module improvements #2629
• Kafka
• Haproxy Adding php-fpm module to metricbeat #2247
• Jenkins
• MongoDB
• Nagios
• Network
  • TX/RX
• Nginx
• Php-fpm Adding php-fpm module to metricbeat #2247
• PostgreSQL
• Redis
• System
  • Uptime + OS Details
  • Disk IO
• Uwsgi
• Zookeeper

[monicasarbu]

I would concentrate in adding more diverse systems than adding more types of databases in order to make Metricbeat be used for collecting different kinds of data.

If we are looking at the community beats, there are a lot of valuable Beats that we could consider merging to Metricbeat like:

• nagios from nagioscheckbeat (update: it exports few metrics, but the use case is different -> deleting it from here)
• redis from redisbeat
• docker from dockerbeat -> only to get the health metrics of the docker container
• nginx from nginxbeat
• php-fpm from phpfpmbeat -> to be discussed, maybe later

[ruflin]

+1 on docker and nginx. I think nagioscheckbeat is somehow different as it does not monitor nagios itself, but oter services.

[mrkschan]

I thought there's a community metricbeat, where could i get started?

[monicasarbu]

There is no community Metricbeat. If you would like to contribute to Metricbeat there are two options:

• add new module to Metricbeat by sending us the pull request to elastic/beats
• use Metricbeat as a library and create a new standalone Beat on top of it, that is available in your own GitHub repository
  All the Metricbeat modules are supported by elastic. The second option is useful in case you want to monitor a proprietary REST-API. In this case, you can create a Beat with a single module that uses Metricbeat as a library.

[ruflin]

@mrkschan To answer your question in more detail and on how your get started I created an initial draft of the developer guide here: https://github.com/ruflin/beats/blob/metricbeat-contributor-guide/metricbeat/docs/developer-guide.asciidoc If you plan to transform nginxbeat to a metricbeat module let me know so I can help you in the process.

[mrkschan]

I definitely would like to move nginxbeat and uwsgibeat to Metricbeat if allowed :) I will go through the guide :)

[ruflin]

@mrkschan That is great :-) We should definitively start with nginxbeat as this is on our short term whishlist. Please be aware that Metricbeat is still under heavy development so interface still might change. Also hapy to jump on a call to kick this off and answer any questions. Just send me an email.

[mrkschan]

Hey @ruflin, I'm jumping into a nginx module that has a stub status metricset. May I ask if we must only have stateless module?

Nginx only provides the total number of client requests in its stub status 1. To obtain the number of client requests since the last status collection (so to find requests/second), we have to keep a count in memory 2. Could you provide me pointers on how to keep state in memory inside metricbeat module? Shall I simply use module level variable?

[ruflin]

@mrkschan Good to hear. The module itself is mostly stateless, the metricsets aren't. I assume that is what you need. Each Metricset has a MetricSeter object which has two interface functions: Setup and Fetch. You can store and update your state in the MetricSeter object. Have a look at the redis implementation: https://github.com/elastic/beats/blob/master/metricbeat/module/redis/info/info.go

Let me know if this helps or you were looking for something else. Also let me know if you I can help somehow with creating the nginx module.

[ruflin]

Two more modules I would like to the list are:

• Kafka
• Zookeeper

We have a Kafka output in the future versions so it would make sense to monitor it. There is an interesting library from LinkedIn that does some Kafka monitoring: https://github.com/linkedin/Burrow Perhaps this app could be used as a library (would need some refactoring).

As Kafka is normally deployed with Zookeeper, it would be good to also have Zookeeper in. Zookeeper has a relatively simple interface to fetch metrics: https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html#sc_zkCommands

[simianhacker]

Additional metrics...

• TX/RX for network traffic
• Uptime and OS details
• Disk IO

[monicasarbu]

Another option would be to add an OpenBSD module that collects different sensors via sysctl(3). The community beat hwsensorsbeat already has support for hw sensors.

[consulthys]

Not sure if kafkabeat is on the radar as it is not listed on the community beats page.

[ruflin]

@consulthys It is on our radar and the plan is to make a metricbeat module out of it (no pressure @gingerwizard 😇 )

[consulthys]

Awesome, thanks Nicolas!

[sirkjohannsen]

+1 on php-fpm

[ruflin]

@sirkjohannsen See #2247

[medcl]

+1 for cassandra

[sherry-ger]

+1 for kafka

[tanuj83]

+1 for Kafka

[jacqvdm]

+100 for Kafka

[ceeeekay]

+1 for Kafka, especially if it's based on Burrow, now that MaxLag is missing from >=0.0.9

[ruflin]

First version of Kafka module made it into master: #2969

@ceeeekay Can you elaborate on this. Burrow also depends on on Sarama library so there should not be really a difference. Is burrow doing something on top which is not available in Sarama? Can you post a link to MaxLag so we can have a closer look?

[ceeeekay]

@ruflin - I was referring to the way Burrow does lag checking: https://github.com/linkedin/Burrow/wiki/Consumer-Lag-Evaluation-Rules. Getting this type of info from a beat would be great.

My apologies - I meant to say ConsumerLag, not MaxLag. At any rate this is still mentioned in the Kafka docs for 0.10.0 but is missing from the kafka.server JMX domain since 0.9.0. The only way I can get anything like this metric is via the Burrow API, which is a bit top-heavy for what I want to be doing, i.e., getting 4 or so metrics and sending them to a live dashboard.

[tanuj83]

Kafka-manager also provides consumer Lag information
https://github.com/yahoo/kafka-manager for both old-consumer and new-consumer.

[vas78]

@ceeeekay If you`re interested in a metricbeat module that uses Burrow feel free to check out my fork: https://github.com/vas78/beats/tree/kafka-consumer/metricbeat/module/kafka/consumer.

@ruflin I could create a PR for my module, please let me know if you`d like to have something like that in the official beats package.

[ruflin]

@vas78 Definitvely interested in a PR for the metricset. I started some time ago on the consumer part but then we focused on first getting the partition metricset right: #2977 The part I'm still not sure about is why we need burrow in between as burrow also uses Sarama behind the scene, or talks to Zookeeper to get the data? I think I'm missing something here.

[vas78]

@ruflin Regarding Burrow: it is a part of some existing monitoring solution we've used before migrating to elastic/beats. That's why I decided to re-use it as beats input and avoid wiring the whole kafka cluster configuration in the beat module myself:). Besides of that, Burrow has some nice features that are not available in "bare" Sarama like consumer lag evaluation rules, multiple Kafka cluster support and alerting. But I agree that for a long-term solution it would be probably better to avoid any additional proxy and use Sarama to directly connect to a kafka cluster.
For now I`ll stick with my Burrow module for consumer lag monitoring as an intermediate solution and plan to look into the kafka consumer metricset in the nearest future.

[ruflin]

@vas78 Whenever possible we should connect directly to the system without an additional collector in between. But if there are cases we can't cover with beats ourself or it makes life much easier, we could also add a burrow module for example. But I'm quite confident we don't have to ;-)

Talking about additional metrics: It seems there are some more additional metrics available through JMX in Kafka: https://kafka.apache.org/documentation#monitoring So having in #3051 could also bring new possibilities here.

[ruflin]

I'm closing this issue as a long list of modules have been added. Not necessarly all in this issue but I suggest from now on to open a feature request for each module which is required and we can decided base on each module the priorities. Thanks everyone for the contribution here.