Metricbeat

[ruflin]

Metricbeat is a beat that allows to fetch metrics and status informations from various services on a predefined schedule. Metricbeat is the foundation that should make the addition of further metrics even simpler then to create a new beat.

This is to describe the Metricbeat structure in more details and track the open issues for a first stable release.

Todo

General

• Create Prototype Metricbeat Prototype #611
• Create dashboards for each module and metricset
• Create basic index patterns and additional patterns per module if needed
• Send empty metric when service is not reachable with special message / field

Features

• Report host / source as part of every event to differentiate between events Add Host information to each metricset event #1185
• Implement dynamic config based on ucfg MetricBeat support for additional configs #1116

Docs

• Getting Started Metricbeat docs and Developer Guide #1108
• Configuration Options Metricbeat docs and Developer Guide #1108
• Developer Guide Metricbeat docs and Developer Guide #1108
• Add metricbeat to the document building

Testing / packaging

• Update beats-packer with metricbeat
• Integration test framework
• System test framework
• Unit Tests
• System Tests
• Add Metricbeat to beat tester
• Add appveyor build Add Metricbeat to Appveyor build #1766
• Add metricbeat to nightly builds

Initial Modules and Metrics

As the initial modules and metrics provided I suggest the following:

• Apache
  • server-status
• Redis
  • Info
• MySQL
  • Version
  • Server Status

Structure

To organise the different metrics which are collected, metricbeat organises metrics as following:

• metricbeat has multiple modules
• modules have multiple metricsets
• each metricsets belongs to a module

Module

A module provides shared libraries and shared configuration for its metricsets. In case of redis this is on the library side the connection to redis and the configurations of the hosts to crawl.

MetricSet

Metricsets fetch the data from the remote host and convert it to a JSON object (MapStr) to be sent to elasticsearch. Each metric can have its own additional configuration. A metric has access to the configuration of the module.

An event provided for redis would for example contain the data contained in the info request: http://redis.io/commands/INFO

[mrkschan]

Would there be a shared library to do the following? E.g.

1. Monitoring multiple metric sources of the same service per go-routine.
2. Parse JSON input and produce libbeat output (golang json stdlib return "float" instead of "int" where libbeat/ES expects int.

[ruflin]

Metricbeat can also be used as library. See example df2beat here: https://github.com/ruflin/df2beat Based on this I would say the answer for (1) is yes. Second part is not covered yet but I think #923 goes in this direction.

[mrkschan]

I would like to bring a few more things to the conversation:

1. The MetricSet output is better to have version number since the metric source would have version as well.
2. If we allow pulling metric from multiple sources of the same Module, the MetricSet output is better to have the source included so that we can slice and dice by source on Kibana.
3. If the metric source returns nested array in the JSON, we better have to break that down into multiple MetricSets since nested array is not compatible with Kibana yet, iirc. (E.g. nginxbeat breaks the nested array into different output document types.)
4. If we put all MetricSets from multiple Modules into the same Elasticsearch index, we may need to provide namespace to MetricSet. E.g. Nginx would have upstream MetricSet while HAproxy would do so as well, how could one easily differentiate the two?
5. Since the metrics obtained from a source is following a fixed JSON schema, shall we provide certain ways to shorten the development time of new module? For example, "JSON-path". Like XPath in the XML context, can we provide a lightweight tree traversal language to extract values from the JSON document into a MetricSet? We may reference the "JSON-path" format from https://github.com/stedolan/jq.

[ruflin]

@mrkschan Thanks a lot for the inputs and sharing your thoughts.

1. What you mean here is that for example for apache / mysql it would report the version number of the service and make this as a requirement for each event?
2. Good point, we must definitively add this to the document. Not sure if we should call it "host"? I added it to the "TODO" list above.
3. We should discuss this further. One thing is to split it up into multiple "logical" metricsets. But this will not fully prevent nested documents I think. We plan together with the Kibana team to make the visualisations of metrics as easy as possible (Timelion also helps here)
4. You can filter by module: https://github.com/elastic/beats/blob/master/metricbeat/beater/metricbeat.go#L8 But not 100% sure if I fully understand the concern here. Can you provide some more details.
5. Definitively. I'm having multiple "helpers" in mind not only for JSON, but also for example to Http request with authentication. We should do this as soon as we see common patterns and abstract it out.

[mrkschan]

Re 1: Nginx has a version for the metric schema in addition to the Nginx version 1. The service version may help in slice and dice but that is just a different thing compared to schema version.

Re 3: Good to hear the Kibana team is involved :) Though, do we need to update the documentation generator as I also agree that not all nested array can be a separate metric set?

Re 4: That's exactly what I think we need in each metric set. Sorry for not spotting that in the code.

[ruflin]

1. We will have to version our schema templates in the future. As we currently push all in one type the question is if schema updates to any module will update the version number for all schemas or if it is related to the module or even metricset
2. I think there will be adjustments needed to the template and doc generator for metricbeat as much more different fields exist.
3. Great to hear

[njam]

Very interesting!
We're maintaining a collection of metrics collectors for various services in Ruby: bipbip. I would be interested in using Beats and sending this data to "ELK".

How about having a beat that can be easily used in existing applications, either by?:

• Using a CLI tool to submit data
• Running an HTTP-server where we can POST records to

Sorry if this is off topic ;)

[andrewkroh]

@njam What about using logstash-input-http?

[njam]

ouch, thanks!

[ruflin]

@njam Also have a look at this issue here: #536

[ruflin]

@mrkschan Quite a lot happened since our last conversation in this issue. Source reporting for each event was added and lots of other small features were added. What isn't in yet so far is versioning of schema and templates. This is something that probably not only affects metricbeat, but all beats.

[ruflin]

Closing this release as all steps for a first release have been completed. All further or incomplete steps are tracked by separate github issues which are linked above.