First draft of metricbeat docs

* Basic overview with configuration
* Module and metricset structure
* Contributor guide
* Add TODO notes

metricbeat/Makefile

@@ -19,7 +19,16 @@ fields:
 	cat etc/fields_base.yml > etc/fields.yml
 	cat module/*/*/fields.yml >> etc/fields.yml
 
+# Collects all module docs
+.PHONY: docs
+docs:
+	cp -r module/*/*.asciidoc docs/modules
+
+
 # This is called by the beats packer before building starts
 .PHONY: before-build
 before-build:
 
+# Runs all collection steps and updates afterwards
+.PHONY: collect
+collect: kibana fields docs update

metricbeat/TODO.md

@@ -1,12 +1,31 @@
-List of things to keep track on:
+List of things to keep track on, thoughts on what should be improved. More can also be found in the Github issue under https://github.com/elastic/beats/issues/619
+
+
+# Kibana Dashboards
+Each metricset and module should have its dashboard. The basic scripts to create and aggregate dashboards for metricsets and modules are in place. The current basic dashboards have to be extended with additional visualizations and cleaned up.
+
+The following points are open question:
+* Should there be one module per metricset or per module or both?
+* Storage of visualizations only for metricsets or modules should be possible. The reasoning is that if metricbeat is used as a library to add a metricset, it should be possible to also have a dashboard inside (self contained)
+
+# Mapping
+Scripts to generate the template from fields.yml are implemented. It must be checked in details what additional default mapping should be introduced for metricbeat. Some ideas:
 
-* Create dashboard for each implemented metricset
-* send also document if remote system is not reachable -> error document
-  * What happens on error of fetch?
 * disable _all for all fields
-* disable _source? for all fields
-* Filters on metricset level
-  * Idea: Minimum, Basic, Full data to have predefined sets / levels
-  * Or is this done via filter implementation
-* Add fields per module / metricsets
-* Add templates
+* disable _source for all fields (use doc values instead)
+
+The mapping for each metricset must be completed and verified
+
+# Filtering
+A more generic filtering should be introduced so not all data is sent by default. In general I think every metricset should be able to provide the full data set, but by default it should only send a reasonable amount of data.
+
+Filtering can happen through generic filtering but a less complex options with levels or something similar would be nice to have. One idea would be to have 3 levels on the data side: Minimum, Basic, Full. Each metricset would have to support that and the level would be configurable. An alternative is that each metricset has to implement its own configuration on how to handle which parts are enabled.
+
+Generic filtering support must be added to the module.
+
+# Topbeat
+Topbeat should be added to metricbeat (see https://github.com/elastic/beats/pull/1081). To make the two better integrated some refactoring on the Topbeat side is needed so MapStr can be consumed directly.
+
+# More
+* Add service host as default event informartion. See https://github.com/elastic/beats/issues/619#issuecomment-185242407
+* Add version number of service. See https://github.com/elastic/beats/issues/619#issuecomment-185242407

metricbeat/docs/configuration.asciidoc

@@ -0,0 +1,71 @@
+= Metricbeat configuration
+
+This page gives an overview on how metricbeat has to be configured.
+
+== Basic Configuration
+
+A Metricbeat configuration contains a list of modules. Each module contains the following config options:
+
+* module: Name of the module to run
+* metricsets: List of metricsets of the module which should be executed
+* hosts: List of hosts that the metricsets should connected to. For some modules this is not required (e.g. system). For the details how to define the hosts string for each module, please check the documentation of the module.
+* period: Period how often the metricsets are executed. If a system is not reachable it will return an error for each period.
+* enabled: Enabled is by default set to true and is not required. It can be set to false to disable a module
+* selectors: Selectors is a list of string that can be defined to reduce the amount of data sent. For the available selectors for each metricset, check the module / metricset documentation.
+* fields: Fields is optional. Here you can define additional fields which should be sent with the metricset event
+* tags: Tags is optional. Here you can define additional tags which should be sent with the metricset event
+* filters: With filters you can reduce the amount of data is sent. For details on the available filter options check (link to filters).
+
+
+```
+metricbeat:
+  modules:
+    - module: apache
+      metricsets: ["status"]
+      hosts: ["http://127.0.0.1/"]
+      period: 10s
+      enabled: true
+      selectors: []
+      fields:
+        dc: west
+      tags: ["tag"]
+      filters:
+        ....
+```
+
+== Configuration Combinations
+
+This configuration allows to create different combinations of module, metricsets, period, hosts and selectors. In the example below, the same hosts is crawled for the `stats` information every second as this is critical data, and the full list of metricsets is only fetched every 30 seconds as it is less critical.
+
+
+```
+metricbeat:
+  modules:
+    - module: redis
+      metricsets: ["info"]
+      hosts: ["host1"]
+      period: 1s
+      enabled: true
+      selectors: ["stats"]
+    - module: apache
+      metricsets: ["info"]
+      hosts: ["host1"]
+      period: 30s
+      enabled: true
+      selectors: []
+```
+
+For a module with multiple metricsets its also possible to run define the module twice and run it in once case with both metricsets and in the second case only with one metricset with a different period:
+
+```
+metricbeat:
+  modules:
+    - module: example
+      metricsets: ["set1"]
+      hosts: ["host1"]
+      period: 10s
+    - module: example
+      metricsets: ["set2"]
+      hosts: ["host1"]
+      period: 2m
+```