Jolokia Module with dynamic JMX Metricset (elastic#3570)

This is the implementation of a module for Jolokia which contains a dynamic jmx metricset.

An example configuration looks as following:
```
- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["localhost:8778"]
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
      attributes:
        - attr: CollectionTime
          field: gc.cms_collection_time
        - attr: CollectionCount
          field: gc.cms_collection_count
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage
```

For each mbeat the attributes which should be fetched can be defined. The field defines under which field name the event will be put. The namespace defines the metricset namespace.

This PR replaces elastic#3051

Further changes:
* Added support for method and body to http helper
* Handle empty fields in generators. This happens for a module which only contains dynamic metricsets which is currently the case for jolokia.

TODO:
* [x] Add system tests
* [x] Check documentation
* [x] Add integration test
* [ ] Open issue for metricset which contains basic memory info

CHANGELOG.asciidoc

@@ -143,6 +143,7 @@ https://github.com/elastic/beats/compare/v5.1.2...v5.2.0[View commits]
 - Experimental Prometheus module. {pull}3202[3202]
 - Add system socket module that reports all TCP sockets. {pull}3246[3246]
 - Kafka consumer groups metricset. {pull}3240[3240]
+- Add jolokia module with dynamic jmx metricset. {pull}3570[3570]
 
 *Winlogbeat*
 

libbeat/scripts/generate_index_pattern.py

@@ -18,6 +18,10 @@
 
 def fields_to_json(section, path, output):
 
+    # Need in case there are no fields
+    if section["fields"] is None:
+        section["fields"] = {}
+
     for field in section["fields"]:
         if path == "":
             newpath = field["name"]

libbeat/scripts/generate_template.py

@@ -128,6 +128,10 @@ def dedot(group):
     fields = []
     dedotted = {}
 
+    # Need in case there are no fields
+    if group["fields"] is None:
+        group["fields"] = {}
+
     for field in group["fields"]:
         if "." in field["name"]:
             # dedot

libbeat/tests/system/beat/beat.py

@@ -387,6 +387,10 @@ def load_fields(self, fields_doc="../../_meta/fields.generated.yml"):
         def extract_fields(doc_list, name):
             fields = []
             dictfields = []
+
+            if doc_list is None:
+                return fields, dictfields
+
             for field in doc_list:
 
                 # Chain together names

metricbeat/_meta/beat.full.yml

@@ -102,6 +102,18 @@ metricbeat.modules:
   #period: 10s
   #hosts: ["tcp://127.0.0.1:14567"]
 
+#------------------------------- Jolokia Module ------------------------------
+#- module: jolokia
+#  metricsets: ["jmx"]
+#  enabled: true
+#  period: 10s
+#  hosts: ["localhost"]
+#  namespace: "metrics"
+#  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
+#  jmx.mapping:
+#  jmx.application:
+#  jmx.instance:
+
 #-------------------------------- kafka Module -------------------------------
 #- module: kafka
   #metricsets: ["partition"]

metricbeat/docker-compose.yml

@@ -7,6 +7,7 @@ services:
       - couchbase
       - mongodb
       - haproxy
+      - jolokia
       - kafka
       - mysql
       - nginx
@@ -52,6 +53,19 @@ services:
   # Overloading kibana with a simple image as it is not needed here
   kibana:
     image: alpine:latest
+    env_file:
+      - ${PWD}/module/apache/_meta/env
+      - ${PWD}/module/couchbase/_meta/env
+      - ${PWD}/module/haproxy/_meta/env
+      - ${PWD}/module/jolokia/_meta/env
+      - ${PWD}/module/kafka/_meta/env
+      - ${PWD}/module/mongodb/_meta/env
+      - ${PWD}/module/mysql/_meta/env
+      - ${PWD}/module/nginx/_meta/env
+      - ${PWD}/module/postgresql/_meta/env
+      - ${PWD}/module/prometheus/_meta/env
+      - ${PWD}/module/redis/_meta/env
+      - ${PWD}/module/zookeeper/_meta/env
 
   # Modules
   apache:
@@ -60,6 +74,12 @@ services:
   couchbase:
     build: ${PWD}/module/couchbase/_meta
 
+  haproxy:
+    build: ${PWD}/module/haproxy/_meta
+
+  jolokia:
+    build: ${PWD}/module/jolokia/_meta
+
   kafka:
     image: spotify/kafka
     expose:

metricbeat/docs/fields.asciidoc

@@ -19,6 +19,7 @@ grouped in the following categories:
 * <<exported-fields-couchbase>>
 * <<exported-fields-docker>>
 * <<exported-fields-haproxy>>
+* <<exported-fields-jolokia>>
 * <<exported-fields-kafka>>
 * <<exported-fields-mongodb>>
 * <<exported-fields-mysql>>
@@ -2387,6 +2388,20 @@ type: integer
 The average queue time in ms over the last 1024 requests.
 
 
+[[exported-fields-jolokia]]
+== Jolokia Fields
+
+[]beta
+Jolokia Module
+
+
+
+[float]
+== jolokia Fields
+
+jolokia contains metrics exposed via jolokia agent
+
+
 [[exported-fields-kafka]]
 == kafka Fields
 

metricbeat/docs/modules/jolokia.asciidoc

@@ -0,0 +1,43 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-module-jolokia]]
+== Jolokia Module
+
+beta[]
+
+This is the Jolokia Module.
+
+
+
+[float]
+=== Example Configuration
+
+The Jolokia module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+#- module: jolokia
+#  metricsets: ["jmx"]
+#  enabled: true
+#  period: 10s
+#  hosts: ["localhost"]
+#  namespace: "metrics"
+#  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
+#  jmx.mapping:
+#  jmx.application:
+#  jmx.instance:
+----
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-jolokia-jmx,jmx>>
+
+include::jolokia/jmx.asciidoc[]
+

metricbeat/docs/modules/jolokia/jmx.asciidoc

@@ -0,0 +1,19 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-jolokia-jmx]]
+include::../../../module/jolokia/jmx/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-jolokia,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../module/jolokia/jmx/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -6,6 +6,7 @@ This file is generated! See scripts/docs_collector.py
   * <<metricbeat-module-couchbase,Couchbase>>
   * <<metricbeat-module-docker,Docker>>
   * <<metricbeat-module-haproxy,HAProxy>>
+  * <<metricbeat-module-jolokia,Jolokia>>
   * <<metricbeat-module-kafka,kafka>>
   * <<metricbeat-module-mongodb,MongoDB>>
   * <<metricbeat-module-mysql,MySQL>>
@@ -23,6 +24,7 @@ include::modules/apache.asciidoc[]
 include::modules/couchbase.asciidoc[]
 include::modules/docker.asciidoc[]
 include::modules/haproxy.asciidoc[]
+include::modules/jolokia.asciidoc[]
 include::modules/kafka.asciidoc[]
 include::modules/mongodb.asciidoc[]
 include::modules/mysql.asciidoc[]

metricbeat/helper/http.go

@@ -0,0 +1,116 @@
+package helper
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+
+	"github.com/elastic/beats/metricbeat/mb"
+)
+
+type HTTP struct {
+	base    mb.BaseMetricSet
+	client  *http.Client // HTTP client that is reused across requests.
+	headers map[string]string
+	method  string
+	body    []byte
+}
+
+// NewHTTP creates new http helper
+func NewHTTP(base mb.BaseMetricSet) *HTTP {
+	return &HTTP{
+		base:    base,
+		client:  &http.Client{Timeout: base.Module().Config().Timeout},
+		headers: map[string]string{},
+		method:  "GET",
+		body:    nil,
+	}
+}
+
+// FetchResponse fetches a response for the http metricset.
+// It's important that resp.Body has to be closed if this method is used. Before using this method
+// check if one of the other Fetch* methods could be used as they ensure that the Body is properly closed.
+func (h *HTTP) FetchResponse() (*http.Response, error) {
+
+	// Create a fresh reader every time
+	var reader io.Reader
+	if h.body != nil {
+		reader = bytes.NewReader(h.body)
+	}
+
+	req, err := http.NewRequest(h.method, h.base.HostData().SanitizedURI, reader)
+	if h.base.HostData().User != "" || h.base.HostData().Password != "" {
+		req.SetBasicAuth(h.base.HostData().User, h.base.HostData().Password)
+	}
+
+	for k, v := range h.headers {
+		req.Header.Set(k, v)
+	}
+
+	resp, err := h.client.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("error making http request: %v", err)
+	}
+
+	return resp, nil
+}
+
+func (h *HTTP) SetHeader(key, value string) {
+	h.headers[key] = value
+}
+
+func (h *HTTP) SetMethod(method string) {
+	h.method = method
+}
+
+func (h *HTTP) SetBody(body []byte) {
+	h.body = body
+}
+
+// FetchContent makes an HTTP request to the configured url and returns the body content.
+func (h *HTTP) FetchContent() ([]byte, error) {
+	resp, err := h.FetchResponse()
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != 200 {
+		return nil, fmt.Errorf("HTTP error %d in %s: %s", resp.StatusCode, h.base.Name(), resp.Status)
+	}
+
+	return ioutil.ReadAll(resp.Body)
+}
+
+// FetchScanner returns a Scanner for the content.
+func (h *HTTP) FetchScanner() (*bufio.Scanner, error) {
+	content, err := h.FetchContent()
+	if err != nil {
+		return nil, err
+	}
+
+	return bufio.NewScanner(bytes.NewReader(content)), nil
+}
+
+// FetchJSON makes an HTTP request to the configured url and returns the JSON content.
+// This only works if the JSON output needed is in map[string]interface format.
+func (h *HTTP) FetchJSON() (map[string]interface{}, error) {
+
+	body, err := h.FetchContent()
+	if err != nil {
+		return nil, err
+	}
+
+	var data map[string]interface{}
+
+	err = json.Unmarshal(body, &data)
+	if err != nil {
+		return nil, err
+	}
+
+	return data, nil
+}

metricbeat/include/list.go

@@ -24,6 +24,8 @@ import (
 	_ "github.com/elastic/beats/metricbeat/module/haproxy"
 	_ "github.com/elastic/beats/metricbeat/module/haproxy/info"
 	_ "github.com/elastic/beats/metricbeat/module/haproxy/stat"
+	_ "github.com/elastic/beats/metricbeat/module/jolokia"
+	_ "github.com/elastic/beats/metricbeat/module/jolokia/jmx"
 	_ "github.com/elastic/beats/metricbeat/module/kafka"
 	_ "github.com/elastic/beats/metricbeat/module/kafka/consumergroup"
 	_ "github.com/elastic/beats/metricbeat/module/kafka/partition"

metricbeat/metricbeat.full.yml

@@ -102,6 +102,18 @@ metricbeat.modules:
   #period: 10s
   #hosts: ["tcp://127.0.0.1:14567"]
 
+#------------------------------- Jolokia Module ------------------------------
+#- module: jolokia
+#  metricsets: ["jmx"]
+#  enabled: true
+#  period: 10s
+#  hosts: ["localhost"]
+#  namespace: "metrics"
+#  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
+#  jmx.mapping:
+#  jmx.application:
+#  jmx.instance:
+
 #-------------------------------- kafka Module -------------------------------
 #- module: kafka
   #metricsets: ["partition"]

metricbeat/module/jolokia/_meta/Dockerfile

@@ -0,0 +1,10 @@
+# Tomcat is started to fetch Jolokia metrics from it
+FROM jolokia/java-jolokia:7
+ENV TOMCAT_VERSION 7.0.55
+ENV TC apache-tomcat-${TOMCAT_VERSION}
+
+EXPOSE 8778
+RUN wget http://archive.apache.org/dist/tomcat/tomcat-7/v${TOMCAT_VERSION}/bin/${TC}.tar.gz
+RUN tar xzf ${TC}.tar.gz -C /opt
+
+CMD env CATALINA_OPTS=$(jolokia_opts) /opt/${TC}/bin/catalina.sh run

metricbeat/module/jolokia/_meta/config.yml

@@ -0,0 +1,10 @@
+#- module: jolokia
+#  metricsets: ["jmx"]
+#  enabled: true
+#  period: 10s
+#  hosts: ["localhost"]
+#  namespace: "metrics"
+#  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
+#  jmx.mapping:
+#  jmx.application:
+#  jmx.instance:

metricbeat/module/jolokia/_meta/docs.asciidoc

@@ -0,0 +1,6 @@
+== Jolokia Module
+
+beta[]
+
+This is the Jolokia Module.
+

metricbeat/module/jolokia/_meta/env

@@ -0,0 +1,2 @@
+JOLOKIA_HOST=jolokia
+JOLOKIA_PORT=8778