Adds a couchbase module for metricbeat #3081

paschdan · 2016-11-30T11:00:45Z

This module has 3 metricsets to collect metrics from couchbase:

cluster overview metrics
node metrics
bucket metrics

elasticsearch-release · 2016-11-30T11:02:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2016-11-30T11:02:04Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

paschdan · 2016-11-30T16:52:06Z

@ruflin I have a question regarding moving the data conversion to the schema method:
if i have a json input for example this:

{
  "foo": "bar",
  "nested": {
    "foo": "nested_bar"
  }
}

and i want my result to be

{
  "foo": "bar",
  "nested_foo": "nested_bar"
}

is this currently anyhow possible with schema / mapstrstr / mapstriface ?

ruflin · 2016-12-06T08:38:18Z

@paschdan I remember we had recently the same question popping up but I don't remember if we found or implemented a solution for it. Perhaps @andrewkroh can help?

ruflin · 2016-12-06T08:41:13Z

@paschdan Thanks a lot for contributing the couchbase module directly with 3 metricsets 🎉 Let me know when it is ready for a first review. For the field names, check out our naming conventions: https://www.elastic.co/guide/en/beats/libbeat/current/event-conventions.html

paschdan · 2016-12-08T10:06:04Z

@ruflin i guess its ready for a review

ruflin

What a great and high quality contribution. I really appreciate all the work you put into this.

I left you a few minor comments. Most of them are related to naming. Please object if you do not agree with the suggestions as the naming of the fields is really hard and people who know the system are best to judge what fits and what not.

For the first release we should mark the module experimental. The reason we do this is so we can still change the naming of the fields or some structure after feedback from customers. Have a look at the docker module on how we added the experimental flag there.

Please have a look at the comments I made and let me know, if you want to adjust them directly in this PR or if we should merge it in as it is and do some cleanup in a second step.

ruflin · 2016-12-08T12:42:30Z

metricbeat/docker-entrypoint.sh

@@ -30,4 +30,5 @@ waitFor ${NGINX_HOST} ${NGINX_PORT} Nginx
 waitFor ${POSTGRESQL_HOST} ${POSTGRESQL_PORT} Postgresql
 waitFor ${REDIS_HOST} ${REDIS_PORT} Redis
 waitFor ${ZOOKEEPER_HOST} ${ZOOKEEPER_PORT} Zookeeper
+waitFor ${COUCHBASE_HOST} ${COUCHBASE_PORT} Couchbase


I tried to sort these alphabetical :-)

ruflin · 2016-12-08T12:44:14Z

metricbeat/module/couchbase/_meta/Dockerfile

@@ -0,0 +1,5 @@
+FROM couchbase:latest


Could we fix a version here? This will make the tests more predictable and we know exactly which version was tested when.

ruflin · 2016-12-08T12:45:43Z

metricbeat/module/couchbase/_meta/configure-node.sh

+waitForCouchbase
+
+# Setup index and memory quota
+curl -v -X POST http://127.0.0.1:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300


I kind of expected this to be in a config file. But I'm not really familiar with couchbase.

haha yes, me too, but unfortunately it is not possible by config, i could only set specific initial params via cli instead of rest, but loading a sample database is not possible via cli, as far as i know

ruflin · 2016-12-08T12:46:10Z

metricbeat/module/couchbase/_meta/configure-node.sh

+curl -v http://127.0.0.1:8091/settings/web -d port=8091 -d username=Administrator -d password=password
+
+# Load travel-sample bucket
+curl -v -u Administrator:password -X POST http://127.0.0.1:8091/sampleBuckets/install -d '["beer-sample"]'


We should probably move this content loading into the tests where it is needed.

ruflin · 2016-12-08T12:49:25Z

metricbeat/module/couchbase/bucket/_meta/fields.yml

+    - name: quota.raw_ram
+      type: long
+      description: >
+        Amount of raw RAM used by the bucket (bytes)


What is the different between raw and not raw?
For bytes you can add format: bytes so Kibana will recognise that it is bytes and will do conversions. In addition we try to have the unit always at the end, so it would be quote.ram.bytes

so this would be correct:

- name: quota.ram.bytes format: bytes type: long description > Amount of RAM used by the bucket

?

Yes, except that it seems format etc. has one space too much. It should be directly under name (probably copy / paste problem)

write on the fly problem ;)

ruflin · 2016-12-08T13:01:33Z