Docker module improvements

[ruflin]

• Add tests to generate data.json files Docker module cleanup #2665
• Check that config files are according to standard (experimental disabled by default) Docker module cleanup #2665
• Review all docker fields for the naming convention Docker module cleanup #2665
• Add CHANGELOG update Docker module cleanup part 2 #2674
• Do manual tests
• Complete fields.yml with all fields Docker module cleanup part 2 #2674
• Add system tests to verify fields.yml Add system tests to check document fields for docker module #2971
• Add more tests
• Add MetricSet for /info: Add info Metricset to Docker module #2672
• Run golint ./... | grep -v vendor and check errors for docker module
• Add dashboard Add sample Docker dashboards #2943.
• Add socket info under host if used: Docker module cleanup #2665 (comment)
• Change and test labels to be taken with . if possible: Docker module cleanup #2665 (comment)
• Write migration guide from dockbeat
• How to handle ports and labels list in container metric
• Limit number of digits for percentage value. Use same logic as in system/cpu metricset
• Fetching CPU data takes > 2s because of average calculation on docker side -> API Problem
• Clean up ssl handling / naming
• Update dashboards

[tsg]

I played with creating a dashboard for a demo and I have a couple of observations/suggestions. Don't take this as criticism, please, I could create some pretty cool dashboards and the demo is going to be great!

On CPU stats:

• Looks like metrics like CPU time are derivates. That's quite convenient for my demo :), but goes against the principle that Metricbeat does total values by default. We should consider indicating somehow in the name that this is a derivate.
• For the CPU time ahead, if I interpret the code correctly, it's in seconds. We should indicate that in the metrics names.
• the docker stats command displays the CPU time as a percentage. Digging up in the code, it seems that it's calculated this way (note the different logic on Unix vs Windows). I think it would be worth exporting this value as well, since it's not trivial to compute via scripts and it's probably what most people are looking for. The system_cpu_usage, which is required to compute the percentage is returned by the API but it's not used in Metricbeat or exported (or I missed it).

On labels:

• Currently exported as an array that has key and value as keys. We could make a lot better use of these in Kibana if they would be exported as a dictionary (simple key->value). It seems that the labels already don't have dots in it, so that's helpful.

On disk/block IO and net IO:

• On my system I only got zero values disk/blog IO, but docker stats shows values in the MB ranges. I'm not sure if it's broken or just a derivative that happens to be always 0. docker stats displays both disk/block IO and net IO as cumulative values, so we should probably just do the same.

[tsg]

I also posted #2944 for a termination issue.

[rikatz]

Small question: Some fields (like labels and ports) came as a Nested array of JSON inside the field. Isn't easier (or at least, better) to convert those nested arrays to new fields?

Explaining, I'm starting to use beats with Kubernetes, to build some dashboards per namespace. Namespaces (and POD names) are contained inside Docker Labels. It's not trivial to use those fields as they are JSON and so, although it's if they were just simple ES fields.

Thanks 😄

[ruflin]

@rikatz This is one of the issues on our list to solve. What would be your preferred format you would like to have it? I'm especially curious for ports as this is a more complex field with nested documents.

[rikatz]

@ruflin Actually, as my primary concern is about the labels, haven't really checked the ports field.

Gonna verify tomorrow this. My suggestion is to set each label as a new field (as docker.io.kubernetes.pod.namespace, as an example), but this would work only on ES 5 and later, as ES2 doesn't support dotted field names.

This would help to aggregate and graph based on namespace name, pod name and so :)

Thanks for your time @ruflin.

[ruflin]

The above would work on version < 2.4 also in case we take the "key" and create a json document out of it before sending it, as this is kind of what elasticsearch will do.

[rikatz]

@ruflin I did some modifications on the code to the 'labels' issue, but i'm not converting the fields name containing '.' to '_', assuming this will be used only on ES > 5. Checking other fields, they contain '.' on the field name also (like docker.container.image), so I assume this is a reasonable field name for the labels also.

Will open a PR today, so you guys can review this 😄

About the 'ports' problem, I can't figure how to solve this, as it's not a map based on key/value, but a map inside a map.

Do you have any suggestion on this?

[ruflin]

For the ports I was thinking if we could use one of the fields as identifier which at least will make accessing the fields easier. Assuming someone is interested in search for the port information, which info will he filter on? Ingoing / outgoing / ip?

Assuming it is the private port and the private port is unique, we could do the following for the json example below:

Current structure

 "ports": [
        {
          "ip": "",
          "port": {
            "private": 9300,
            "public": 0
          },
          "type": "tcp"
        },
        {
          "ip": "",
          "port": {
            "private": 9200,
            "public": 0
          },
          "type": "tcp"
        }
      ],

Port based:

docker.container.port.9300.ip:
docker.container.port.9300.port.public:
docker.container.port.9200.ip:

But that is far from public. So far our approach was when we had such "arrays" is that we introduced a separate metricset. We could do the same here and introduce a port metricset which only contains the port info so it can be queried if needed.

@rikatz It would be interesting to hear how you would like to use the data when it is in elasticsearch. That could give us some hints on how to store it. One use case I have in mind is having alerts in watcher that send a notification as soon as a container with a specific port is open.

[rikatz]

In my opinion, ports only add value if you're looking who is the owner of a specific public port, on a cluster of docker hosts.

I don't know, as an example, how swarm deals with this with services. I know that kubernetes uses kube-proxy, so on kubernetes this information is not useful at all.

As a suggestion, I would export this info only if the container have a public port, like this:

docker.container.publicport.tcp8080 {
   privateport: 80,
   publicip: 192.168.15.23
}

But this is my opinion, and only if you think searching ports is important. The private ports I would keep as that array of jsons :)

@ruflin about watcher, I think this would be great, but also this is more related to public ports than private, right?

[ruflin]

@rikatz Based on your inputs I think we can actually remove ports for the moment from the events and deal with it when requests for it pop up. Then we have hopefully also a very specific use case.

Agree on the public ports. The part I don't like too much in the above json example is what we mix port and "type" which is tcp here. I see you did it to prevent conflicts in case 8080 exist over udp and tcp (I assume). At the moment I'm very tempted to say this is a separate metricset to monitor ports, so info would be under docker.port.*.

[ruflin]

PR for removing ports: #3040

[rikatz]

@ruflin Exactly, did that because of the protocol and conflicts. But also agree with you about separating this metric to other kind of metricset.

With some more tests and also with more usage of my environment, if I see some new need to ports, I can try to propose a new approach!

[ruflin]

@rikatz Nice. I assume that means you are actively using our snapshot build with the docker module. Keep us posted if you hit any issues so we can iron out all the edges.

[rikatz]

@ruflin Actually I'm working to replace Telegraf to beats, as I need those informations on ES and Influx haven't accepted our ES output module (and also because Beats works nicelly with stats!)

But I'll keep you updated about them, opening issues and trying to correct them also :)

[rikatz]

Guys, I'm testing somethings here, aparently the Total CPU Usage percent expected by Kibana Dashboard is not the same field being generated by Metricbeat.

Will do some more research, but it appears that the name changed to docker.cpu.total.pct

[ruflin]

@rikatz I didn't test this but I assume you are right. I updated the structure of the event but didn't update the dashboards :-( @monicasarbu In case you have some time to take a look ...

[memelet]

Any thoughts to capturing docker events, like OOM events?

[ruflin]

@memelet Interesting idea. Could you open a new github issue for this as a feature request?

[ruflin]

Closing this as we covered almost all the issues in this meta issues.