Monitoring dynamic microservices by service (service oriented monitoring)

[panda87]

Well I didn't decide yet if this is not a pure issue or an open discussion.

We and I think we are not alone here, starting to use \ checking docker schedulers systems, like dcos, k8s, nomad and so on, and the environment become more and more dynamic.

This means that if I had specific service on specific host, now it changed and on one host we have many services which dynamically created on hosts (scaling), and dynamically removed from hosts in favour of any other services, according the schedulers decisions.

We have many microservices that dynamically creating with random ports so it doesn't let us get metrics for services just by it's ports.

My point is the following
We need to think how we can monitor metrics of specific service like elasticsearch (metrics like number of indices) from a docker container which just created now on host, as well as the basic metrics like the docker plugin has today.

What do you think?

[sparrc]

sounds like a request for service discovery? #272

[panda87]

Yes @sparrc, very close.
With containers integration.
How difficult you think it is?

[sparrc]

to make something that is flexible and fits all use-cases, I would say is very difficult. There are existing models, I think we should more-or-less shoot for doing something similar to prometheus (see https://prometheus.io/blog/2015/06/01/advanced-service-discovery/)

I'm going to close this case because it's a dupe of #272