service.type and service.name

[ruflin]

Currently ECS has a service.name and the expected content when for example talking to a MySQL service is that it contains mysql. APM on the other hand uses service.name as the manual set name by the user for the service in which the agent is running and if the service talks to MySQL, service.type is set as mysql. The following proposal is to add service.type and define the usage of both.

A few examples to explain the usage

Metricbeat monitoring MySQL and service foo

Metricbeat monitors with the MySQL module a MySQL host and with the HTTP module a user built service foo. The events collected from these will look as following:

MySQL events:

service.name:
service.type: mysql

Foo events:

service.name:
service.type: foo

It will be up to user to define the field service.type in the HTTP module. We should make it easier on our end.

Questions

• Should be default service.name be filled with service.type if it's empty?

Metricbeat monitoring 2 MySQL servers

Metricbeat is monitoring two different MySQL servers on the same host. One server stores user data the other one is for tracking. The events will look as following:

User data MySQL instance:

service.name: user-data-mysql
service.type: mysql

Tracking data MySQL instance

service.name: mysql-tracking
service.type: mysql

The user can set a name for each service he is monitoring. This will allow later to differentitate between the two services. We should recommend to prefix the name with the service.type.

Metricbeat monitoring an Elasticsearch cluster

Metricbeat monitors and Elasticsearch cluster across different hosts. The cluster name is baz.

service.name: baz
service.type: elasticsearch

The name of the cluster is put in as service.name. This allows to group together all nodes related to a cluster. Currently elasticsearch.cluster.id or elasticsearch.cluster.name are use for this. One of these fields should be copied to the service.name.

APM agent running in service foo

An apm agent running inside the service foo will report it's data as following. The service is built on top of node.js:

service.name: foo
service.type: node.js

The service.name is set by the user inside the agent. It could also be foo-prod1 and foo-prod2. In this case these would be two different services.

APM agent running in service foo talking to mysql service

An apm agent running inside the service foo is talking to a MySQL instance.

service.name: mysql
service.type: mysql

The service.type will be set to MySQL as the info is about a connection / query to MySQL. If not name of the service it connects to can be found, it will fill in service.type.

Question

• Is the above correct?
• Is the service.name needed in thsi context?

[roncohen]

if the service talks to MySQL, service.type is set as mysql

We don't use service.type at all at the moment and don't really have any need for it

[roncohen]

I think it's mostly correct, however:

An apm agent running inside the service foo is talking to a MySQL instance.

service.name: foo
service.type: mysql

The service.type will be set to MySQL as the info is about a connection / query to MySQL.

For the APM solution, "service.name" refers to the name of the service where the data was captured. I think it would be confusing for have service.type refer to the type of the service that a service was talking to when service.name refers to the name of the service where the data was captured. It's also only in some of the documents (some of the spans) that denote that a service is talking to other services. On the other hand, all our documents have service.name because it denotes the name of the service where the data was captured.

If APM documents were to have a service.type it would make sense to put the type of the service of where the data was captured, e.g. a service.type: nodejs for a service written in Node.js, as an example.

[ruflin]

I updated the APM part above accordingly with service.type: mysql and service.type: nodejs.

For the spans which connect to MySQL, my assumption would be that they would not contain service.type: nodejs but as you suggested service.type: mysql. Only the transaction document will contain the information about the node.js service type. Most spans will not contain the service information. Does this make sense?

[roncohen]

@ruflin I don't think so. We use service.name on all our documents. If the events are captured in service A, all the documents will have service.name: A. We don't use service.type at the moment and as I wrote:

I think it would be confusing to have service.type refer to the type of the service that a service was talking to when service.name refers to the name of the service where the data was captured.

To move this forward, it would be useful to get have some more use-cases for context

[ruflin]

@roncohen Agree that we should not mix service.type and service.name of two different things together in one event. That is why I was thinking in the transaction event the service data is fully related to service the agent is running in and in spans the service data could be empty or related to the service it's talking to.

For the use cases: I think on the Metricbeat side there aren't much more use cases. Any chance you could provide some more on the APM side?

[roncohen]

@ruflin at the moment

We use service.name on all our documents.

which includes the span document. Are you suggesting that we remove the service.name from span documents?

I think it makes sense to have the service name of where the data was captured on the span. This makes it simple to query for all the SQL queries for a specific service, as an example. We could move it from service.name on span to somewhere else, but then it's inconsistent with the transaction documents.

[ruflin]

@roncohen I think the service.name and service.type split up as discussed above makes sense from a Beats perspective so I opened #151 It would be nice if you or @graphaelli could do a follow up PR with examples on how APM uses the two fields.

For the span documents: I'm wondering if you need the service.name entries at all there? Is it required for some queries or is it only to be shown in the UI?

[graphaelli]

@ruflin where would you like those additional examples? service.name is in https://github.com/elastic/ecs/blob/master/use-cases/apm.yml currently.

For the span documents: I'm wondering if you need the service.name entries at all there? Is it required for some queries or is it only to be shown in the UI?

I'm not sure I follow - besides the case @roncohen mentioned where we pull the top SQL queries for a service? ES queries filtering on service.name within spans is very common.

[ruflin]

@graphaelli I think we could even have an example in the description of service.name as soon as the PR is merged.

For the spans: Got it, didn't know also there filtering on service.name is happening.