Add new service field to Intake API + ES

[simitt]

A new field service.node.configured_name will be sent from the agents. The server needs to add it to the Intake API as part of the metadata.

The server then needs to populate the new ECS field service.node.name with following logic (copied from elastic/apm#141 (comment)):

If `service.node.configured_name` is set - use it
Otherwise, if `system.container.id` is set - use it
Otherwise, use host name (apply same logic applied today to `host.name`)

(edited 2019/09/16: move logic how to derive value of service.node.name also to apm server)

Detailed information in elastic/apm#141.

Things to clarify:

• Exact field name for Intake API, see Adding service.node.name field apm#141 (comment)
  will be service.node.configured_name
• wait for Add service node name ecs#538 to settle on the ES field
  settled on service.node.name
• service.name only allows values following the pattern ^[a-zA-Z0-9 _-]+$, do we need to apply the same restrictions to the new field?
  not necessary here, as the concerns where for cases where the service name should be part of the index, which is not discussed for this field
• As far as I understand the issue, the information should be used for aggregations, so indexing will be necessary. I suggest to index as a keyword with a length limit of 1024 on the Intake API. @eyalkoren do you agree?
  see Add new service field to Intake API + ES #2696 (comment)

[eyalkoren]

@simitt there is still a discussion going on some of that, even including whether this field be used for aggregations or only labelling.

I assume we will use the ECS name pattern in the intake as well, no need to use instance here and node there.

service.name only allows values following the pattern ^[a-zA-Z0-9 _-]+$, do we need to apply the same restrictions to the new field?

I don't think so, unless there is a good reason.

[simitt]

The restricting pattern for service.name is something we need to investigate from the APM Server / ES side. There is an open issue #1134, we should probably pick up once more and make a final investigation and decision around this.

[simitt]

After some more context about how this field is going to be used, we agreed on indexing it, which also requires a limit of 1024 on the Intake API.

[simitt]

@eyalkoren for which event types should the new field be added (transaction, span, error, metric)?

For error and transaction all service fields can be overwritten on a per event basis. In case the field should also be added for spans and metrics, should it be overwritable per event for those?

[simitt]

Summary of the discussion with @eyalkoren:

• Overwriting metadata per event within one request for metrics is not implemented. Behavior should not be changed for service.node.name, as in the agent config the field can't be set per service.
• For error and transaction all service related metadata can theoretically be overwritten per event. Agents only support overwriting a subset of this fields (if any). Agents will not implement behavior to overwrite service.node.name per event, but there is no need to treat this different than other service related field on the server.
• service.node.name is currently only used for metrics. It is automatically added to transactions and errors, as they store all information from the metadata. spans are an exception where only minimal information is stored, as they always refer to a transaction where additional metadata can be found. Also checked with @dgieselaar that the APM UI currently does not have a use case for the field being stored at span events. Therefore the minimal service information stored with span events will not include service.node.name.