ECS: Set host.name along with host.hostname in exported data

[simianhacker]

APM server should be shipping host.name along with host.hostname per ECS. host.name should default to host.hostname but allow the user to override it with their host identifier if it's different. This is necessary to ensure cross-linking between Infra UI and APM is consistent.

[roncohen]

Related: elastic/ecs#498 (comment)

[roncohen]

We should probably set this in agents instead of in APM Server

[graphaelli]

In what situation does cross-linking happen based on host.name?

[simianhacker]

Everything except APM. In the Infra UI we have to put exceptions in for APM because they using host.hostname instead of host.name. As a company, we really need to standardize on one field across everything to represent an ID for a host. For Kubernetes Pods and Docker Containers we are all using the same ID fields. Although, it looks like there is host.id in the ECS documentation but Metricbeat doesn't use it for some reason.

Work Around Examples:

• https://github.com/elastic/kibana/pull/42197/files#diff-0f965401975f16849fcc929d348aeebcR221-R227
• https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/infra/public/components/waffle/node_context_menu.tsx#L49-L57

@elastic/beats @ruflin Any reason why we are not setting host.id in Metricbeat/Filebeat? Seems like that should be the blessed identifier field for Hosts instead of host.name

[simitt]

Linking to some context around discussions of host.name vs. host.hostname: #1609 (comment). Haven't followed changes on host since then, but it seemed fair to leave the field empty if not user overwritten (which the agents don't support yet afaik).

[graphaelli]

Thanks for digging that up @simitt, I had forgotten some of that context.

We can certainly copy host.hostname to host.name for APM, and let agents override that value. We'd still need to keep some workaround in place as the curated UIs should fall back to linking based on host.hostname if host.name is not present, to support older 7.x data.

host.id would be great if multiple processes could arrive at the same value for it independently - that's why kubernetes and docker links work so well - but I don't think there is a portable way to do that well.

[simitt]

Turns out agent collected information in system.hostname can already be overwritten by the users.

Current behavior server side (after merging #2531 ):

• map system.hostname to host.hostname if system.kubernetes is empty, otherwise set to system.kubernetes.namespace or to nil if not available.
• map system.name to host.name if available, otherwise copy value from host.hostname.

We can change the behavior in the agents to send auto retrieved information in system.hostname and customized information in system.name only if it is actually user overwritten. That would theoretically require a major version bump or could lead to inconsistent data depending on agent version. I tend to consider this a bug fix.

Other option would be to change behavior in the server to map system.hostname to host.name and introduce new field for what should be stored as host.hostname. In case kubernetes information is sent up there would be no way for information sent from current agents to identify whether or not system.hostname is customized and therefore should overrule the kubernetes information or not.

I therefore suggest to change agent behavior and consider it a bug fix, as the customized information should have never ended up in the host.hostname information in the first place.

Suggested behavior:

• send auto retrieved host name information in system.hostname. Server keeps current logic of using this information if no kubernetes information is provided.
• send customized host name information only if overwritten by user configuration in system.name. Server stores this information if given, otherwise falls back to store information from host.hostname.

cc @elastic/apm-agent-devs

[watson]

If the agents are updated to send the real hostname in system.hostname and the optional user-provided hostname in system.name, wouldn't we then lose the user-provided hostname in case the agent is talking to an older APM Server that doesn't yet know to read system.name?

[SergeyKleyman]

Maybe we should leave existing system.hostname alone for backward compatibility and introduce two fields with clearer names? For example system.detected_hostname and system.configured_hostname? Then in time we can phase out system.hostname.

[simitt]

Agree with @SergeyKleyman 's proposal. In case at least one of the two new fields are set, information in system.hostname would be ignored.

[graphaelli]

I agree on leaving system.hostname alone but would prefer shorter names for the new information, especially if they can match the agent configuration. Will the configuration for this information will be name on the agent side? How about name instead of configured_hostname and add detected_hostname, since the user will never interact with anything called that directly.

[SergeyKleyman]

Will the configuration for this information will be name on the agent side?

According to Elastic APM Backend Agent Config Comparison the name of the configuration option is hostname (ELASTIC_APM_HOSTNAME environment variable)

[graphaelli]

Thanks, missed that somehow. I'd still prefer name and detected_hostname instead of configured_hostname but am not too fussed about the intake naming if anyone feels really strongly otherwise.

[SergeyKleyman]

I don't think it's that important but the main reason I suggested more descriptive names is to make it clear that system.detected_hostname and system.configured_hostname are independent and there is no logic on agent side trying to deduce one from the other thus deferring this logic to APM Server.

How about name instead of configured_hostname and add detected_hostname, since the user will never interact with anything called that directly.

Do you think the connection between hostname configuration option, system.name in Intake API and host.name Elasticsearch field is clear? To me the meaning of system.name would very hard to deduce without referencing the spec because it's not very clear to which entity system object refers. It seems that system object is used as an umbrella for everything outside monitored application/service.

Also I'm not sure how much of an optimization shortening system.configured_hostname would be. If most of the users rely on system.detected_hostname and don't use hostname configuration option (I don't know if it's indeed the case) then system.configured_hostname will be omitted from the payload anyway.

What is the motivation for shorter names in Intake API if the Elasticsearch fields will be different anyway? Is it mostly to reduce network traffic volume?

[graphaelli]

What is the motivation for shorter names in Intake API if the Elasticsearch fields will be different anyway? Is it mostly to reduce network traffic volume?

Yes, mostly that. Also (as you well know) more verbosity makes larger events which we need to split (per api_request_size) among more payloads. Agreed it's minor in this case.

Also agreed with the rest of what you wrote. Only agent authors see the intake api so I can see making the field names make the most sense for that audience, and consider it a bonus when the persistence matches as well.

[simitt]

IMO system.detected_hostname and system.configured_hostname is way more clear than system.name and system.detected_hostname, but as already mentioned, the API is mainly used by agent devs and as long as there is a good comment explaining the meaning, the name should not matter too much.

@graphaelli want to make a final call on the naming, so we can move on?

[graphaelli]

I'm convinced, let's proceed as proposed, thanks.

[simitt]

I created a dedicated issue for further discussions around host.id to keep the two issues separated #2608.

[simitt]

Fixed by #2540