Investigate how we can set host.id for events

[simitt]

From the ECS Host fields definition:

Unique host id. As hostname is not always unique, use values that are meaningful in your environment.
Example: The current usage of beat.name.
type: keyword

@simianhacker started discussion around storing host.id #2502 (comment):

As a company, we really need to standardize on one field across everything to represent an ID for a host. For Kubernetes Pods and Docker Containers we are all using the same ID fields. Although, it looks like there is host.id in the ECS documentation but Metricbeat doesn't use it for some reason.

@elastic/beats @ruflin Any reason why we are not setting host.id in Metricbeat/Filebeat? Seems like that should be the blessed identifier field for Hosts instead of host.name

Follow up comment from @graphaelli #2502 (comment):

host.id would be great if multiple processes could arrive at the same value for it independently - that's why kubernetes and docker links work so well - but I don't think there is a portable way to do that well.

[graphaelli]

machine-id is one of the non portable ways I had in mind. Some projects use one of the MAC addresses of the machine on windows. Perhaps a path forward is creation of an elastic machine id? All processes on a machine that care about consistently+uniquely identifying a host agree on a path to persist the id.

I expect @elastic/beats has some thoughts on this, assuming they've explored making various beats on the same host report some consistent identifier across processes.

[ruflin]

Agree we should populate host.id on the Beat / Agent side.

@graphaelli Are you thinking of a UUID generated by the first service running on the host or something that is also user configurable? I think the closes we have in Beats is beat.name as it can be user configured but unfortunately defaults to hostname, which is not unique. @ph might chime in here?

[simitt]

I don't think we can use the beat.name in the APM case, as the host information is the information where the agents are running and not where the APM Server is running and we need to ensure we stay consistent.

[ruflin]

@simitt Agree, we should not use beat.name, this was only to give some example for the user configurability, sorry for the confusion. host.id is the way to go, but we need to figure out how we populate it to be consistent and unique.

[graphaelli]

@graphaelli Are you thinking of a UUID generated by the first service running on the host or something that is also user configurable?

The former, though if /etc/machine-id exists its contents could be used. That would allow users to configure it through traditional mechanisms (systemd --machine-id=, etc) rather than an Elastic specific one.

[reilee]

In my case, I created one logstash server on VM and then did copy construction. This makes /etc/machine-id same on all copied hosts. When I tried to setup monitoring by metricbeat I can see only 1 host in Kibana.

[axw]

Closing this one in favour of #4368