[Asset Manager] Extract host assets from metrics

[klacabane]

Summary

Create a function that queries metrics indices and returns host asset documents constructed from the query response. The function should be implemented in this repository and run as part of this etl process.

See collectServicesFromSummaries for a blueprint.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[klacabane]

Paired with @miltonhultgren on this today aiming at refining what the query would look like initially and what data we can/want to extract out of the documents returned.

Observations:

• targeted metrics-*,metricbeat-* indices
• there are not a lot of hosts (<10) available in edge-oblt
• the query will collapse on host.hostname. We'll likely want to surface documents that have more metadata included (cloud.*, kubernetes.*..), how can we do that ? For example, if a hostname is represented by two documents and only one of them have cloud.service.name, can collapse favors it instead of the other ?
• how can we rule out false positive (ie host.hostname is wrongly filled by apm php) ? We won't at first, but one potential solution is to build a blocklist of offending sources
• some hosts may not have the metadata necessary to set the correct type (ie gce/ec2), what asset.type and asset.ean should we persist ? Should we ignore those for the moment or can we persist these values as a generic host:<id>. What if this asset is stored with the correct type from another source ? We won't be able to dedup based on ean. Note that this will also prevent us from routing the asset document to the right asset-{type} stream
• Host data can surface containers and/or k8s pod informations. Should we also create assets out of them ? Yes, we can take the same approach when looking at services in apm data. For situations where it will create duplicate and two way references, the API accounts for that
• how do we categorize hosts with cloud.provider (eg aws) but no cloud.service.name (eg ec2) ? Can we safely fallback to a default service ?

[miltonhultgren]

how can we rule out false positive (ie host.hostname is wrongly filled by apm php) ? We won't at first, but one potential solution is to build a blocklist of offending sources

Just to note also that the OpenTelemetry agent seems to do the same thing (the pod is reported as host.name).