[Filebeat] The host.name sent from Filebeat doesn't match the same field from Metricbeat

[simianhacker]

One of the common issues we see in the Infrastructure UI is that when shipping data from the same host for Metricbeat and Filebeat, the host.name does not match up. The Filebeat host.name typically ends up looking like filebeat-{some kind of hash id} instead of the actual system hostname. This results in the following nodes on the waffle map:

When you click on the filebeat- prefixed hosts to view the metric, you end up with nothing. Until this is fixed we can't move forward with plans to add logging metrics OR a log view to our detail page for any of the hosts. This also affects our Logging UI for all hosts (with metrics) because host.name doesn't match any of the hosts logs; essentially "View logs" for hosts is broken.

[jsoriano]

@simianhacker thanks for opening this issue, this will help to keep track of possible solutions for this issue.

I think that the issue here is that in the reference configurations we are using host network for metricbeat, but not for filebeat. Then the hostname for metricbeat is the same as the hostname of its host, but the hostname for filebeat is the name of the pod (filebeat-{some kind of hash id}). I'd say this issue will happen with any beat that don't use the host network.

There are some possible solutions to this:

• Use host network in all our reference configurations. I think this would be ok by now as a workaround, but actually long-term we should try to require less privileges and not more, so we should think in solutions that don't require access to the host network.
• Add a flag (--system.hostname?) to override the host name, as we do with --system.hostfs for the host filesystem.
• Add some kind of "hostname providers" to autodetect when the beat is run as a container and try to obtain the host hostname from the runtime, or from the apiserver in the case of Kubernetes. This may be complex and would still require something like the flag to cover corner cases.

[exekias]

This was fixed here: #12790, so deployments using newer manifests should not be affected. Do we still see this happening?

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[exekias]

I've seen that our helm-charts have hostNetwork disabled by default: https://github.com/elastic/helm-charts/blob/480a19413024cab7f9ea96e2d40078bc2247ce2d/filebeat/values.yaml#L53, which would cause this issue. Maybe we can open a PR to switch the default to true, like in our reference manifests.

As for a longer term solution I think @jsoriano points here are good, some options are:

• Make add_kubernetes_metadata set host.name (taking the node name)
• Always override host.name from the downward API

I like the first one more because it doesn't depend on the deployment model. In the past we discussed around the diference between K8S nodes hostname & node name, but in most cases they are the same, right?

[ChrsMark]

I've seen that our helm-charts have hostNetwork disabled by default: https://github.com/elastic/helm-charts/blob/480a19413024cab7f9ea96e2d40078bc2247ce2d/filebeat/values.yaml#L53, which would cause this issue. Maybe we can open a PR to switch the default to true, like in our reference manifests.

I think that Helm charts have hostNetwork disabled intentionally (cc: @jmlrt ). See https://github.com/elastic/observability-dev/issues/666#issuecomment-607220803.

[jmlrt]

Yes, we have hostNetwork disabled in default values for both Filebeat and Metricbeat charts.

The main reason is that some K8S providers may not allow enabling hostNetwork.

But also that only one Filebeat/Metricbeat pod can be deployed per node with hostNetwork (unless youmanually configure a different port for each pod) which may not be compatible with some multi-tenant k8s architecture like some of our elastic-apps clusters or the ephemeral GKE clusters launched by our CI which are testing several chart releases with different configurations in parallel on the same nodes.

Note that we provide an option to enable hostNetwork which is documented in usage notes and FAQ (usage note and FAQ for Filebeat).

[LucaWintergerst]

in case anyone needs a quick fix when using k8s, until we have a proper solution for the different scenarios. This replaces the host.name field with the value of kubernetes.node.name :

processors:
  - add_kubernetes_metadata
  - drop_fields:
      fields: ["host.name"]
      ignore_missing: true
  - copy_fields:
      fields:
        - from: kubernetes.node.name
          to: host.name
      fail_on_error: false
      ignore_missing: true

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[PrplHaz4]

pretty sure this same issue is still present when running in docker containers (and there is a larger "fix all the hostnames" issue for running beats in containers elsewhere that I can't find now...