Module dashboards should not use non-ECS agent.hostname fields

[MikePaquette]

Background:
As we transitioned beats to ECS in 6.x and 7.0, we made breaking changes to change beat.hostname to agent.hostname. However, agent.hostname is not an ECS field, and is not expected to ever become an ECS field (see elastic/ecs#178 (comment))

ECS has two relevant fields host.name and host.hostname. host.name is populated for most beats, except for heartbeat (see #12107) and APM, even if add_host_metadata is not enabled.

host.hostname is added when add_host_metadata is enabled.

Generally, host.name and host.hostname will contain the same information.

Describe the enhancement:
Modify any dashboard, saved search, alert, ML job, or other module content or solution widget that uses agent.hostname to instead use ECS host.name.

• Packetbeat
• Filebeat
• Winlogbeat
• Metricbeat
• Auditbeat
• SIEM App Widgets
• Logging App Widgets
• Infrastructure App Widgets

** Strikethrough if not applicable

Note 1: If the module or solution expressly and by default enables add_host_metadata, and there is a specific need the capture the (uncommon) distinction of host.hostname vs. host.name, then host.hostname may be used. Otherwise, all use of agent.hostname should be changed to host.name.

Note 2: It is probably best to continue to populate agent.hostname in the events themselves since users have built content using it, and we don't want to make yet another breaking change for them.

Describe a specific use case for the enhancement or feature:
Users are confused as to which field they should be using in their dashboards, as there has been a number of changes starting with 6.3. They hoped that they'd need to make one set of changes to get to ECS. They see agent.hostname being used in our dashboards and think it is the default, and that they should use it, but it's not an ECS field.

Related Issues:
#11860
#11790

[elasticmachine]

Pinging @elastic/secops

[MikePaquette]

cc: @webmat ^^

[webmat]

Thanks Mike!

Note 2 addendum: we should leave agent.hostname for the life of 7.x but mark it as deprecated ASAP, then remove agent.hostname in 8.0.

I think agent.hostname should be replaced by host.hostname, however host.hostname is only present when add_host_metadata is configured for the events.

So I think we should also make Beats populate host.hostname regardless of add_host_metadata, just like host.name. We could instead deprecate it in favour of host.name, but the semantics are different: host.name is meant to be customizable (even though it defaults to host.hostname) whereas host.hostname is meant to be the "real" value on that host, just like agent.hostname is.

[MikePaquette]

@webmat we could mark agent.hostname as deprecated, but when it is removed, it will again break users current dashboards. Not sure how fast to push it.

Yes, having beats always populate host.hostname as a direct replacement for agent.hostname regardless of whether add_host_metadata is enabled is indeed an option. Let's decide on one approach. In fact, add_host_metadata is still listed as beta.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.