Revised Network Metrics

[robgil]

Inbound and outbound are ambiguous with regard to netflow data. When
using src and destination, the direction is explicit. inbound and
outbound don't map back to src and dest and becomes ambiguous. I don't
see a use case for inbound and outbound as this is already
defined by src and dest. This removes the inbound and outbound
sections and replaces them with a single transfer field which lines up with
netflow, gcp flow logs, aws flowlogs, and so on.

[ruflin]

+1 on this change. Would like to get some feedback from @MikePaquette on this one.

Could you rebase this one on master to get the proper CI checks and update the CHANGELOG file?

[ruflin]

Can you put this under Breaking Changes?

[ruflin]

We often use the first line as a quick description and then more details below. Can you reformat this as below and add a dot at the end?

Network bytes transferred. 

Typically used with netflow, sflow, or cloud provider flow logs.

Same applies to the description below.

[ruflin]

I think this field we still need?

[robgil]

@ruflin src and dest are explicit in the direction. This is redundant imo.

[ruflin]

It's a convention field to filter on. We for example use it here in auditbeat: https://github.com/elastic/beats/blob/master/auditbeat/module/auditd/_meta/fields.yml#L162

@andrewkroh can perhaps share some more background here?

[MikePaquette]

The scope of the common schema is not limited to netflow or other flow data, but must also accommodate events from host systems, security devices, and network devices, all of which have network interfaces with an inherent receive and transmit directions. The directionality, when populated adds value to IT operations and Security Analytics use cases, for example, allowing the monitoring of changes to inbound:outbound byte/packet ratios within a given environment. If directionality is not discernible in a data set, flow-related fields should be mapped to network.total_bytes and network.total_packets and leave the inbound and outbound fields empty.

[Update]I am just realizing that network.total_packets and network.total_bytes did not make it into the current network fields list, possibly leading to the creation of this PR.

[robgil]

@MikePaquette I think I partially understand the reasoning. Conceptually though inbound and outbound is still ambigious as to the direction, but I think security zones is more analogous to the intent. For my own edification and for that of others, can you provide specific use cases and/or vendor products that utilize an inbound and outbound designation? I have not personally come across any vendor products which utilize this notation with the exception of perhaps firewall rulesets, but even then its a prefix list which utilizes sources and destinations.

Another thought is that inbound and outbound can refer to RFC1918 address ranges vs. public ranges. This however breaks when doing private peering, forcing one to fall back on src/dest or src-zone to dst-zone.

I could see the inbound/outbound concepts evolving in to security zones which could be arbitrarily defined by an operator for their particular environment and derived based on importance to the operator.

Ex.
security.source.zone: public (loosely defined by the operator as public internet)
security.destination.zone: DMZ (loosely defined by the operator as their DMZ)

Another example with private peering..

network.source.zone: radianz
network.destination.zone: big-bank-1

Example of public peering

network.source.zone: level3 (upstream peered network)
network.destination.zone: dc5 (datacenter in dc5)

Example with ASNs (which could be considered inbound/outbound)
network.source.asn: 7224
network.destination.asn:

The ambiguity for me is what specifically derives inbound and outbound. Inbound and outbound to what? The value for direction is only inbound and outbound, but this could mean anything. src and destination are explicit.

[MikePaquette]

@robgil Thanks for the feedback. One challenge in defining a common schema is that it should accommodate a wide variety of applications. As you point out, in "network provider" type applications, directionality is highly ambiguous, and in those and similar applications, we'd expect that the inbound and outbound-labeled fields and the direction field may not be populated.

However this common schema must also provide value in other application types, such as traditional enterprises, data centers, or hosting facilities where the directionality of communication has clear meaning to the team operating the logging and monitoring infrastructure. As you mentioned, devices such as firewalls, intrusion prevention systems, DDoS Defense, WAFs, load balancers, home routers, and even full duplex network taps have multiple ports and/or asymmetry in their port usage that can be easily associated with directionality concepts in many such applications. For these applications, we defined the direction, inbound and outbound bytes/packet fields in the common schema.

For example, for network.inbound_bytes, we'll describe this as: "Bytes in - If endpoint host, use RX Bytes. If network device, use Ext-to-Int, Ext-to-DMZ, DMZ-to-Int Bytes." For network.direction we'll describe this as: "Direction of the communication flow. For TCP, direction of the SYN packet. For UDP, ICMP, direction of initial request." with corresponding values including: { "inbound" , "outbound" , "unknown" }

This directionality concept is not new for log formats and information models. For example Splunk's Common Information Model includes bytes_in bytes_out and bytes ; ArcSight CEF includes bytesIn and bytesOut ; Apache SPOT ODM includes in_bytes and out_bytes.

For "network provider" type applications, I agree we need to add in network.total_bytes and network.total_packets so flow details where directionality is ambiguous can be mapped properly. The network.direction could either remain unmapped, or populated with "unknown".

[ruflin]

We have an example here in beats for docker where we use inbound and outbound in the same event: elastic/beats#7253

[robgil]

@MikePaquette oooo so you're referring to aggregate inbound/outbound on a specific device/endpoint/host. That makes sense now.

For the netflow case, I think we'll want to add the network.total_packets and network.total_bytes since netflow is not an aggregate. It might also be worth noting that inbound/outbound is an aggregation for a specific device/server/container/router/switch.

I can clarify that this is an aggregation for the device and add the network.total_* stuff if that makes sense?

[ruflin]

I see that these are very generic metric fields but I'm personally still struggling to fully understand how metrics fit into ECS as most of the other ECS fields are focused on correlation / aggregation. For metrics it's not so obvious. At the same time I agree that there are potentially some metrics which are generic enough to be in ECS.

Perhaps as inspiration here are the network metrics we collect in metricbeat: https://github.com/elastic/beats/blob/master/metricbeat/module/system/network/_meta/fields.yml Based on ECS it should be in the future inbound/outbound.

[ruflin]

Closing this for now as lots of changes happened since this discussion. @robgil Please reopen if you think we should continue this.