Propose new top-level prefix 'related'

[dcode]

What I really like about ECS is the well-defined semantics of the data model. I understand we're still evolving what that means, of course. One downside to the way that Kibana is structured to analyzed event-based, time-based data, is that it's impossible to pivot across fields that may have the same content, but different field names. Of course, this is one item that ECS aims to unify, but semantics have to come first.

I'd like to propose a new top-level object called related. This object will hold fields that are lists of necessary types, which allows a single-click pivot to related records where the data may exist in different fields. Some examples:

• related.ip: List of related IPs (IPv4 or IPv6)
• related.hostname: List of related DNS hostname
• related.id: List of related event IDs
• related.hash: List of all hashes listed in the event

We do this today in RockNSM, but under the field names like @meta.related_id, @meta.related_ip, etc. I'm in the process of migrating this over to ECS and I think it would be especially useful to have this across other datasets.

A tangible example: Bro files log.

The bro files log has a unique identifier for a given analyzed file. Usually, a file was analyzed as part of one or more network data streams. An extreme, but not uncommon example, is a file transferred over the bittorrent protocol. In this case, bro tracks a single file that was transferred from and to many, many hosts. In this case, the following transformation captures the relevant pivotable data:

• fuid: copy to event.id and related.id
• tx_hosts: copy to related.ip
• rx_hosts: copy to related.ip
• conn_uids: copy to related.id
• md5: copy to related.hash
• sha1: copy to related.hash
• sha256: copy to related.hash
• sha512: copy to related.hash

Of course, data needs to be moved around to make the rest of it conformant, but now we can pivot to the related connection-oriented logs and events from other data sources, that perhaps match hashes.

[ruflin]

This sounds pretty similar to what I'm proposing here with the global host.name in this example: #51 (comment)

I definitively agree we need fields that can contain multiple fields. I wonder if we need a special prefix for it or could use the existing fields and just assume the most generic ones are the ones that will contain all the copied values?

[webmat]

I like the idea of having all "pivot-only" fields grouped under their own distinct structure. They will inherently be messy as they'll contain a potentially long list of a given type of information.

Having all such fields under one namespace will make them self-explanatory. It will also be more straightforward to decide "where" to put new fields like this in the future, regardless of the specific use case. Everything that's a simple accumulation for pivoting always goes under related.*.

[webmat]

related. now exists: #206. There's only related.ip in there for now, however.

[praseodym]

Should the ECS index template also include copy_to definitions to create the related fields, e.g. add copy_to: "related.ip" to the source.ip template, to ensure that the related fields are always filled as expected?

[webmat]

OMG I didn't realize that copy_to was actually additive in this way. That's a great idea :-)

[ebeahan]

The last outstanding item from the original proposal is establishing something equivalent to related.id in ECS.

We created meta-issue ##1547 to assess correlated event support in ECS. Closing in favor of the meta issue.