Document how / whether objects should be reused in different places in ECS

[webmat]

In recent conversations we've determined that the following objects should be reusable under specific objects:

• geo
  • Expected under source, destination, device and host
  • It's not expected at the top level
• os
  • Expected under host, device, user_agent
  • It's not expected at the top level
• user
  • Expected under source, destination and host
  • It can also be at the top level

Some additional reusable objects have also been discussed but I'm not sure agreement is there yet. Listing them out:

• threat
  • Could be nested under the part of the event that's problematic (like source if the source IP is cause for the match)
  • Counterpoint: if we go with a model of saving threat matches to a distinct index, we do not need to nest it. One match per document in this approach. So we can copy the full event data (longer retention on matches than raw events), we can copy the full threat feed indicator that matched, and we can have a field that documents what the matching field is. E.g. match.field_name: "source.ip"
• host
  • Had a 1-1 discussion with @graphaelli, so the team isn't up to speed yet. But I think this is a good case.
  • Expected at top level or under agent
  • It should be under agent when the monitoring agent is deployed on another host (e.g. APM server is the agent, and we also track the host information of the monitored app servers).
  • Other similar case could be Metricbeat monitoring multiple external hosts (e.g. one Mb monitoring lots of RDS instances).

Note that it will be important to "pre-determine" where we expect these objects to be re-used for two reasons:

• To set people's expectations clearly as to what data belongs where
• To properly generate the index template
  • A corollary to this is therefore that this reuseability will have to be represented in the yml files

[webmat]

See #72, among others

[ruflin]

Sounds similar to #71 ?

[webmat]

Not really the same thing. And I still need to flesh out the body of this issue, but I think a bit of a free form discussion would be in order before I can do that (about #71 and about this issue).

#71 is about deciding if we let people merge the objects together (compose) or if we add the explicit nesting everywhere we reuse a reuseable object.

What I want to do here is actually have an explanation in the docs about how to do this reuse the right way. The goal being to avoid having tons of tools say they're "ECS compliant" but they're actually all reusing the objects in a way that's different from one source to another.

• An example (probably a bad one) could be that we say "host is reuseable", but also "here are the 3 places where you should put it".
• Another example would be the "endpoint object" (just noticing we don't have a generic name for it yet) that we currently have as source an as destination, and are considering letting people have as client and server as well. I think we need guidance on where this object can be reused, in order to avoid having this information spread out in too many places. I could easily see people putting it in: top level, agent, container, device and event (in addition to src/dsv, src/cli).

So this issue is about actually adding the guidance to the doc, once we've figured it out.

Hopefully this clarifies it a bit.

[webmat]

@ruflin @MikePaquette I've fleshed out the body of this issue. Not sure this generates a notification. But I'd like you to check it out and be ready to discuss it at today's ECS meeting, if you can.