[RFC] Cyber Threat Intelligence Fields Stage 1

[shimonmodi]

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?
• For proposing substantial changes or additions to the schema, have you reviewed the RFC process?
• If submitting code/script changes, have you verified all tests pass locally using make test?
• If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes?
• Is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
• Have you added an entry to the CHANGELOG.next.md?

RFC Preview

[webmat]

Thanks for opening the follow-up PR, @shimonmodi.

Let's start with a simple request, before diving into the big picture stuff: Please change your handle in the "People" section to your Github handle @shimonmodi :-)

From stage 1 going forward, we will want this RFC document to be the main reference for the proposal. We can keep the link to the Google Doc as a historical reference to your working document. But since it's not version controlled, we'll want this information captured here as well.

You can look at the raw Markdown to see HTML comments (<!-- ... -->) on what to focus on at stage 1, in each of the section of this document. These are general comments applicable to any RFC.

But since you've already done a lot of work in your working doc, here's a few more specific suggestions, per section of the RFC template.

Fields

We could split this one in two logical parts:

• In one part you can list all of the new fields you think would be necessary for this proposal.
• In the other part, you can list the mapping of typical threat ioc fields to pre-existing ECS fields.

Both of these can be bullet lists with a short explanation. No need to worry about formatting and nitty gritty details, yet. Let's discuss the concepts first :-)

Usage

This section can be used to give details how this is meant to be used. Is this a way to append IOC details to existing events, or is this documenting how an index that stores databases of IOCs should be structured?

This can eventually serve as the basis for how we document the usage of this part of the schema.

Source data

You're welcome to start listing a few examples of indicators along a few axes:

• List the different kinds of IOCs, as you already do in your working doc for type: ip_address, ip_range, domain, domain_with_ip, file_hash, email_address, URL, file_extension.
• List different schemas/standards that exist, to define these IOCs (I think STIX is such a standard, correct?) but also other de-facto standards, for example if some prominent data sources don't follow a third party standard.

Note that stage 1 is still an early draft, and it's fine if things are still a little rough, or not fully explained yet.

Final note, today we closed #1023 in favor of this RFC, so that the discussion would happen in one place. But I like the proposal from issue #1023 to nest the IOC fields under ioc.* instead of under threat.*. The threat fields in ECS are meant to capture a mapping to a threat taxonomy like Mitre ATT&CK. If we add a second purpose to that same group of fields, I think that will be confusing to people.

Do you think nesting the new fields under ioc.* would work for what you had in mind, or am I misunderstanding the intent of your proposal?

[shimonmodi]

@webmat - thanks for your patience and guidance :-) . I have followed through on your recommended changes.

The discussion about nesting under ioc.* instead of threat.* is something I want to dig a bit deeper into. The current logic is to use event.type to represent indicator type and nest all threat intelligence attributes under the existing threat.* field set. This way we would minimize adding new fields and also expanding the purpose of threat to what security users typically associate with this concept. I do want to point out that keeping threat.* restricted to only MITRE taxonomy is extremely restricting - users think of it as a more expansive concept.

To answer your question - the proposal of nesting under ioc.* could work, but just want to work through if that could create issues down the road. I don't know the answer - it needs some thinking.

[ebeahan]

Looks like some additional changes possibly meant for this PR got placed into another: https://github.com/elastic/ecs/pull/1043/files

@shimonmodi - you should be able to make any adjustments in your forked repo's shimonmodi-patch-1 branch. Once committed those changes will reflect here in the PR as well. If you need another set of eyes, feel free to reach out, and I'll be glad to assist.

[MikePaquette]

@webmat

The threat fields in ECS are meant to capture a mapping to a threat taxonomy like Mitre ATT&CK.

That is true, but this limit was not by design - we had intended to have threat.* be more expansive when we created it, but started only with the framework fields.

If we add a second purpose to that same group of fields, I think that will be confusing to people.

I'm not sure about that, let's try to get more feedback on this PR.

I think it's appropriate to use the threat.* fields for the IOC-type information as Shimon is proposing in his RFC, and I favor that over adding a new ioc.* field set.

[SHolzhauer]

@MikePaquette

The threat fields in ECS are meant to capture a mapping to a threat taxonomy like Mitre ATT&CK.

That is true, but this limit was not by design - we had intended to have threat.* be more expansive when we created it, but started only with the framework fields.

If we add a second purpose to that same group of fields, I think that will be confusing to people.

I'm not sure about that, let's try to get more feedback on this PR.
When I created #1023 it was with this in mind. How I understand (and apply) ECS is fields present "concepts" at a higher level. A bit generalised so to speak. When trying to map IOC's into ECS I ran into the challenge how to distinguish between the indicator itself and "what it is an indicator for". With the latter being the translation/mapping to MITRE ATT&CK.

What I was specifically trying to do was capture information for an IOC and have the Technique and Tactic information present.

The second stage of my thinking evolved around the "range of threat intelligence" possible. Without complicating it extremely I wanted to scope down onto one specific threat intelligence "type".

However ioc shouldn't have to be a top level field after thinking, this could be combined with the threat field possibly;

Options forward i can see

1. Nest the ioc field under threat (subsequent underlying fields to be discussed) but this would capture threat intelligence specifically for Indicators of Compromise.
2. Keep ioc as proposed in Proposal - Add IOC fields #1023

With the discussion I would now go for option 1, this method allows for future expansion by adding threat.x fields for other types of threat intelligence.

Why hold on to the ioc field
To prevent confusion and keep the goal ECS fields as clear and distinguished as possible:
threat = high level threat mapping; a.k.a Mitre ATT&CK
ioc = indicator of compromise information

[shimonmodi]

@SHolzhauer - thanks for the feedback. We are also aligned with your point about holding onto the ioc field to provide context on how to interpret the various fields under threat.* Our current thinking is to use event.type = indicator to indicate that threat.* fields should be viewed in that context. We can see event.type being extended to other STIX SDO's (infrastructure, campaign etc).

[peasead]

Options forward i can see

1. Nest the ioc field under threat (subsequent underlying fields to be discussed) but this would capture threat intelligence specifically for Indicators of Compromise.
2. Keep ioc as proposed in Proposal - Add IOC fields #1023

I think there are some concepts that we can harvest from #1023 and nest under threat.*.

[SHolzhauer]

@shimonmodi sure, the event.type = indicator could work. Had a re-read through the sheet and a couple of my views:

1. original intel isn't stored/explained at the moment.; This can be stored in event.original, maybe add it to the sheet for clarification?
2. the threat.??? field. The description is on "intent", doesn't the Mitre ATT&CK framework cover this? For example an IOC on an IP address used for data ex-filtration can be mapped to the tactic Exfiltration
3. event.severity and event.risk_score are "non-normalized" in the current thinking. For risk_score there is the risk_score_norm but severity doesn't have a field like this. This could make different intelligence non-comparable.
4. I agree with the comment from Andrew Pease on threat.time_* fields, making threat.time.* an object with last_seen, first_seen underlaying date fields. I would propose to also add last_updated this would enable a user to think on "the age" of intelligence.

[gimmic]

Nesting 'ioc' under threat.* works but has potential for some real nesting rabbit holes. It would be nice to aim for alignment with existing threat taxonomy frameworks such as STIX and CIF as examples. Personally I think STIX is a little heavy/verbose and CIF has a much smaller common field set to look at which means it will be easier to align in ECS. Compatibility/mapping here would also help for bidirectional data flow down the road(APIs).

Currently I'm much more interested in CIF and we have started to put fields in the format of threat.cif.*. Nesting it in threat.ioc.* isn't bad, and I don't think it would be advantageous to nest it in threat.ioc.cif when there is likely to be much more overlap in a more common threat.ioc.* field.

Here's the basics of our CIF mappings:

threat.cif.confidence
threat.cif.count  (count seen in IOC source)
threat.cif.description  (human desc)
threat.cif.firsttime   (first time seen)
threat.cif.group      (visibility rules for/from source)
threat.cif.indicator  (unique value)
threat.cif.lasttime  (the last time this IOC was seen in CIF)
threat.cif.provider   (the provider of the IOC)
threat.cif.rdata   (related data)
threat.cif.reporttime (time the data was pulled)
threat.cif.tags   (tags from the CIF data)
threat.cif.tlp    (Traffic Light protocol, sharing sensitivity)

We do map several of these up to threat.*, such as threat.cif.confidence == threat.confidence.

Many fields can be arrays as there may be multiple CIF entries for a single IOC. The IOC is the unique key which builds arrays of values for the multiple different sources. The idea of nesting it under cif is also that we might have different threat sources with overlapping data. Ideally, we can normalize this inside ECS to be mostly all under threat.* and threat.ioc.*

The MISP project has also already done a lot of work here we might be able to benefit from: MISP Data Models. MISP and The-Hive are popular security projects where mutual compatibility with would be extremely useful for all parties.

[webmat]

So looping back on where we should add these fields, I'm fine if we don't add them at the root in ioc.*, and we add them inside threat.*.

However I do think it would make things clearer if the new IOC fields were nested at threat.ioc.*.

This way we would continue to have a clean separation (with the exception of threat.framework), where threat.tactic.* and threat.technique.* are where we map on the taxonomy. And all IOC fields would be under threat.ioc.*.

Does this suggestion make sense? If it does, we should adjust the proposal to make it so.

By the way this is looking really good already for a stage 1. We could merge and finish hashing out the details in a stage 2 PR.

For now let's see if there's already agreement on making the adjustment for threat.ioc.*. If so, let's make the adjustment and I think we can merge stage 1. If there's no agreement yet, let's instead capture the idea in the "Concerns" section, merge stage 1 and finish hammering this out in stage 2.

[shimonmodi]

@webmat - I will capture the proposal of threat.ioc.* in the Concerns section so we don't lose track of it. Let's move RFC to Stage 2 and resolve the nesting recommendation in Stage 2.

[webmat]

I'm good to merge this stage 1 PR, the proposal looks good. We can finish hashing out the field details in the stage 2 PR.

[ebeahan]

The date wasn't updated prior to merging. I opened up #1100 to correct.