Email object and Email Activity updates. Deprecate Email URL Activity and Email File Activity. #1259

pagbabian-splunk · 2024-11-20T21:56:48Z

Related Issue: N/A

Description of changes:

Added domains files urls and message_trace_uid to the dictionary for use with the email object and email_activity class.
Deprecated the two other classes in favor of a single email_activity class where the email object can also hold files domains and urls.

Signed-off-by: Paul Agbabian <[email protected]>

…favor of an updated email_activity class. Updated the email object to include domains, files, urls arrays. Updated the email_activity class to add the message_trace_uid ID. Updated the email_activity class to use the references[] for the Trace activity_id instead of the description URL. Updated the email_activity class description to reflect its SMTP protocol and the possible URLs and files attachments. Signed-off-by: Paul Agbabian <[email protected]>

Signed-off-by: Paul Agbabian <[email protected]>

…on to fail!! Signed-off-by: Paul Agbabian <[email protected]>

…ed an at_least_one constraint on all the to and from attributes. Not all email logs have the 'to' and 'from' but must have at least those or 'smtp_to' and 'smtp_from' in the log. Signed-off-by: Paul Agbabian <[email protected]>

Signed-off-by: Paul Agbabian <[email protected]>

mikeradka

Looks great

davemcatcisco

The requested changes are only the grammatical things in descriptions. My other comments are just suggestions or observations.

davemcatcisco · 2024-11-21T11:37:47Z

dictionary.json

@@ -1781,6 +1781,12 @@
      "type": "domain_contact",
      "is_array": true
    },
+    "domains": {
+      "caption": "Domains",
+      "description": "The domains that pertain to the event or object",


Consistency: The vast majority of descriptions in this dictionary end with a full stop (period). This comment applies to all of the descriptions in this PR.

Good catch.

davemcatcisco · 2024-11-21T11:42:09Z

dictionary.json

@@ -1781,6 +1781,12 @@
      "type": "domain_contact",
      "is_array": true
    },
+    "domains": {
+      "caption": "Domains",
+      "description": "The domains that pertain to the event or object",


The word domains is quite general in meaning, e.g. I work in the security domain. Presumably the meaning in our dictionary with be DNS domains. Consider if this attribute would be more appropriately named domain_names or dns_domains.

The fact that domains is general in meaning might be a good thing, then we have a higher likelihood that we can re-use that dictionary attribute. Maybe we should add See specific usage. to the description.

The fact that domains is general in meaning might be a good thing, then we have a higher likelihood that we can re-use that dictionary attribute. Maybe we should add See specific usage. to the description.

That's a very constructive way of seeing the generality. I like it.

The fact that domains is general in meaning might be a good thing, then we have a higher likelihood that we can re-use that dictionary attribute. Maybe we should add See specific usage. to the description.

Agreed on making the dictionary description generalized and using the more specific description where applied. This is a common practice that has worked very well to better future-proof OCSF.

Yes, we've had many discussions on domain, subdomain, path, hostname, suffix etc. domain is in the dictionary already and in use. We have a subdomain and subdomains array and so I want to be consistent with that. Interestingly in URL we have an example that tries to indicate what we mean by domain as a string, but it could be a see specific usage to be more general. In the email class, it should be similar in meaning to the URL meaning.

The dictionary description is almost tautological but I added See specific usage. to both domain and domains. Note that the descriptions for subdomain and subdomains are more specific, and relate to URLs.

Note that this now creates warnings for about 5 or 6 other classes where domain is used. I don't want to only do See specific usage. for domains which is new, and not do it for domain but that is the side effect that will need to be addressed. In a separate PR though, as we need to give it some thought as to how to specifically define it for those classes.

davemcatcisco · 2024-11-21T11:46:24Z

dictionary.json

@@ -3083,6 +3095,11 @@
      "description": "The description of the event/finding, as defined by the source.",
      "type": "string_t"
    },
+    "message_trace_uid": {
+      "caption": "Message Trace UID",
+      "description": "The identifier that tracks a message that travels through multiple points of a messaging service.",


Would it be helpful to give examples of what this maps to in real email products and/or protocol exchanges? One of the challenges I often have when working with the OCSF schema is that it's not clear to me what exactly is intended to go in a particular attribute.

@davemcatcisco I agree, examples help! @pagbabian-splunk if you choose to add an example to this description, here is one from o365 Message Tracing: MessageTraceId = ae4ad8f6-7613-411c-e67e-08cfc740629

With Adam's proposal we will have a trace object (and profile) and a trace.uid within it. Are we trying to keep this attribute at the highest level?

Yes, I looked at that first, but I didn't want to use a trace object for this - in fact if I wanted to be more specific to Microsoft (which I'm trying to avoid), the MessageTraceId would be the source and it is directly in the event. Adam's PR is mirroring the OTel Trace concept which is related but likely its description and definitely usage will be different. MS has their ID looking like a UUID/GUID but I don't want to be that specific given there may be other implementations.

I'm going to add Mike's Microsoft link example to a references for the attribute, although I will say "For example..." so as to not make it authoritative.

davemcatcisco · 2024-11-21T11:52:35Z

CHANGELOG.md

@@ -74,6 +74,7 @@ Thankyou! -->
    1. Added new `11: Basic Authentication` enum value to `auth_protocol_id`. #1239
    1. Added `values` as an array of `string_t`. #1251
    1. Added `kernel_release` as a `string_t`.
+    1. Added `domains` `files` `urls` and `message_trace_uid`. #1259


Put commas between the attribute names, and optionally put an Oxford comma before the and. This comment applies to other additions to CHANGELOG.md in this PR.

Aniak5 · 2024-11-22T15:08:00Z

dictionary.json

@@ -2083,6 +2089,12 @@
      "description": "The result of the file change. It should contain the new values of the changed attributes.",
      "type": "file"
    },
+    "files": {
+      "caption": "Files",
+      "description": "The files that are part of the event or object",


Nit: Add a period to the dscription.

Aniak5 · 2024-11-22T15:52:42Z

events/network/email_activity.json

+    "message_trace_uid": {
+      "group": "primary",
+      "requirement": "recommended"
+    },
    "smtp_hello": {


Do we maybe want to make this more generic since we are encapsulating more than just SMTP with this event class? Maybe using command?

SMTP Commands: HELO, MAIL FROM, RCPT TO, DATA, QUIT, etc. POP3 Commands: USER, PASS, RETR, DELE, QUIT, etc. IMAP Commands: LOGIN, SELECT, FETCH, LOGOUT, etc.

The only issue I see is that if we make it smtp_command or just command we will have a value returned but we won't know which SMTP command it was. If we want to cover all commands, we'd probably want to have an SMTP Command object and objects for POP3 and IMAP. Or a single object such as Email Command that includes an enum that describes what the command is. Please think about that @Aniak5 and we can have a separate PR for it, but I'll just leave the current smtp_hello intact for now.

Aniak5 · 2024-11-22T15:55:54Z

events/network/email_activity.json

@@ -2,7 +2,7 @@
  "uid": 9,
  "caption": "Email Activity",
  "category": "network",
-  "description": "Email events report activities of emails.",
+  "description": "Email Activity events report SMTP protocol and email activities including those with embedded URLs and files. See the <code>Email</code> object for details.",
  "extends": "base_event",
  "name": "email_activity",
  "attributes": {


We should probably add a field to represent protocol so we can specify SMTP, POP3, IMAP, etc. We could either create a new protocol field or change the description of protocol_name in the dictionary and add See specific usage.

Let's do this at the same time as figuring out the SMTP Command approach since they are related, and we haven't really had much other discussion about it.

pagbabian-splunk · 2024-11-22T16:03:38Z

Good suggestions all - I will update and convert from DRAFT.

…ific usage.` for the domain and domains attributes in the dictionary. Added another references[] to the dictionary definition of message_trace_uid. Updated O365 to Office 365 in email.json Trace references[] to be consistent. Signed-off-by: Paul Agbabian <[email protected]>

Signed-off-by: Paul Agbabian <[email protected]>

pagbabian-splunk added 8 commits October 4, 2024 11:59

Removed the constraint from group_managenment.

a76f087

Signed-off-by: Paul Agbabian <[email protected]>

Merge branch 'main' of https://github.com/ocsf/ocsf-schema into main

cbe6ff6

Merge branch 'main' of https://github.com/ocsf/ocsf-schema into main

d15e704

Merge branch 'main' of https://github.com/ocsf/ocsf-schema into main

294a294

Merge branch 'main' of https://github.com/ocsf/ocsf-schema into main

f2b1d72

Merge branch 'main' of https://github.com/ocsf/ocsf-schema into main

7103620

Merge branch 'main' of https://github.com/ocsf/ocsf-schema into main

b197e14

pagbabian-splunk requested review from floydtree, Aniak5, mikeradka, zschmerber, jonrau-at-queryai and davemcatcisco as code owners November 20, 2024 21:56

pagbabian-splunk marked this pull request as draft November 20, 2024 21:57

pagbabian-splunk added 4 commits November 20, 2024 14:01

Added changed for PR #1259

df7fc18

Signed-off-by: Paul Agbabian <[email protected]>

removed the optional tag for email_uid as it was causing the validati…

e69358f

…on to fail!! Signed-off-by: Paul Agbabian <[email protected]>

Added the constraint and relaxed requirement to the email object.

4e17230

Signed-off-by: Paul Agbabian <[email protected]>

mikeradka previously approved these changes Nov 20, 2024

View reviewed changes

davemcatcisco requested changes Nov 21, 2024

View reviewed changes

Aniak5 reviewed Nov 22, 2024

View reviewed changes

mikeradka self-requested a review November 22, 2024 17:43

pagbabian-splunk dismissed mikeradka’s stale review via f92f36a November 22, 2024 23:59

pagbabian-splunk marked this pull request as ready for review November 23, 2024 00:01

Merge branch 'main' into email_update

eddefe5

Signed-off-by: Paul Agbabian <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Email object and Email Activity updates. Deprecate Email URL Activity and Email File Activity. #1259

Email object and Email Activity updates. Deprecate Email URL Activity and Email File Activity. #1259

pagbabian-splunk commented Nov 20, 2024

mikeradka left a comment

davemcatcisco left a comment

davemcatcisco Nov 21, 2024

pagbabian-splunk Nov 22, 2024

davemcatcisco Nov 21, 2024

Aniak5 Nov 22, 2024

davemcatcisco Nov 22, 2024

mikeradka Nov 22, 2024 •

edited

Loading

pagbabian-splunk Nov 22, 2024 •

edited

Loading

pagbabian-splunk Nov 22, 2024 •

edited

Loading

davemcatcisco Nov 21, 2024

mikeradka Nov 21, 2024

Aniak5 Nov 22, 2024

pagbabian-splunk Nov 22, 2024 •

edited

Loading

pagbabian-splunk Nov 22, 2024

davemcatcisco Nov 21, 2024

Aniak5 Nov 22, 2024

Aniak5 Nov 22, 2024

pagbabian-splunk Nov 22, 2024

Aniak5 Nov 22, 2024

pagbabian-splunk Nov 22, 2024

pagbabian-splunk commented Nov 22, 2024

Email object and Email Activity updates. Deprecate Email URL Activity and Email File Activity. #1259

Are you sure you want to change the base?

Email object and Email Activity updates. Deprecate Email URL Activity and Email File Activity. #1259

Conversation

pagbabian-splunk commented Nov 20, 2024

Related Issue: N/A

Description of changes:

mikeradka left a comment

Choose a reason for hiding this comment

davemcatcisco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikeradka Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

pagbabian-splunk Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

pagbabian-splunk Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pagbabian-splunk Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pagbabian-splunk commented Nov 22, 2024

mikeradka Nov 22, 2024 •

edited

Loading

pagbabian-splunk Nov 22, 2024 •

edited

Loading

pagbabian-splunk Nov 22, 2024 •

edited

Loading

pagbabian-splunk Nov 22, 2024 •

edited

Loading