- Stage: 3 (finished)
- Date: 2022-02-23
This RFC proposes a new top-level field set to facilitate email use cases, email.*. The email.* field set adds fields for the sender, recipient, message header fields, and other attributes of an email message typically seen logs produced by mail transfer agent (MTA) and email gateway applications.
| field | type | description |
|---|---|---|
email.origination_timestamp |
date | The date and time the email message was composed. Many email clients will fill this in automatically when the message is sent by a user. |
email.delivery_timestamp |
date | The date and time the email message was received by the service or client. |
email.from.address |
keyword (array) | Stores the from email address from the RFC5322 From: header field. |
email.sender.address |
keyword | When the from field contains more than one address or the sender and from are distinct then this field is populated. |
email.to.address |
keyword (array) | The email address of message recipient |
email.cc.address |
keyword (array) | The email address of a carbon copy (CC) recipient |
email.bcc.address |
keyword (array) | The email address of the blind carbon copy (CC) recipient(s) |
email.reply_to.address |
keyword (array) | The address that replies should be delivered to from the RFC 5322 Reply-To: header field. |
email.subject |
keyword (.text text multi-field) |
A brief summary of the topic of the message |
email.content_type |
keyword | Information about how the message is to be displayed. Typically a MIME type |
email.message_id |
wildcard | Identifier from the RFC5322 Message-ID: header field that refers to a particular version of a particular message. |
email.local_id |
keyword | Unique identifier given to the email by the source (MTA, gateway, etc.) that created the event and is not persistent across hops (for example, the X-MS-Exchange-Organization-Network-Message-Id id). |
email.direction |
keyword | Direction of the message based on the sending and receiving domains |
email.x_mailer |
keyword | What application was used to draft and send the original email. |
email.attachments |
nested | Nested object of attachments on the email. |
email.attachments.file.mime_type |
keyword | MIME type of the attachment file. |
email.attachments.file.name |
keyword | Name of the attachment file including the extension. |
email.attachments.file.extension |
keyword | Attachment file extension, excluding the leading dot. |
email.attachments.file.size |
long | Attachment file size in bytes. |
email.attachments.file.hash.* |
reuse of hash.* field set |
Field reuse of hash.* for file attachments. |
Email events may benefit from an additional ECS allowed event categorization value: event.category: email.
Email use cases stretch across all three Elastic solutions - Search, Observe, Protect. Whether it's searching for content within email, ensuring email infrastructure is operational, or detecting email-based attacks, there are many possibilities for email fields within ECS.
- Email Analytics: Hubspot, Marketo, Salesforce Pardot
- Email Server: O365 Message Tracing, Postfix
- Email Security: Barracuda, Forcepoint, Mimecast, Proofpoint
{
"EndDate": "2021-11-10T22:12:34.8196921Z",
"FromIP": "8.8.8.8",
"Index": 25,
"MessageId": "\\u003c95689d8d5e7f429390a4e3646eef75e8-JFBVALKQOJXWILKBK4YVA7APGM3DKTLFONZWCZ3FINSW45DFOJ6EAQ2ENFTWK43UL4YTCMBYGIYHYU3NORYA====@microsoft.com\\u003e",
"MessageTraceId": "ff1a64a3-cafb-41b7-1efb-08d8848aedc3",
"Organization": "testdomain.onmicrosoft.com",
"Received": "2020-11-09T04:50:06.3312635",
"RecipientAddress": "[email protected]",
"SenderAddress": "[email protected]",
"Size": 64329,
"StartDate": "2020-11-08T22:12:34.8196921Z",
"Status": "Delivered",
"Subject": "Weekly digest: Microsoft service updates",
"ToIP": null
}{
"@timestamp": 1626984241830,
"email": {
"timestamp": "2020-11-08T22:12:34.8196921Z",
"from": {
"address": [
"[email protected]"
]
},
"to": {
"address": [
"[email protected]"
]
},
"subject": "Weekly digest: Microsoft service updates",
"message_id": "\\u003c95689d8d5e7f429390a4e3646eef75e8-JFBVALKQOJXWILKBK4YVA7APGM3DKTLFONZWCZ3FINSW45DFOJ6EAQ2ENFTWK43UL4YTCMBYGIYHYU3NORYA====@microsoft.com\\u003e"
},
"event": {
"action": "delivered",
"kind": "event",
"category": [
"email",
"network"
]
}
}{
"EndDate": "2020-11-10T22:12:34.8196921Z",
"FromIP": null,
"Index": 8,
"MessageId": "\\u003c72872e16-f4c2-4eef-a393-e5621748a0ff@AS8P19vMB1605.EURP191.PROD.OUTLOOK.COM\\u003e",
"MessageTraceId": "a4bd8c4c-3a4f-427f-8952-08d8850f9c20",
"Organization": "testdomain.onmicrosoft.com",
"Received": "2020-11-10T00:28:56.3306834",
"RecipientAddress": "[email protected]",
"SenderAddress": "[email protected]",
"Size": 96627,
"StartDate": "2020-11-08T22:12:34.8196921Z",
"Status": "Delivered",
"Subject": "Undeliverable: Message Center Major Change Update Notification",
"ToIP": "8.8.8.8"
}{
"@timestamp": 1626984241830,
"email": {
"timestamp": "2020-11-10T22:12:34.8196921Z",
"from": {
"address": [
"[email protected]"
]
},
"to": {
"address": [
"[email protected]"
]
},
"subject": "Undeliverable: Message Center Major Change Update Notification",
"message_id": "\\u003c72872e16-f4c2-4eef-a393-e5621748a0ff@AS8P19vMB1605.EURP191.PROD.OUTLOOK.COM\\u003e"
},
"event": {
"action": "delivered",
"kind": "event",
"category": [
"email",
"network"
]
}
}<38>1 2021-06-24T21:00:08Z - ProofpointTAP - MSGBLK [tapmsg@21139 messageTime="2021-06-24T21:18:38.000Z" messageID="[email protected]" recipient="[email protected], [email protected]" sender="[email protected]" senderIP="192.0.2.255" phishScore="46" spamScore="4" QID="r2FNwRHF004109" GUID="c26dbea0-80d5-463b-b93c-4e8b708219ce" subject="Please find a totally safe invoice attached." quarantineRule="module.sandbox.threat" quarantineFolder="Attachment Defense" policyRoutes="default_inbound,executives" modulesRun="sandbox,urldefense,spam,pdr" headerFrom="\"A. Badguy\" <[email protected]>" headerTo="\"Clark Kent\" <[email protected]>; \"Diana Prince\" <[email protected]>" headerCC="\"Bruce Wayne\" <[email protected]>" headerReplyTo="null" toAddresses="[email protected],[email protected]" ccAddresses="[email protected]" fromAddress="[email protected]" replyToAddress="null" clusterId="pharmtech_hosted" messageParts="[{\"contentType\":\"text/plain\",\"disposition\":\"inline\",\"filename\":\"text.txt\",\"md5\":\"008c5926ca861023c1d2a36653fd88e2\",\"oContentType\":\"text/plain\",\"sandboxStatus\":\"unsupported\",\"sha256\":\"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281\"},{\"contentType\":\"application/pdf\",\"disposition\":\"attached\",\"filename\":\"Invoice for Pharmtech.pdf\",\"md5\":\"5873c7d37608e0d49bcaa6f32b6c731f\",\"oContentType\":\"application/pdf\",\"sandboxStatus\":\"threat\",\"sha256\":\"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\"}]" xmailer="Spambot v2.5"]
{
"@timestamp": "2021-06-24T21:00:08Z",
"email": {
"timestamp": "2021-06-24T21:18:38.000Z",
"message_id": "[email protected]",
"local_id": "c26dbea0-80d5-463b-b93c-4e8b708219ce",
"to": {
"address": [
"[email protected]",
"[email protected]"
]
},
"cc": {
"address": [
"[email protected]"
]
},
"from": {
"address": [
"[email protected]"
]
},
"sender": {
"address": "[email protected]"
},
"subject": "Please find a totally safe invoice attached.",
"reply_to": {
"address": "null"
},
"x_mailer": "Spambot v2.5",
"attachments": [
{
"file": {
"mime_type": "application/pdf",
"name": "Invoice for Pharmtech.pdf",
"extension": "pdf",
"hash": {
"md5": "5873c7d37608e0d49bcaa6f32b6c731f",
"sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
}
}
}
]
},
"event": {
"id": "c26dbea0-80d5-463b-b93c-4e8b708219ce",
"kind": "event",
"category": "email",
"action": "MSGBLK"
},
"source": {
"address": "192.0.2.255",
"ip": "192.0.2.255"
}
}datetime=2021-05-26T16:47:41+0100|aCode=7O7I7MvGP1mj8plHRDuHEA|acc=C0A0|SpamLimit=0|IP=123.123.123.123|Dir=Internal|MsgId=<[email protected]>|Subject=\message subject\|[email protected]|[email protected]|[email protected]|SpamInfo=[]|Act=Acc|TlsVer=TLSv1|Cphr=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|SpamProcessingDetail={"spf":{"info":"SPF_FAIL","allow":true},"dkim":{"info":"DKIM_UNKNOWN","allow":true}}|SpamScore=1
{
"@timestamp": "2021-05-26T16:47:41+0100",
"source": {
"address": "123.123.123.123",
"ip": "123.123.123.123"
},
"email": {
"message_id": "<[email protected]>",
"from": {
"address": [
"[email protected]"
]
},
"to": {
"address": [
"[email protected]"
]
},
"subject": "message subject",
"direction": "internal"
},
"tls": {
"cipher": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
"version": "1.0",
"version_protocol": "tls"
}
}This is a new field set, and the changes introduced will not affect existing ECS implementations.
Integrations or other data sources mapping to ECS will need to map their original events to the new fields.
The fields proposed in this document focus on an email message's content but not on specific fields for email protocols. However, should protocols like SMTP, POP3, IMAP, etc., be represented in ECS?
For example, users may need to compare the email address from the SMTP (envelope) sender to the From: header email address.
Resolution: Focus on email message content in this initial phase. Additional protocol details can be added later on.
Does the initial set of email fields need to consider observability and email monitoring use cases, for example, spam, metrics, deliverables, and logging?
Resolution: This initial field set focuses on email message content.
Should a new event.category field (email) be created, and, if so, which event.type values should be used for the email category?
Resolution: Propose to add event.category: email and make info an expected event type for the category.
Should the display name be captured separately from the email address for senders and recipients? If so, how do we accomplish this in a document while keeping the 1:1 of a display name to email address?
Resolution: Initially, this proposal considered using nested types to allows arrays of objects containing both the email address and display name for the to, cc, and bcc recipients. However, after more consideration of the limitations to using nested fields types and limited support for nested fields in Kibana, that decision was reversed.
Should fields intended to capture details around spam processing like sender policy framework (SPF), domainkeys identified mail (DKIM), or domain-based message authentication, reporting, and conformance (DMARC) be in scope for this proposal as well?
Resolution: This initial field set focuses on email message content.
The following are the people that consulted on the contents of this RFC.
- @ebeahan | Co-author
- @P1llus | Co-author, subject matter expert
- @jamiehynds | Co-sponsor
- @devonakerr | Co-sponsor
- Stage 1 (formerly proposal stage): #999
- RFC ID correction: #1157
- Stage 1 (draft): #1219
- Stage 2 (candidate): #1593
- Stage 3 (finished): https://github.com/elastic/ecs/pull/YYYY