[SIEM] Office 365 module #16386

adriansr · 2020-02-18T16:16:46Z

What does this PR do?

This includes a new fileset, o365.audit, that uses the o365audit input
to ingest logs using the Office 365 Management API.

TODO

Map more API types.
Validate ECS mappings for SIEM app. (Looking good to me.)
Documentation.
Simplified testing logs.
Dashboard.
Closes [Meta] Module to ingest Office365 audit events #16196
Depends New input for Office 365 audit logs #16244

elasticmachine · 2020-02-18T16:16:50Z

Pinging @elastic/siem (Team:SIEM)

adriansr · 2020-02-18T16:20:47Z

Forgot to make it a draft PR.

This includes the changes in #16244

x-pack/filebeat/module/o365/audit/config/pipeline.js

andrewkroh · 2020-02-25T02:18:40Z

@leehinman can you please take a look at the ECS mappings in this PR.

adriansr · 2020-02-27T15:32:53Z

@andrewkroh @leehinman this is still a WIP (should be a draft, I clicked the wrong button and made it a PR), but I will push the final mappings soon.

adriansr · 2020-03-02T12:20:26Z

Ready for final reviews / ECS validation. I've mapped all the API schemas that made sense to me. Please have a look only at the last commit (483e3bb22) as the rest is just the o365audit input which is in review at #16244.

Docs / dashboards will be added on a separate PR after this is merged.

leehinman

Looking really good, couple of minor things and questions below. I did notice that we could probably make use of the new event.category, iam. Not sure if we want to do that now, or after ECS 1.5 is officially released.

leehinman · 2020-03-02T14:32:29Z

x-pack/filebeat/module/o365/audit/_meta/fields.yml

+      type: keyword
+
+    - name: CreationTime
+      type: keyword


should this be type date?

This is the original date from the API, which is already parsed to date as @timestamp. I'd like to keep the fields under o365.audit as raw as possible.

leehinman · 2020-03-02T14:33:00Z

x-pack/filebeat/module/o365/audit/_meta/fields.yml

+      type: keyword
+
+    - name: ClientIPAddress
+      type: keyword


should this be type ip?

In that case we risk that if the API provides something that is not a valid IP, it won't be possible to ingest the event. This is already converted to type ip in client.ip.

leehinman · 2020-03-02T14:35:27Z

x-pack/filebeat/module/o365/audit/_meta/fields.yml

+    - name: ClientInfoString
+      type: keyword
+
+    - name: ClientIP


Are the ClientIP & ClientIPAddress fields different?

Not 100% sure.

ClientIP:

The IP address of the device that was used when the activity was logged. The IP address is displayed in either an IPv4 or IPv6 address format.

For some services, the value displayed in this property might be the IP address for a trusted application (for example, Office on the web apps) calling into the service on behalf of a user and not the IP address of the device used by person who performed the activity.

Also, for Azure Active Directory-related events, the IP address isn't logged and the value for the ClientIP property is null.

ClientIPAddress:

The IP address of the device that was used when the operation was logged. The IP address is displayed in either an IPv4 or IPv6 address format.

What I did is set client.ip from both. Keeping the value from ClientIPAddress if it exists, as I understand sometimes ClientIP can be the IP of an intermediate service.

In my log samples, the few times ClientIPAddress exists, it has the same value as ClientIP (::1, which is useless).

leehinman · 2020-03-02T14:49:00Z

x-pack/filebeat/module/o365/audit/config/pipeline.js

+            {from: 'o365audit.SharePointMetaData.From', to: 'user.id'},
+            {from: 'o365audit.SharePointMetaData.FileName', to: 'file.name'},
+            {from: 'o365audit.SharePointMetaData.FilePathUrl', to: 'url.original'},
+            {from: 'o365audit.SharePointMetaData.UniqueId', to: 'file.inode'},


is this maybe a duplicate? UniqueId & UniqueID?

It's documented as UniqueId but the API is producing UniqueID. Was just being a little too defensive in here, just in case they realize the discrepancy and change the generated field instead of the docs.

leehinman · 2020-03-02T14:57:52Z

x-pack/filebeat/module/o365/audit/config/pipeline.js

+        if (maxSeverity > -1) {
+            evt.Put("event.severity", maxSeverity);
+        }
+        evt.Put("event.outcome", allowed ||  isBlockOverride(evt)? "success" : "failure");


I think if allowed is true we want to set event.outcome to "success" not true. .... Oh, I see where this gets mapped from true -> success later. I wonder if it would be better to have the second half or the || statement also set a boolean value. That might make it more consistent and easier to follow a year from now.

The operator precedence of javascript is different from what you might expect. This expression is equivalent to:

( allowed || isBlockOverride(evt) )? "success" : "failure"

I've added parentheses around it to make it more clear.

leehinman · 2020-03-02T15:45:42Z

x-pack/filebeat/module/o365/audit/test/01-exchange-admin.log-expected.json

+        "organization.name": "testsiem.onmicrosoft.com",
+        "server.address": "HE1PR0102MB3228 (15.20.2707.017)",
+        "service.type": "o365",
+        "user.hash": "NT AUTHORITY\\SYSTEM (Microsoft.Exchange.ServiceHost)",


hmmm. this doesn't look like a hash to me.

Right. I was too eager to try to map as many fields as possible. The source field "UserKey" is really just an alternate user ID (also with different meanings across systems). I think it's better not to map it to ECS at all.

x-pack/filebeat/module/o365/audit/test/02-exchange-item.log-expected.json

leehinman · 2020-03-02T15:58:34Z

x-pack/filebeat/module/o365/audit/test/11-dlp-sharepoint.log-expected.json

+        "event.type": "access",
+        "file.inode": "9cc7be1c-dd5a-4895-b7cb-757de6d51b42",
+        "file.name": "Customers Financial Data.docx",
+        "file.owner": "Alan Smithee",


might be nice to have this in related.user as well

Good idea. Done.

leehinman · 2020-03-02T16:03:59Z

x-pack/filebeat/module/o365/audit/test/22-yammer.log-expected.json

+        "@timestamp": "2020-02-28T09:42:45.000Z",
+        "client.address": "79.159.10.151",
+        "client.ip": "79.159.10.151",
+        "event.action": "GroupCreation",


groupcreate would probably be event.category = iam, event.type = ["create", "group"]

Thanks for the suggestion. I wasn't aware of this new category value.

leehinman · 2020-03-02T16:06:56Z

x-pack/filebeat/module/zeek/connection/test/connection-json.log-expected.json

@@ -165,4 +165,4 @@
        "zeek.connection.state_message": "No SYN seen, just midstream traffic (a 'partial connection' that was not later closed).",
        "zeek.session_id": "Cc6NJ3GRlfjE44I3h"
    }
-]
+]


Probably shouldn't be part of this PR.

andrewkroh · 2020-03-04T16:10:14Z

Please add an overview dashboard to accompany the module and highlight some of the key signals coming from the data (activity over time, geo data, failures, etc). Thanks!

andrewkroh

LGTM. The pipeline really looks comprehensive. 👍

andrewkroh · 2020-03-10T15:03:09Z

x-pack/filebeat/module/o365/audit/config/pipeline.js

+       var outcome = evt.Get("event.outcome");
+       // As event.type is an array, this sets both the traditional
+       // "authentication_success"/"authentication_failure"
+       // and the soon to be ECS standard "start".


event.type: start is no longer "soon to be" 😄 .

adriansr · 2020-03-15T15:18:43Z

jenkins, test this

This includes a new fileset, o365.audit, that uses the o365audit input to ingest logs using the Office 365 Management API.

Avoid error when trying to dissect.

Max retention is 7 days, that's 168h, not 178.

This patches the o365audit input to accept resource and authentication_endpoint configuration options without a scheme.

leehinman

2 small items. But I would be OK with this going in as is.

x-pack/filebeat/module/o365/audit/test/15-azuread-sts-logon.log-expected.json

leehinman · 2020-03-19T15:30:21Z

x-pack/filebeat/module/o365/audit/test/22-yammer.log-expected.json

+        "event.module": "o365",
+        "event.outcome": "success",
+        "event.provider": "Yammer",
+        "event.type": "group",


would it be possible to have "creation" added to event.type. So it would be event.cateogory = iam, event.type = "creation, group"

This includes a new fileset, o365.audit, that uses the o365audit input to ingest logs using the Office 365 Management API. (cherry picked from commit 1cc1d33)

adriansr added enhancement in progress Pull request is currently in progress. review Filebeat Filebeat Team:SIEM labels Feb 18, 2020

adriansr requested a review from a team as a code owner February 18, 2020 16:16

adriansr mentioned this pull request Feb 18, 2020

[Meta] Module to ingest Office365 audit events #16196

Closed

18 tasks

andrewkroh reviewed Feb 25, 2020

View reviewed changes

x-pack/filebeat/module/o365/audit/config/pipeline.js Outdated Show resolved Hide resolved

x-pack/filebeat/module/o365/audit/config/pipeline.js Outdated Show resolved Hide resolved

andrewkroh added the ecs label Feb 25, 2020

andrewkroh requested a review from leehinman February 25, 2020 02:17

adriansr force-pushed the feature_o365_module branch 2 times, most recently from 62f0ff8 to c77f2d7 Compare March 2, 2020 11:05

adriansr removed the in progress Pull request is currently in progress. label Mar 2, 2020

adriansr changed the title ~~[Draft][SIEM] Office 365 module~~ [SIEM] Office 365 module Mar 2, 2020

adriansr force-pushed the feature_o365_module branch 3 times, most recently from 774e66b to 483e3bb Compare March 2, 2020 12:18

leehinman requested changes Mar 2, 2020

View reviewed changes

adriansr force-pushed the feature_o365_module branch from ebc53a0 to 7ff69a6 Compare March 5, 2020 09:53

andrewkroh approved these changes Mar 10, 2020

View reviewed changes

adriansr added 3 commits March 18, 2020 11:12

New module o365 for Office 365 log ingestion

8aa512b

This includes a new fileset, o365.audit, that uses the o365audit input to ingest logs using the Office 365 Management API.

address review comments

4eb820a

Set related.user

ee4d2ac

adriansr added 10 commits March 18, 2020 11:12

Better mapping for Yammer event fields

d1c4934

Revert changes to zeek golden file

c969d6f

Convert numeric user.id to string

454ffeb

Avoid error when trying to dissect.

Support ip:port in client.address

277c9a5

Correct comment about ECS event.type

c8a6b62

Correct o365audit docs

02ea070

Max retention is 7 days, that's 168h, not 178.

Append https scheme to endpoint URLs if needed

2712947

This patches the o365audit input to accept resource and authentication_endpoint configuration options without a scheme.

Added sample dashboard

3adebc7

Add dashboard screenshot to docs

8181a9b

Document setup process

4a6ea69

adriansr force-pushed the feature_o365_module branch from 4949b21 to 4a6ea69 Compare March 18, 2020 11:39

Store dashboard in correct format

3388089

andrewkroh requested a review from leehinman March 19, 2020 14:36

leehinman reviewed Mar 19, 2020

View reviewed changes

leehinman approved these changes Mar 19, 2020

View reviewed changes

improve iam / group categorisation

2196d35

leehinman approved these changes Mar 19, 2020

View reviewed changes

adriansr merged commit 1cc1d33 into elastic:master Mar 19, 2020

adriansr mentioned this pull request Mar 20, 2020

Cherry-pick #16386 to 7.x: [SIEM] Office 365 module #17158

Merged

5 tasks

adriansr added the v7.7.0 label Mar 20, 2020

adriansr mentioned this pull request Mar 20, 2020

Add missing CHANGELOG entry for o365 module #17159

Merged

adriansr added a commit that referenced this pull request Mar 20, 2020

[SIEM] Office 365 module (#16386) (#17158)

c03dec4

This includes a new fileset, o365.audit, that uses the o365audit input to ingest logs using the Office 365 Management API. (cherry picked from commit 1cc1d33)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] Office 365 module #16386

[SIEM] Office 365 module #16386

adriansr commented Feb 18, 2020 •

edited

Loading

elasticmachine commented Feb 18, 2020

adriansr commented Feb 18, 2020 •

edited

Loading

andrewkroh commented Feb 25, 2020

adriansr commented Feb 27, 2020

adriansr commented Mar 2, 2020

leehinman left a comment

leehinman Mar 2, 2020

adriansr Mar 3, 2020

leehinman Mar 2, 2020

adriansr Mar 3, 2020

leehinman Mar 2, 2020

adriansr Mar 3, 2020 •

edited

Loading

leehinman Mar 2, 2020

adriansr Mar 3, 2020

leehinman Mar 2, 2020

adriansr Mar 3, 2020 •

edited

Loading

leehinman Mar 2, 2020

adriansr Mar 3, 2020

leehinman Mar 2, 2020

adriansr Mar 3, 2020

leehinman Mar 2, 2020

adriansr Mar 3, 2020

leehinman Mar 2, 2020

adriansr Mar 3, 2020

andrewkroh commented Mar 4, 2020

andrewkroh left a comment

andrewkroh Mar 10, 2020

adriansr commented Mar 15, 2020

leehinman left a comment

leehinman Mar 19, 2020

[SIEM] Office 365 module #16386

[SIEM] Office 365 module #16386

Conversation

adriansr commented Feb 18, 2020 • edited Loading

What does this PR do?

TODO

elasticmachine commented Feb 18, 2020

adriansr commented Feb 18, 2020 • edited Loading

andrewkroh commented Feb 25, 2020

adriansr commented Feb 27, 2020

adriansr commented Mar 2, 2020

leehinman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr Mar 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr Mar 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Mar 4, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr commented Mar 15, 2020

leehinman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr commented Feb 18, 2020 •

edited

Loading

adriansr commented Feb 18, 2020 •

edited

Loading

adriansr Mar 3, 2020 •

edited

Loading

adriansr Mar 3, 2020 •

edited

Loading