[SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields #65740

marshallmain · 2020-05-07T17:51:45Z

Summary

This is a stopgap change to start integrating the SIEM signals schema with the endpoint alert schema. This schema was built using the ECS tooling and the custom yaml schemas found at https://github.com/elastic/endpoint-app-team/tree/master/custom_schemas. When the ECS tooling upgrades are finished (elastic/ecs#837 and elastic/ecs#820) we can add yml files for the signal fields and generate the entire mapping with the ECS tooling.

At some point we should move the yml files into the kibana in the SIEM folder, but when we do we should remove them from the endpoint-app-team repo so we don't have multiple copies that could be out of sync.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-05-07T17:51:47Z

Pinging @elastic/endpoint-response (Team:Endpoint Response)

marshallmain · 2020-05-07T17:52:16Z

@elasticmachine merge upstream

elasticmachine · 2020-05-08T14:20:04Z

Pinging @elastic/siem (Team:SIEM)

spong · 2020-05-08T21:40:26Z

x-pack/plugins/siem/server/lib/detection_engine/routes/index/ecs_mapping.json

@@ -2516,4 +4276,4 @@
      "refresh_interval": "5s"
    }
  }
-}
+}


Missing newline?

FrankHassanabad · 2020-05-11T13:07:04Z

@elasticmachine merge upstream

FrankHassanabad · 2020-05-11T14:17:17Z

Tested by going to this directory:

x-pack/plugins/siem/server/lib/detection_engine/scripts

And running:

./hard_reset.sh
 ./post_rule.sh 
./post_rule.sh ./rules/queries/query_with_everything.json

And then looking at each rule run and ensuring that the expected histograms and data looks correct.

FrankHassanabad

👍 Thanks for the ECS upgrade!

FrankHassanabad · 2020-05-11T17:27:47Z

@elasticmachine merge upstream

madirey · 2020-05-12T16:13:44Z

x-pack/plugins/siem/server/lib/detection_engine/routes/index/ecs_mapping.json

@@ -441,6 +444,141 @@
          }
        }
      },
+      "dll": {
+        "properties": {
+          "code_signature": {


@marshallmain Are we going to address splitting these (trusted/untrusted) in a later PR?

I'm hoping to address it with the ECS team. I think multiple code signatures is something we should address at the ECS level rather than making our own custom changes to it.

Thinking about it more, addressing at the ECS level may take some time. I'd like to merge this so we can start and then merge more changes in future PRs.

Sounds good, thanks!

marshallmain · 2020-05-13T21:01:43Z

@elasticmachine merge upstream

kibanamachine · 2020-05-13T23:03:34Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 19392bd

History

💚 Build #47032 succeeded 722e229
💔 Build #46942 failed 88dd778
💔 Build #46917 failed 147f9bb
💚 Build #46715 succeeded a6a09c8
💔 Build #46406 failed 0906b0a10a4fdae2e2991e667ee2b05ff39f58d9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

spong · 2020-05-15T17:13:54Z

@marshallmain -- you should be good to backport to 7.x now that #66221 has been merged 🚀

…t fields (elastic#65740) * update ECS schema to 1.6-dev, add endpoint alert fields * use updated endpoint team schemas * add newline * remove extra options Co-authored-by: Elastic Machine <[email protected]>

kibanamachine · 2020-05-18T00:18:44Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

kibanamachine · 2020-05-19T00:28:43Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

…t fields (#65740) (#66789) * update ECS schema to 1.6-dev, add endpoint alert fields * use updated endpoint team schemas * add newline * remove extra options Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

elasticmachine · 2021-09-23T14:34:57Z

Pinging @elastic/security-solution (Team: SecuritySolution)

update ECS schema to 1.6-dev, add endpoint alert fields

4c87bc4

marshallmain added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Response Endpoint Response Team v7.9.0 labels May 7, 2020

marshallmain requested review from peluja1012 and FrankHassanabad May 7, 2020 17:51

marshallmain requested review from a team as code owners May 7, 2020 17:51

marshallmain marked this pull request as draft May 7, 2020 22:12

FrankHassanabad added the Team:SIEM label May 8, 2020

marshallmain added 2 commits May 8, 2020 16:10

use updated endpoint team schemas

b428d0b

Merge branch 'master' into add-alert-mapping

a6a09c8

marshallmain force-pushed the add-alert-mapping branch from 0906b0a to a6a09c8 Compare May 8, 2020 20:11

spong reviewed May 8, 2020

View reviewed changes

spong mentioned this pull request May 9, 2020

[SIEM][Timeline] Rename Signals table to Alerts #65944

Closed

elasticmachine and others added 3 commits May 11, 2020 06:07

Merge branch 'master' into add-alert-mapping

147f9bb

add newline

85e5dd5

remove extra options

88dd778

FrankHassanabad approved these changes May 11, 2020

View reviewed changes

marshallmain marked this pull request as ready for review May 11, 2020 14:21

FrankHassanabad added release_note:enhancement and removed release_note:skip Skip the PR/issue when compiling release notes labels May 11, 2020

FrankHassanabad changed the title ~~Update signals index ECS mapping to 1.6-dev, add endpoint alert fields~~ [SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields May 11, 2020

Merge branch 'master' into add-alert-mapping

722e229

madirey reviewed May 12, 2020

View reviewed changes

peluja1012 approved these changes May 13, 2020

View reviewed changes

Merge branch 'master' into add-alert-mapping

19392bd

madirey approved these changes May 13, 2020

View reviewed changes

marshallmain merged commit 11d2374 into elastic:master May 14, 2020

marshallmain deleted the add-alert-mapping branch May 14, 2020 00:14

marshallmain mentioned this pull request May 15, 2020

[7.x] [SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields (#65740) #66789

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 18, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 19, 2020

marshallmain mentioned this pull request Jun 11, 2020

Remove endpoint alert fields from signal mapping #68934

Merged

2 tasks

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields #65740

[SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields #65740

marshallmain commented May 7, 2020

elasticmachine commented May 7, 2020

marshallmain commented May 7, 2020

elasticmachine commented May 8, 2020

spong May 8, 2020

FrankHassanabad commented May 11, 2020

FrankHassanabad commented May 11, 2020

FrankHassanabad left a comment

FrankHassanabad commented May 11, 2020

madirey May 12, 2020

marshallmain May 12, 2020 •

edited

Loading

marshallmain May 13, 2020

madirey May 13, 2020

marshallmain commented May 13, 2020

kibanamachine commented May 13, 2020

spong commented May 15, 2020

kibanamachine commented May 18, 2020

kibanamachine commented May 19, 2020

elasticmachine commented Sep 23, 2021

[SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields #65740

[SIEM] Update signals index ECS mapping to 1.6-dev, add endpoint alert fields #65740

Conversation

marshallmain commented May 7, 2020

Summary

Checklist

For maintainers

elasticmachine commented May 7, 2020

marshallmain commented May 7, 2020

elasticmachine commented May 8, 2020

spong May 8, 2020

Choose a reason for hiding this comment

FrankHassanabad commented May 11, 2020

FrankHassanabad commented May 11, 2020

FrankHassanabad left a comment

Choose a reason for hiding this comment

FrankHassanabad commented May 11, 2020

madirey May 12, 2020

Choose a reason for hiding this comment

marshallmain May 12, 2020 • edited Loading

Choose a reason for hiding this comment

marshallmain May 13, 2020

Choose a reason for hiding this comment

madirey May 13, 2020

Choose a reason for hiding this comment

marshallmain commented May 13, 2020

kibanamachine commented May 13, 2020

💚 Build Succeeded

History

spong commented May 15, 2020

kibanamachine commented May 18, 2020

kibanamachine commented May 19, 2020

elasticmachine commented Sep 23, 2021

marshallmain May 12, 2020 •

edited

Loading