[ML] Adding v3 modules for Security_Linux and Security_Windows and De…

…precating v1 + v2 (#131166) * consolidate Security ML Modules * removal of auditbeat host processes ecs module * removing siem_winlogbeat_auth after consolidating into windows_security * renamed to avoid job collisions * Update recognize_module.ts removed references to deprecated v1 modules which no longer exist * test fixes remove references to deprecated module and modify module names to match the latest v3 modules being committed. * Update recognize_module.ts think this is what the linter wants * deprecating winlogbeat and auditbeat modules * fixes test post-deprecation of modules * fixes typo in test * revert linting changes * revert linting changes pt2 * fixing test in setup_module.ts * ml module refactor * manifest, job, and datafeed cleanup based on PR feedback * commenting out security solution tests for ML Modules * modified ml module tests and job descriptions * Update datafeed_auth_high_count_logon_events_for_a_source_ip.json added test for existence of source.ip field per #131376 * Update datafeed_auth_high_count_logon_events_for_a_source_ip.json formatting * descriptions standardized descriptions between Linux and Windows jobs; removed the term "services" from the rare process jobs because it has a special meaning under Windows and is the target of a different job; added a sentence to the sudo job description, I think this was a stub description that never got fleshed out when it was developed. * tags added job tags * tags added Linux job tags * tags * linting remove a dup json element * Update v3_windows_anomalous_script.json add the Security: Windows prefix which was missing * Update v3_linux_anomalous_network_activity.json missing bracket * Update v3_windows_anomalous_script.json the prefix was in the wrong place Co-authored-by: Craig Chamberlain <[email protected]> Co-authored-by: Kibana Machine <[email protected]>
elastic · May 18, 2022 · f85c39e · f85c39e
1 parent 31bb2c7
commit f85c39e
Show file tree

Hide file tree

Showing 148 changed files with 2,220 additions and 3,314 deletions.
diff --git a/...ditbeat_process_hosts_ecs/kibana/dashboard/ml_auditbeat_hosts_process_event_rate_ecs.json b/...ditbeat_process_hosts_ecs/kibana/dashboard/ml_auditbeat_hosts_process_event_rate_ecs.json
diff --git a/...auditbeat_process_hosts_ecs/kibana/dashboard/ml_auditbeat_hosts_process_explorer_ecs.json b/...auditbeat_process_hosts_ecs/kibana/dashboard/ml_auditbeat_hosts_process_explorer_ecs.json
diff --git a/...ules/auditbeat_process_hosts_ecs/kibana/search/ml_auditbeat_hosts_process_events_ecs.json b/...ules/auditbeat_process_hosts_ecs/kibana/search/ml_auditbeat_hosts_process_events_ecs.json
diff --git a/..._hosts_ecs/kibana/visualization/ml_auditbeat_hosts_process_event_rate_by_process_ecs.json b/..._hosts_ecs/kibana/visualization/ml_auditbeat_hosts_process_event_rate_by_process_ecs.json
diff --git a/...process_hosts_ecs/kibana/visualization/ml_auditbeat_hosts_process_event_rate_vis_ecs.json b/...process_hosts_ecs/kibana/visualization/ml_auditbeat_hosts_process_event_rate_vis_ecs.json
diff --git a/...eat_process_hosts_ecs/kibana/visualization/ml_auditbeat_hosts_process_occurrence_ecs.json b/...eat_process_hosts_ecs/kibana/visualization/ml_auditbeat_hosts_process_occurrence_ecs.json
diff --git a/...ck/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_hosts_ecs/logo.json b/...ck/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_hosts_ecs/logo.json
diff --git a/...lugins/ml/server/models/data_recognizer/modules/auditbeat_process_hosts_ecs/manifest.json b/...lugins/ml/server/models/data_recognizer/modules/auditbeat_process_hosts_ecs/manifest.json
diff --git a/.../modules/auditbeat_process_hosts_ecs/ml/datafeed_hosts_high_count_process_events_ecs.json b/.../modules/auditbeat_process_hosts_ecs/ml/datafeed_hosts_high_count_process_events_ecs.json
diff --git a/...izer/modules/auditbeat_process_hosts_ecs/ml/datafeed_hosts_rare_process_activity_ecs.json b/...izer/modules/auditbeat_process_hosts_ecs/ml/datafeed_hosts_rare_process_activity_ecs.json
diff --git a/...ecognizer/modules/auditbeat_process_hosts_ecs/ml/hosts_high_count_process_events_ecs.json b/...ecognizer/modules/auditbeat_process_hosts_ecs/ml/hosts_high_count_process_events_ecs.json
diff --git a/...ta_recognizer/modules/auditbeat_process_hosts_ecs/ml/hosts_rare_process_activity_ecs.json b/...ta_recognizer/modules/auditbeat_process_hosts_ecs/ml/hosts_rare_process_activity_ecs.json
diff --git a/x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/manifest.json b/x-pack/plugins/ml/server/models/data_recognizer/modules/security_auth/manifest.json
@@ -41,6 +41,10 @@
     {
       "id": "auth_rare_user",
       "file": "auth_rare_user.json"
+    },
+    {
+      "id": "suspicious_login_activity",
+      "file": "suspicious_login_activity.json"
     }
   ],
   "datafeeds": [
@@ -73,6 +77,11 @@
       "id": "datafeed-auth_rare_user",
       "file": "datafeed_auth_rare_user.json",
       "job_id": "auth_rare_user"
+    },
+    {
+      "id": "datafeed-suspicious_login_activity",
+      "file": "datafeed_suspicious_login_activity.json",
+      "job_id": "suspicious_login_activity"
     }
   ]
 }
diff --git a/...nizer/modules/security_auth/ml/datafeed_auth_high_count_logon_events_for_a_source_ip.json b/...nizer/modules/security_auth/ml/datafeed_auth_high_count_logon_events_for_a_source_ip.json
@@ -5,19 +5,16 @@
   ],
   "max_empty_searches": 10,
   "query": {
-    "bool": {
-      "filter": [
-        {
-          "term": {
-            "event.category": "authentication"
-          }
-        },
-        {
-          "term": {
-            "event.outcome": "success"
-          }
+        "bool": {
+            "filter": [{"exists": {"field": "source.ip"}}],
+            "must": [
+                {"bool": {
+                    "should": [
+                    {"term": {"event.category": "authentication"}},
+                    {"term": {"event.outcome": "success"}}
+                    ]
+                }}
+            ]
         }
-      ]
-    }
   }
 }
diff --git a/...tafeed_suspicious_login_activity_ecs.json → ...l/datafeed_suspicious_login_activity.json b/...tafeed_suspicious_login_activity_ecs.json → ...l/datafeed_suspicious_login_activity.json
diff --git a/...uth/ml/suspicious_login_activity_ecs.json → ...ty_auth/ml/suspicious_login_activity.json b/...uth/ml/suspicious_login_activity_ecs.json → ...ty_auth/ml/suspicious_login_activity.json
diff --git a/x-pack/plugins/ml/server/models/data_recognizer/modules/security_linux/logo.json b/x-pack/plugins/ml/server/models/data_recognizer/modules/security_linux/logo.json
@@ -1,3 +1,3 @@
 {
   "icon": "logoSecurity"
-}
+}