Fix spelling and formatting issues

Alerting/Sample Watches/README.md

@@ -2,22 +2,22 @@
 
 ## Overview
 
-This package provides a collection of example  watches.  These watches have been developed for the purposes of POC's and demonstrations.  Each makes independent assumptions as to the data structure, volume and mapping.  For each watch a description, with assumptions is provided, in addition to a mapping file.  Whilst functionally tested, these watches have not been tested for effectiveness or query performance in production environments.  The reader is therefore encouraged to test and review all watches with production data volumes prior to deployment.
+This package provides a collection of example watches.  These watches have been developed for the purposes of POC's and demonstrations.  Each makes independent assumptions as to the data structure, volume and mapping.  For each watch a description, with assumptions is provided, in addition to a mapping file.  Whilst functionally tested, these watches have not been tested for effectiveness or query performance in production environments.  The reader is therefore encouraged to test and review all watches with production data volumes prior to deployment.
 
-# Generic Assumptions
+## Generic Assumptions
 
 * Elasticsearch 6.2 + x-pack
 * All watches use the log output for purposes of testing. Replace with output e.g. email, as required.
-* Painless scripts, located within the "scripts" folder of each watch, must be indexed first.  
+* Painless scripts, located within the "scripts" folder of each watch, must be indexed first.
 * All watches assume Watcher is running in the same cluster as that in which the relevant data is hosted.  They all therefore use the search input.  In a production deployment this is subject to change i.e. a http input maybe required.
 
-# Structure
+## Structure
 
 In each watch directory the following is provided:
 
 * README - describes the watch including any assumptions regards mapping, data structure and behaviour.
 * mapping.json - An re-usable mapping which is also appropriate for the test data provided.
-* watch.json - Body of the watch. Used in the below tests. 
+* watch.json - Body of the watch. Used in the below tests.
 * /tests - Directory of tests.  Each test is defined as JSON file.  See Below.
 * /scripts - Directory of painless scripts utilised by the watch.
 
@@ -32,11 +32,11 @@ The parent directory includes the following utility scripts:
 
 If username, password, and protocol are not specified, the above scripts assume the x-pack default of "elastic", "changeme", and "http" respectively.
 
-# Watches
+## Watches
 
 * Errors in log files - A watch which alerts if errors are present in a log file. Provides example errors as output.
 * Port Scan - A watch which aims to detect and alert if a server established a high number of connections to a destination across a large number of ports.
-* Social Media Trending - A watch which alerts if a social media topic/tag begins to show increase activity
+* Twitter Trending - A watch which alerts if a social media topic/tag begins to show increase activity
 * Unexpected Account Activity - A watch which aims detect and to alert if a user is created in Active Directory/LDAP and subsequently deleted within N mins.
 * New Process Started - A watch which aims to detect if a process is started on a server for the first time.
 * New User-Server Communication - A watch which aims to detect if a user logs onto a server for the first time within the current time period.
@@ -46,7 +46,7 @@ If username, password, and protocol are not specified, the above scripts assume
 * Monitoring Cluster Health - A watch which monitors an ES cluster for red or yellow cluster state.  Assumes use of X-Pack Monitoring.
 * Monitoring Free Disk Space - A watch which monitors an ES cluster for free disk usage on hosts.  Assumes use of X-Pack Monitoring.
 
-# Testing
+## Testing
 
 Each watch includes a test directory containing a set of tests expressed as JSON files.  Each JSON file describes a single isolated test and includes:
 
@@ -70,7 +70,7 @@ The run_test.py performs the following when running a test file:
 1. Refreshes the index.
 1. Adds the watch
 1. Executes the watch
-1. Confirms the watch matches the intended outcome. matched and confirms the output of the watch (log text)
+1. Confirms the watch matches the intended outcome. Matched and confirms the output of the watch (log text)
 
 ## Requirements
 

Alerting/Sample Watches/errors_in_logs/README.md

@@ -4,13 +4,13 @@
 
 A watch which alerts if errors are present in a log file. Provides example errors as output.
 
-The following watch utilises a basic query_string search, as used by Kibana, to find all documents in the last N minutes which either contain the word “error” or have a value of “ERROR” for the field “loglevel” i.e. the log level under which the message was generated.  The query returns the ids of upto 10 hits ordered by @timestamp in descending order.
+The following watch utilizes a basic query_string search, as used by Kibana, to find all documents in the last N minutes which either contain the word “error” or have a value of “ERROR” for the field “loglevel” i.e. the log level under which the message was generated.  The query returns the ids of up to 10 hits ordered by @timestamp in descending order.
 
 ## Mapping Assumptions
 
 A mapping is provided in mapping.json.  Watches require data producing the following fields:
 
-* @timestamp - authorative date field for each log message
+* @timestamp - authoritative date field for each log message
 * message (string) - contents of the log message as generated with Logstash
 * loglevel (string not_analyzed) - field with the value "ERROR", "DEBUG", "INFO" etc
 
@@ -24,5 +24,5 @@ The Watch assumes each log message is represented by an Elasticsearch document.
 
 # Configuration
 
-* The watch is scheduled to find errors very minute. Modify through the schedule.
-* The watch will raise a maximum of 1 alert every 15 minutes, even if the condition is satsified more than once. Modify through the throttle parameter.
+* The watch is scheduled to find errors every minute. Modify through the schedule.
+* The watch will raise a maximum of 1 alert every 15 minutes, even if the condition is satisfied more than once. Modify through the throttle parameter.

Alerting/Sample Watches/errors_in_logs/tests/test1.json

@@ -1,42 +1,41 @@
 {
-  "watch_name":"errors_in_logs",
-  "mapping_file":"./errors_in_logs/mapping.json",
-  "index":"logs",
-  "type":"doc",
+  "watch_name": "errors_in_logs",
+  "mapping_file": "./errors_in_logs/mapping.json",
+  "index": "logs",
+  "type": "doc",
   "match": true,
-  "watch_file":"./errors_in_logs/watch.json",
-  "events":[
-  {
-    "id":"1",
-    "offset":-35,
-    "message":"A normal message that should match due to the loglevel",
-    "loglevel":"ERROR"
-  },
-  {
-    "id":"2",
-    "offset":-30,
-    "message":"A normal log message",
-    "loglevel":"INFO"
-  },
-  {
-    "id":"3",
-    "offset":-40,
-    "message":"Error in this log message despite being INFO",
-    "loglevel":"INFO"
-  },
-  {
-    "id":"4",
-    "offset":-15,
-    "message":"Error in this log message",
-    "loglevel":"ERROR"
-  },
-  {
-    "id":"5",
-    "offset":-90,
-    "message":"Error in this log message but outside of the time range",
-    "loglevel":"ERROR"
-  }
+  "watch_file": "./errors_in_logs/watch.json",
+  "events": [
+    {
+      "id": "1",
+      "offset": -35,
+      "message": "A normal message that should match due to the loglevel",
+      "loglevel": "ERROR"
+    },
+    {
+      "id": "2",
+      "offset": -30,
+      "message": "A normal log message",
+      "loglevel": "INFO"
+    },
+    {
+      "id": "3",
+      "offset": -40,
+      "message": "Error in this log message despite being INFO",
+      "loglevel": "INFO"
+    },
+    {
+      "id": "4",
+      "offset": -15,
+      "message": "Error in this log message",
+      "loglevel": "ERROR"
+    },
+    {
+      "id": "5",
+      "offset": -90,
+      "message": "Error in this log message but outside of the time range",
+      "loglevel": "ERROR"
+    }
   ],
-"expected_response":"3 Errors have occured in the logs:4:1:3:"
+  "expected_response": "3 Errors have occurred in the logs:4:1:3:"
 }
-

Alerting/Sample Watches/errors_in_logs/watch.json

@@ -55,8 +55,8 @@
     "log": {
       "logging": {
         "level": "info",
-        "text": "{{ctx.payload.hits.total}} Errors have occured in the logs:{{#ctx.payload.hits.hits}}{{_id}}:{{/ctx.payload.hits.hits}}"
+        "text": "{{ctx.payload.hits.total}} Errors have occurred in the logs:{{#ctx.payload.hits.hits}}{{_id}}:{{/ctx.payload.hits.hits}}"
       }
     }
   }
-}
+}

Alerting/Sample Watches/filesystem_usage/tests/test1.json

@@ -1,32 +1,36 @@
 {
-  "watch_name":"filesystem_usage",
-  "mapping_file":"./filesystem_usage/mapping.json",
-  "index":"logs",
-  "type":"filesystem",
-  "watch_file":"./filesystem_usage/watch.json",
-  "comments":"Tests filesystem being above 0.9. Server 1 & 4 should alert as within 60 seconds. Server 2 should not (10 mins). 3rd server should not alert as < 0.9.",
-  "scripts":[{"name":"transform","path":"./filesystem_usage/scripts/transform.json"}],
-  "events":[
-  {
-    "hostname": "test_server1",
-    "used_p": 0.99,
-    "offset":"-60"
-  },
-  {
-    "hostname": "test_server2",
-    "used_p": 0.98,
-    "offset":"-600"
-  },
-  {
-    "hostname": "test_server3",
-    "used_p": 0.89,
-    "offset":"-60"
-  },
-  {
-    "hostname": "test_server4",
-    "used_p": 0.95
-  }
+  "watch_name": "filesystem_usage",
+  "mapping_file": "./filesystem_usage/mapping.json",
+  "index": "logs",
+  "type": "filesystem",
+  "watch_file": "./filesystem_usage/watch.json",
+  "comments": "Tests filesystem being above 0.9. Server 1 & 4 should alert as within 60 seconds. Server 2 should not (10 mins). 3rd server should not alert as < 0.9.",
+  "scripts": [
+    {
+      "name": "transform",
+      "path": "./filesystem_usage/scripts/transform.json"
+    }
   ],
-  "expected_response":"Some hosts are over 90% utilized:99%-test_server1:95%-test_server4:"
+  "events": [
+    {
+      "hostname": "test_server1",
+      "used_p": 0.99,
+      "offset": "-60"
+    },
+    {
+      "hostname": "test_server2",
+      "used_p": 0.98,
+      "offset": "-600"
+    },
+    {
+      "hostname": "test_server3",
+      "used_p": 0.89,
+      "offset": "-60"
+    },
+    {
+      "hostname": "test_server4",
+      "used_p": 0.95
+    }
+  ],
+  "expected_response": "Some hosts are over 90% utilized:99%-test_server1:95%-test_server4:"
 }
-

Alerting/Sample Watches/lateral_movement_in_user_comm/README.md

@@ -2,32 +2,32 @@
 
 ## Description
 
-A watch which aims to detect and alert if users log onto a server for which they have not accessed within the same time period previously.  The time period here is a configurable window either side of time the watch is executed. For example if the watch checks at 11:15 and the window size is 1hr, the watch will check if any users who have logged in within the last N seconds had logged into the same servers between 10:15 and 12:15 previously.  Any user-server communication which is "new", will result in an alert.
+A watch which aims to detect and alert if users log onto a server which they have not accessed within the same time period previously.  The time period here is a configurable window either side of time the watch is executed. For example if the watch checks at 11:15 and the window size is 1hr, the watch will check if any users who have logged in within the last N seconds had logged into the same servers between 10:15 and 12:15 previously.  Any user-server communication which is "new", will result in an alert.
 
-The watch achieves the above by using a three stage query chain. The first identifies a time window based on the configuration. The second periodically checks for user logins in the last N secs (default 30s), using a terms aggregation on the user_server field.  This list is then used to query against the index during the calculated time period, again aggregating on the user_server.  Values identified the list collected durting the second stage, which do not appear in the third stage list, are highlighted as new communication.
+The watch achieves the above by using a three stage query chain. The first identifies a time window based on the configuration. The second periodically checks for user logins in the last N secs (default 30s), using a terms aggregation on the user_server field.  This list is then used to query against the index during the calculated time period, again aggregating on the user_server.  Values identified in the list collected during the second stage, which do not appear in the third stage list, are highlighted as new communication.
 
-This watch represents a complex variant of the "first process exeuction" watch, which could be easily adapted to detect just new user logons to servers, adding a time period constraint.
+This watch represents a complex variant of the "first process execution" watch, which could be easily adapted to detect just new user logons to servers, adding a time period constraint.
 
 ## Mapping Assumptions
 
 A mapping is provided in mapping.json.  Watches require data producing the following fields:
 
-* user_server (non-analyzed string) - Contains the user and server as a concatenated string e.g. userA_testServerB.  Watch assumes the delimiter is an _ char.
+* user_server (non-analyzed string) - Contains the user and server as a concatenated string e.g. userA_testServerB.  Watch assumes the delimiter is an `_` char.
 * @timestamp (date field) - Date of log message.
-* time (date field) - time at which the logon occured based on a strict_time_no_millis format.
+* time (date field) - time at which the logon occurred based on a strict_time_no_millis format.
 
 ## Data Assumptions
 
 The watch assumes each document in Elasticsearch represents a logon to a server by a user.
 
 ## Other Assumptions
 
-* All events are index "log" and type "doc".
+* All events are index "log".
 * The watch assumes no more than 1000 user logons occur within the time period i.e. by default the last 30s.  This value can be adjusted, with consideration for scaling, for larger environments.
 
 # Configuration
 
 The following watch metadata parameters influence behaviour:
 
-* window_width- The period in N (minutes) during which the user should have logged onto the server previously.  The window is calcuated as T-N to T+N, where T is the time the watch executed. Defaults to 30mins, giving a total window width of approximiately 1hr.
-* time_period - The period for which user server logons are aggregated, and compared against the time period to check as to whether they represent new communication. Defaults to 30s.  This should be equal to the schedule interval to ensure no logins are not evaluated.
+* window_width- The period in N (minutes) during which the user should have logged onto the server previously.  The window is calculated as T-N to T+N, where T is the time the watch executed. Defaults to 30mins, giving a total window width of approximately 1hr.
+* time_period - The period for which user server logons are aggregated, and compared against the time period to check as to whether they represent new communication. Defaults to 30s.  This should be equal to the schedule interval to ensure no logins are not evaluated.