split all the .md documentation into different files

idaholab · Sep 22, 2022 · 952936d · 952936d
1 parent 6a7003a
commit 952936d
Show file tree

Hide file tree

Showing 206 changed files with 4,883 additions and 5,273 deletions.
diff --git a/README.md b/README.md
diff --git a/docs/alerting.md b/docs/alerting.md
@@ -0,0 +1,34 @@
+### <a name="Alerting"></a>Alerting
+
+Malcolm uses the Alerting plugins for [OpenSearch](https://github.com/opensearch-project/alerting) and [OpenSearch Dashboards](https://github.com/opensearch-project/alerting-dashboards-plugin). See [Alerting](https://opensearch.org/docs/latest/monitoring-plugins/alerting/index/) in the OpenSearch documentation for usage instructions.
+
+A fresh installation of Malcolm configures an example [custom webhook destination](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#create-destinations) named **Malcolm API Loopback Webhook** that directs the triggered alerts back into the [Malcolm API](#API) to be reindexed as a session record with `event.dataset` set to `alerting`. The corresponding monitor **Malcolm API Loopback Monitor** is disabled by default, as you'll likely want to configure the trigger conditions to suit your needs. These examples are provided to illustrate how triggers and monitors can interact with a custom webhook to process alerts.
+
+#### <a name="AlertingEmail"></a>Email Sender Accounts
+
+When using an email account to send alerts, you must [authenticate each sender account](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account) before you can send an email. The [`auth_setup`](#AuthSetup) script can be used to securely store the email account credentials:
+
+```
+./scripts/auth_setup 
+
+Store administrator username/password for local Malcolm access? (Y/n): n
+
+(Re)generate self-signed certificates for HTTPS access (Y/n): n
+
+(Re)generate self-signed certificates for a remote log forwarder (Y/n): n
+
+Store username/password for primary remote OpenSearch instance? (y/N): n
+
+Store username/password for secondary remote OpenSearch instance? (y/N): n
+
+Store username/password for email alert sender account? (y/N): y
+
+Email account username: [email protected]
+[email protected] password: 
+[email protected] password (again): 
+Email alert sender account variables stored: opensearch.alerting.destination.email.destination_alpha.password, opensearch.alerting.destination.email.destination_alpha.username
+
+(Re)generate internal passwords for NetBox (Y/n): n
+```
+
+This action should only be performed while Malcolm is [stopped](#StopAndRestart): otherwise the credentials will not be stored correctly.
diff --git a/docs/anomaly-detection.md b/docs/anomaly-detection.md
@@ -0,0 +1,12 @@
+### <a name="AnomalyDetection"></a>Anomaly Detection
+
+Malcolm uses the Anomaly Detection plugins for [OpenSearch](https://github.com/opensearch-project/anomaly-detection) and [OpenSearch Dashboards](https://github.com/opensearch-project/anomaly-detection-dashboards-plugin) to identify anomalous log data in near real-time using the [Random Cut Forest](https://api.semanticscholar.org/CorpusID:927435) (RCF) algorithm. This can be paired with [Alerting](#Alerting) to automatically notify when anomalies are found. See [Anomaly detection](https://opensearch.org/docs/latest/monitoring-plugins/ad/index/) in the OpenSearch documentation for usage instructions on how to create detectors for any of the many fields Malcolm supports.
+
+A fresh installation of Malcolm configures [several detectors](dashboards/anomaly_detectors) for detecting anomalous network traffic:
+
+* **network_protocol** - Detects anomalies based on application protocol (`network.protocol`)
+* **action_result_user** - Detects anomalies in action (`event.action`), result (`event.result`) and user (`related.user`) within application protocols (`network.protocol`)
+* **file_mime_type** - Detects anomalies based on transferred file type (`file.mime_type`)
+* **total_bytes** - Detects anomalies based on traffic size (sum of `network.bytes`)
+
+These detectors are disabled by default, but may be enabled for anomaly detection over streaming or [historical data](https://aws.amazon.com/about-aws/whats-new/2022/01/amazon-opensearch-service-elasticsearch-anomaly-detection/).
diff --git a/docs/api-aggregations.md b/docs/api-aggregations.md
@@ -0,0 +1,24 @@
+#### Field Aggregations
+
+`GET` or `POST` - /mapi/agg/`<fieldname>`
+
+Executes an OpenSearch [bucket aggregation](https://opensearch.org/docs/latest/opensearch/bucket-agg/) query for the requested fields across all of Malcolm's indexed network traffic metadata.
+
+Parameters:
+
+* `fieldname` (URL parameter) - the name(s) of the field(s) to be queried (comma-separated if multiple fields) (default: `event.provider`)
+* `limit` (query parameter) - the maximum number of records to return at each level of aggregation (default: 500)
+* `from` (query parameter) - the time frame ([`gte`](https://opensearch.org/docs/latest/opensearch/query-dsl/term/#range)) for the beginning of the search based on the session's `firstPacket` field value in a format supported by the [dateparser](https://github.com/scrapinghub/dateparser) library (default: "1 day ago")
+* `to` (query parameter) - the time frame ([`lte`](https://opensearch.org/docs/latest/opensearch/query-dsl/term/#range)) for the beginning of the search based on the session's `firstPacket` field value in a format supported by the [dateparser](https://github.com/scrapinghub/dateparser) library (default: "now")
+* `filter` (query parameter) - field filters formatted as a JSON dictionary
+
+The `from`, `to`, and `filter` parameters can be used to further restrict the range of documents returned. The `filter` dictionary should be formatted such that its keys are field names and its values are the values for which to filter. A field name may be prepended with a `!` to negate the filter (e.g., `{"event.provider":"zeek"}` vs. `{"!event.provider":"zeek"}`). Filtering for value `null` implies "is not set" or "does not exist" (e.g., `{"event.dataset":null}` means "the field `event.dataset` is `null`/is not set" while `{"!event.dataset":null}` means "the field `event.dataset` is not `null`/is set").
+
+Examples of `filter` parameter:
+
+* `{"!network.transport":"icmp"}` - `network.transport` is not `icmp`
+* `{"network.direction":["inbound","outbound"]}` - `network.direction` is either `inbound` or `outbound`
+* `{"event.provider":"zeek","event.dataset":["conn","dns"]}` - "`event.provider` is `zeek` and `event.dataset` is either `conn` or `dns`"
+* `{"!event.dataset":null}` - "`event.dataset` is set (is not `null`)"
+
+See [Examples](#APIExamples) for more examples of `filter` and corresponding output.
diff --git a/docs/api-document-lookup.md b/docs/api-document-lookup.md
@@ -0,0 +1,90 @@
+#### Document Lookup
+
+`GET` or `POST` - /mapi/document
+
+Executes an OpenSearch [query](https://opensearch.org/docs/latest/opensearch/bucket-agg/) query for the matching documents across all of Malcolm's indexed network traffic metadata.
+
+Parameters:
+
+* `limit` (query parameter) - the maximum number of documents to return (default: 500)
+* `from` (query parameter) - the time frame ([`gte`](https://opensearch.org/docs/latest/opensearch/query-dsl/term/#range)) for the beginning of the search based on the session's `firstPacket` field value in a format supported by the [dateparser](https://github.com/scrapinghub/dateparser) library (default: the UNIX epoch)
+* `to` (query parameter) - the time frame ([`lte`](https://opensearch.org/docs/latest/opensearch/query-dsl/term/#range)) for the beginning of the search based on the session's `firstPacket` field value in a format supported by the [dateparser](https://github.com/scrapinghub/dateparser) library (default: "now")
+* `filter` (query parameter) - field filters formatted as a JSON dictionary (see **Field Aggregations** for examples)
+
+<details>
+<summary>Example cURL command and output:</summary>
+
+```
+$ curl -k -u username -L -XPOST -H 'Content-Type: application/json' \
+    'https://localhost/mapi/document' \
+    -d '{"limit": 10, filter":{"zeek.uid":"CYeji2z7CKmPRGyga"}}'
+```
+
+```json
+{
+    "filter": {
+        "zeek.uid": "CYeji2z7CKmPRGyga"
+    },
+    "range": [
+        0,
+        1643056677
+    ],
+    "results": [
+        {
+            "_id": "220124-CYeji2z7CKmPRGyga-http-7677",
+            "_index": "arkime_sessions3-220124",
+            "_score": 0.0,
+            "_source": {
+                "@timestamp": "2022-01-24T20:31:01.846Z",
+                "@version": "1",
+                "agent": {
+                    "hostname": "filebeat",
+                    "id": "bc25716b-8fe7-4de6-a357-65c7d3c15c33",
+                    "name": "filebeat",
+                    "type": "filebeat",
+                    "version": "7.10.2"
+                },
+                "client": {
+                    "bytes": 0
+                },
+                "destination": {
+                    "as": {
+                        "full": "AS54113 Fastly"
+                    },
+                    "geo": {
+                        "city_name": "Seattle",
+                        "continent_code": "NA",
+                        "country_code2": "US",
+                        "country_code3": "US",
+                        "country_iso_code": "US",
+                        "country_name": "United States",
+                        "dma_code": 819,
+                        "ip": "151.101.54.132",
+                        "latitude": 47.6092,
+                        "location": {
+                            "lat": 47.6092,
+                            "lon": -122.3314
+                        },
+                        "longitude": -122.3314,
+                        "postal_code": "98111",
+                        "region_code": "WA",
+                        "region_name": "Washington",
+                        "timezone": "America/Los_Angeles"
+                    },
+                    "ip": "151.101.54.132",
+                    "port": 80
+                },
+                "ecs": {
+                    "version": "1.6.0"
+                },
+                "event": {
+                    "action": [
+                        "GET"
+                    ],
+                    "category": [
+                        "web",
+                        "network"
+                    ],
+…
+```
+</details>
diff --git a/docs/api-event-logging.md b/docs/api-event-logging.md
@@ -0,0 +1,73 @@
+#### Event Logging
+
+`POST` - /mapi/event
+
+A webhook that accepts alert data to be reindexed into OpenSearch as session records for viewing in Malcolm's [dashboards](#Dashboards). See [Alerting](#Alerting) for more details and an example of how this API is used.
+
+<details>
+<summary>Example input:</summary>
+
+```json
+
+{
+  "alert": {
+    "monitor": {
+      "name": "Malcolm API Loopback Monitor"
+    },
+    "trigger": {
+      "name": "Malcolm API Loopback Trigger",
+      "severity": 4
+    },
+    "period": {
+      "start": "2022-03-08T18:03:30.576Z",
+      "end": "2022-03-08T18:04:30.576Z"
+    },
+    "results": [
+      {
+        "_shards": {
+          "total": 5,
+          "failed": 0,
+          "successful": 5,
+          "skipped": 0
+        },
+        "hits": {
+          "hits": [],
+          "total": {
+            "value": 697,
+            "relation": "eq"
+          },
+          "max_score": null
+        },
+        "took": 1,
+        "timed_out": false
+      }
+    ],
+    "body": "",
+    "alert": "PLauan8BaL6eY1yCu9Xj",
+    "error": ""
+  }
+}
+```
+</details>
+
+<details>
+<summary>Example output:</summary>
+
+```json
+
+{
+  "_index": "arkime_sessions3-220308",
+  "_type": "_doc",
+  "_id": "220308-PLauan8BaL6eY1yCu9Xj",
+  "_version": 4,
+  "result": "updated",
+  "_shards": {
+    "total": 1,
+    "successful": 1,
+    "failed": 0
+  },
+  "_seq_no": 9045,
+  "_primary_term": 1
+}
+```
+</details>