documentation links work in progress

docs/main.md → README.md

@@ -22,101 +22,101 @@ You can help steer Malcolm's development by sharing your ideas and feedback. Ple
 ## <a name="TableOfContents"></a>Table of Contents
 
 * [Automated Build Workflows Status](#BuildBadges)
-* [Quick start](quickstart.md#QuickStart)
-    * [Getting Malcolm](#GetMalcolm)
-    * [User interface](#UserInterfaceURLs)
+* [Quick start](docs/quickstart.md#QuickStart)
+    * [Getting Malcolm](docs/quickstart.md#GetMalcolm)
+    * [User interface](docs/quickstart.md#UserInterfaceURLs)
 * [Overview](#Overview)
-* [Components](#Components)
-* [Supported Protocols](#Protocols)
-* [Development](#Development)
-    * [Building from source](#Build)
-* [Pre-Packaged installation files](#Packager)
+* [Components](docs/components.md#Components)
+* [Supported Protocols](docs/protocols.md#Protocols)
+* [Development](docs/development.md#Development)
+    * [Building from source](docs/development.md#Build)
+    * [Pre-Packaged installation files](docs/development.md#Packager)
 * [Preparing your system](#Preparing)
     * [Recommended system requirements](#SystemRequirements)
-    * [System configuration and tuning](#ConfigAndTuning)
-        * [`docker-compose.yml` parameters](#DockerComposeYml)
+    * [System configuration and tuning](docs/malcolm-config.md#ConfigAndTuning)
+        * [`docker-compose.yml` parameters](docs/malcolm-config.md#DockerComposeYml)
         * [Linux host system configuration](#HostSystemConfigLinux)
         * [macOS host system configuration](#HostSystemConfigMac)
         * [Windows host system configuration](#HostSystemConfigWindows)
 * [Running Malcolm](#Running)
-    * [OpenSearch instances](#OpenSearchInstance)
+    * [OpenSearch instances](docs/opensearch-instances.md#OpenSearchInstance)
         * [Authentication and authorization for remote OpenSearch clusters](#OpenSearchAuth)
-    * [Configure authentication](#AuthSetup)
-        * [Local account management](#AuthBasicAccountManagement)
-        * [Lightweight Directory Access Protocol (LDAP) authentication](#AuthLDAP)
+    * [Configure authentication](docs/authsetup.md#AuthSetup)
+        * [Local account management](docs/authsetup.md#AuthBasicAccountManagement)
+        * [Lightweight Directory Access Protocol (docs/LDAP) authentication](authsetup.md#AuthLDAP)
             - [LDAP connection security](#AuthLDAPSecurity)
-    * [TLS certificates](#TLSCerts)
-    * [Starting Malcolm](#Starting)
-    * [Stopping and restarting Malcolm](#StopAndRestart)
-    * [Clearing Malcolm's data](#Wipe)
+    * [TLS certificates](docs/authsetup.md#TLSCerts)
+    * [Starting Malcolm](docs/running.md#Starting)
+    * [Stopping and restarting Malcolm](docs/running.md#StopAndRestart)
+    * [Clearing Malcolm's data](docs/running.md#Wipe)
     * [Temporary read-only interface](#ReadOnlyUI)
-* [Capture file and log archive upload](#Upload)
-    - [Tagging](#Tagging)
-    - [Processing uploaded PCAPs with Zeek and Suricata](#UploadPCAPProcessors)
+* [Capture file and log archive upload](docs/upload.md#Upload)
+    - [Tagging](docs/upload.md#Tagging)
+    - [Processing uploaded PCAPs with Zeek and Suricata](docs/upload.md#UploadPCAPProcessors)
 * [Live analysis](#LiveAnalysis)
-    * [Using a network sensor appliance](#Hedgehog)
+    * [Using a network sensor appliance](docs/live-analysis.md#Hedgehog)
     * [Monitoring local network interfaces](#LocalPCAP)
     * [Manually forwarding logs from an external source](#ExternalForward)
 * [Arkime](#Arkime)
-    * [Zeek log integration](#ArkimeZeek)
-        - [Correlating Zeek logs and Arkime sessions](#ZeekArkimeFlowCorrelation)
+    * [Zeek log integration](docs/arkime.md#ArkimeZeek)
+        - [Correlating Zeek logs and Arkime sessions](docs/arkime.md#ZeekArkimeFlowCorrelation)
     * [Help](#ArkimeHelp)
-    * [Sessions](#ArkimeSessions)
+    * [Sessions](docs/arkime.md#ArkimeSessions)
         * [PCAP Export](#ArkimePCAPExport)
     * [SPIView](#ArkimeSPIView)
-    * [SPIGraph](#ArkimeSPIGraph)
+    * [SPIGraph](docs/arkime.md#ArkimeSPIGraph)
     * [Connections](#ArkimeConnections)
-    * [Hunt](#ArkimeHunt)
+    * [Hunt](docs/arkime.md#ArkimeHunt)
     * [Statistics](#ArkimeStats)
     * [Settings](#ArkimeSettings)
-* [OpenSearch Dashboards](#Dashboards)
+* [OpenSearch Dashboards](docs/dashboards.md#Dashboards)
     * [Discover](#Discover)
         - [Screenshots](#DiscoverGallery)
-    * [Visualizations and dashboards](#DashboardsVisualizations)
+    * [Visualizations and dashboards](docs/dashboards.md#DashboardsVisualizations)
         - [Prebuilt visualizations and dashboards](#PrebuiltVisualizations)
             - [Screenshots](#PrebuiltVisualizationsGallery)
-        - [Building your own visualizations and dashboards](#BuildDashboard)
+        - [Building your own visualizations and dashboards](docs/dashboards.md#BuildDashboard)
             + [Screenshots](#NewVisualizationsGallery)
-* [Search Queries in Arkime and OpenSearch](#SearchCheatSheet)
+* [Search Queries in Arkime and OpenSearch](docs/queries-cheat-sheet.md#SearchCheatSheet)
 * [Other Malcolm features](#MalcolmFeatures)
-    - [Automatic file extraction and scanning](#ZeekFileExtraction)
-    - [Automatic host and subnet name assignment](#HostAndSubnetNaming)
-        + [IP/MAC address to hostname mapping via `host-map.txt`](#HostNaming)
-        + [CIDR subnet to network segment name mapping via `cidr-map.txt`](#SegmentNaming)
-        + [Defining hostname and CIDR subnet names interface](#NameMapUI)
+    - [Automatic file extraction and scanning](docs/file-scanning.md#ZeekFileExtraction)
+    - [Automatic host and subnet name assignment](docs/host-and-subnet-mapping.md#HostAndSubnetNaming)
+        + [IP/MAC address to hostname mapping via `host-map.txt`](docs/host-and-subnet-mapping.md#HostNaming)
+        + [CIDR subnet to network segment name mapping via `cidr-map.txt`](docs/host-and-subnet-mapping.md#SegmentNaming)
+        + [Defining hostname and CIDR subnet names interface](docs/host-and-subnet-mapping.md#NameMapUI)
         + [Applying mapping changes](#ApplyMapping)
-    - [OpenSearch index management](#IndexManagement)
-    - [Event severity scoring](#Severity)
+    - [OpenSearch index management](docs/index-management.md#IndexManagement)
+    - [Event severity scoring](docs/severity.md#Severity)
         + [Customizing event severity scoring](#SeverityConfig)
-    - [Zeek Intelligence Framework](#ZeekIntel)
-        + [STIX™ and TAXII™](#ZeekIntelSTIX)
-        + [MISP](#ZeekIntelMISP)
+    - [Zeek Intelligence Framework](docs/zeek-intel.md#ZeekIntel)
+        + [STIX™ and TAXII™](docs/zeek-intel.md#ZeekIntelSTIX)
+        + [MISP](docs/zeek-intel.md#ZeekIntelMISP)
     - [Anomaly Detection](#AnomalyDetection)
-    - [Alerting](#Alerting)
+    - [Alerting](docs/alerting.md#Alerting)
         + [Email Sender Accounts](#AlertingEmail)
-    - ["Best Guess" Fingerprinting for ICS Protocols](#ICSBestGuess)
-    - [Asset Management with NetBox](#NetBox)
+    - ["Best Guess" Fingerprinting for ICS Protocols](docs/ics-best-guess.md#ICSBestGuess)
+    - [Asset Management with NetBox](docs/netbox.md#NetBox)
     - [CyberChef](#CyberChef)
-    - [API](#API)
-        + [Examples](#APIExamples)
+    - [API](docs/api.md#API)
+        + [Examples](docs/api-examples.md#APIExamples)
 * [Ingesting Third-party Logs](#ThirdPartyLogs)
-* [Malcolm installer ISO](#ISO)
-    * [Installation](#ISOInstallation)
+* [Malcolm installer ISO](docs/malcolm-iso.md#ISO)
+    * [Installation](docs/malcolm-iso.md#ISOInstallation)
     * [Generating the ISO](#ISOBuild)
     * [Setup](#ISOSetup)
     * [Time synchronization](#ConfigTime)
-    * [Hardening](#Hardening)
+    * [Hardening](docs/hardening.md#Hardening)
         * [Compliance Exceptions](#ComplianceExceptions)
-* [Installation example using Ubuntu 22.04 LTS](#InstallationExample)
-* [Upgrading Malcolm](#UpgradePlan)
+* [Installation example using Ubuntu 22.04 LTS](docs/ubuntu-install-example.md#InstallationExample)
+* [Upgrading Malcolm](docs/malcolm-upgrade.md#UpgradePlan)
 * [Modifying or Contributing to Malcolm](#Contributing)
 * [Forks](#Forks)
 * [Copyright](#Footer)
 * [Contact](#Contact)
 
 ## <a name="BuildBadges"></a>Automated Builds Status
 
-See [**Building from source**](#Build) to read how you can use GitHub [workflow files](./.github/workflows/) to build Malcolm.
+See [**Building from source**](docs/development.md#Build) to read how you can use GitHub [workflow files](./.github/workflows/) to build Malcolm.
 
 ![api-build-and-push-ghcr](https://github.com/mmguero-dev/Malcolm/workflows/api-build-and-push-ghcr/badge.svg)
 ![arkime-build-and-push-ghcr](https://github.com/mmguero-dev/Malcolm/workflows/arkime-build-and-push-ghcr/badge.svg)
@@ -142,9 +142,9 @@ See [**Building from source**](#Build) to read how you can use GitHub [workflow
 
 ![Malcolm Network Diagram](./images/malcolm_network_diagram.png)
 
-Malcolm processes network traffic data in the form of packet capture (PCAP) files or Zeek logs. A [sensor](#Hedgehog) (packet capture appliance) monitors network traffic mirrored to it over a SPAN port on a network switch or router, or using a network TAP device. [Zeek](https://www.zeek.org/index.html) logs and [Arkime](https://molo.ch/) sessions are generated containing important session metadata from the traffic observed, which are then securely forwarded to a Malcolm instance. Full PCAP files are optionally stored locally on the sensor device for examination later.
+Malcolm processes network traffic data in the form of packet capture (docs/PCAP) files or Zeek logs. A [sensor](live-analysis.md#Hedgehog) (packet capture appliance) monitors network traffic mirrored to it over a SPAN port on a network switch or router, or using a network TAP device. [Zeek](https://www.zeek.org/index.html) logs and [Arkime](https://molo.ch/) sessions are generated containing important session metadata from the traffic observed, which are then securely forwarded to a Malcolm instance. Full PCAP files are optionally stored locally on the sensor device for examination later.
 
-Malcolm parses the network session data and enriches it with additional lookups and mappings including GeoIP mapping, hardware manufacturer lookups from [organizationally unique identifiers (OUI)](http://standards-oui.ieee.org/oui/oui.txt) in MAC addresses, assigning names to [network segments](#SegmentNaming) and [hosts](#HostNaming) based on user-defined IP address and MAC mappings, performing [TLS fingerprinting](#https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967), and many others.
+Malcolm parses the network session data and enriches it with additional lookups and mappings including GeoIP mapping, hardware manufacturer lookups from [organizationally unique identifiers (docs/OUI)](http://standards-oui.ieee.org/oui/oui.txt) in MAC addresses, assigning names to [network segments](host-and-subnet-mapping.md#SegmentNaming) and [hosts](host-and-subnet-mapping.md#HostNaming) based on user-defined IP address and MAC mappings, performing [TLS fingerprinting](#https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967), and many others.
 
 The enriched data is stored in an [OpenSearch](https://opensearch.org/) document store in a format suitable for analysis through two intuitive interfaces: OpenSearch Dashboards, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Arkime, a powerful tool for finding and identifying the network sessions comprising suspected security incidents. These tools can be accessed through a web browser from analyst workstations or for display in a security operations center (SOC). Logs can also optionally be forwarded on to another instance of Malcolm.
 

docs/alerting.md

@@ -2,11 +2,11 @@
 
 Malcolm uses the Alerting plugins for [OpenSearch](https://github.com/opensearch-project/alerting) and [OpenSearch Dashboards](https://github.com/opensearch-project/alerting-dashboards-plugin). See [Alerting](https://opensearch.org/docs/latest/monitoring-plugins/alerting/index/) in the OpenSearch documentation for usage instructions.
 
-A fresh installation of Malcolm configures an example [custom webhook destination](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#create-destinations) named **Malcolm API Loopback Webhook** that directs the triggered alerts back into the [Malcolm API](#API) to be reindexed as a session record with `event.dataset` set to `alerting`. The corresponding monitor **Malcolm API Loopback Monitor** is disabled by default, as you'll likely want to configure the trigger conditions to suit your needs. These examples are provided to illustrate how triggers and monitors can interact with a custom webhook to process alerts.
+A fresh installation of Malcolm configures an example [custom webhook destination](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#create-destinations) named **Malcolm API Loopback Webhook** that directs the triggered alerts back into the [Malcolm API](api.md#API) to be reindexed as a session record with `event.dataset` set to `alerting`. The corresponding monitor **Malcolm API Loopback Monitor** is disabled by default, as you'll likely want to configure the trigger conditions to suit your needs. These examples are provided to illustrate how triggers and monitors can interact with a custom webhook to process alerts.
 
 #### <a name="AlertingEmail"></a>Email Sender Accounts
 
-When using an email account to send alerts, you must [authenticate each sender account](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account) before you can send an email. The [`auth_setup`](#AuthSetup) script can be used to securely store the email account credentials:
+When using an email account to send alerts, you must [authenticate each sender account](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account) before you can send an email. The [`auth_setup`](authsetup.md#AuthSetup) script can be used to securely store the email account credentials:
 
 ```
 ./scripts/auth_setup 
@@ -31,4 +31,4 @@ Email alert sender account variables stored: opensearch.alerting.destination.ema
 (Re)generate internal passwords for NetBox (Y/n): n
 ```
 
-This action should only be performed while Malcolm is [stopped](#StopAndRestart): otherwise the credentials will not be stored correctly.
+This action should only be performed while Malcolm is [stopped](running.md#StopAndRestart): otherwise the credentials will not be stored correctly.

docs/anomaly-detection.md

@@ -1,6 +1,6 @@
 ### <a name="AnomalyDetection"></a>Anomaly Detection
 
-Malcolm uses the Anomaly Detection plugins for [OpenSearch](https://github.com/opensearch-project/anomaly-detection) and [OpenSearch Dashboards](https://github.com/opensearch-project/anomaly-detection-dashboards-plugin) to identify anomalous log data in near real-time using the [Random Cut Forest](https://api.semanticscholar.org/CorpusID:927435) (RCF) algorithm. This can be paired with [Alerting](#Alerting) to automatically notify when anomalies are found. See [Anomaly detection](https://opensearch.org/docs/latest/monitoring-plugins/ad/index/) in the OpenSearch documentation for usage instructions on how to create detectors for any of the many fields Malcolm supports.
+Malcolm uses the Anomaly Detection plugins for [OpenSearch](https://github.com/opensearch-project/anomaly-detection) and [OpenSearch Dashboards](https://github.com/opensearch-project/anomaly-detection-dashboards-plugin) to identify anomalous log data in near real-time using the [Random Cut Forest](https://api.semanticscholar.org/CorpusID:927435) (RCF) algorithm. This can be paired with [Alerting](alerting.md#Alerting) to automatically notify when anomalies are found. See [Anomaly detection](https://opensearch.org/docs/latest/monitoring-plugins/ad/index/) in the OpenSearch documentation for usage instructions on how to create detectors for any of the many fields Malcolm supports.
 
 A fresh installation of Malcolm configures [several detectors](dashboards/anomaly_detectors) for detecting anomalous network traffic:
 

docs/api-aggregations.md

@@ -21,4 +21,4 @@ Examples of `filter` parameter:
 * `{"event.provider":"zeek","event.dataset":["conn","dns"]}` - "`event.provider` is `zeek` and `event.dataset` is either `conn` or `dns`"
 * `{"!event.dataset":null}` - "`event.dataset` is set (is not `null`)"
 
-See [Examples](#APIExamples) for more examples of `filter` and corresponding output.
+See [Examples](api-examples.md#APIExamples) for more examples of `filter` and corresponding output.

docs/api-event-logging.md

@@ -2,7 +2,7 @@
 
 `POST` - /mapi/event
 
-A webhook that accepts alert data to be reindexed into OpenSearch as session records for viewing in Malcolm's [dashboards](#Dashboards). See [Alerting](#Alerting) for more details and an example of how this API is used.
+A webhook that accepts alert data to be reindexed into OpenSearch as session records for viewing in Malcolm's [dashboards](dashboards.md#Dashboards). See [Alerting](alerting.md#Alerting) for more details and an example of how this API is used.
 
 <details>
 <summary>Example input:</summary>