diff --git a/docs/README.md b/docs/README.md index dfe7d73d3..4cea66ec6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,4 +1,4 @@ -## Overview +# Overview ![Malcolm Network Diagram](./images/malcolm_network_diagram.png) @@ -12,8 +12,7 @@ The enriched data is stored in an [OpenSearch](https://opensearch.org/) document For smaller networks, use at home by network security enthusiasts, or in the field for incident response engagements, Malcolm can also easily be deployed locally on an ordinary consumer workstation or laptop. Malcolm can process local artifacts such as locally-generated Zeek logs, locally-captured PCAP files, and PCAP files collected offline without the use of a dedicated sensor appliance. -## Table of Contents - + * [Quick start](quickstart.md#QuickStart) - [Getting Malcolm](quickstart.md#GetMalcolm) - [User interface](quickstart.md#UserInterfaceURLs) diff --git a/docs/alerting.md b/docs/alerting.md index 6596335f8..4aa5f83ab 100644 --- a/docs/alerting.md +++ b/docs/alerting.md @@ -1,4 +1,4 @@ -### Alerting +# Alerting * [Alerting](#Alerting) - [Email Sender Accounts](#AlertingEmail) @@ -7,7 +7,7 @@ Malcolm uses the Alerting plugins for [OpenSearch](https://github.com/opensearch A fresh installation of Malcolm configures an example [custom webhook destination](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#create-destinations) named **Malcolm API Loopback Webhook** that directs the triggered alerts back into the [Malcolm API](api.md#API) to be reindexed as a session record with `event.dataset` set to `alerting`. The corresponding monitor **Malcolm API Loopback Monitor** is disabled by default, as you'll likely want to configure the trigger conditions to suit your needs. These examples are provided to illustrate how triggers and monitors can interact with a custom webhook to process alerts. -#### Email Sender Accounts +## Email Sender Accounts When using an email account to send alerts, you must [authenticate each sender account](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account) before you can send an email. The [`auth_setup`](authsetup.md#AuthSetup) script can be used to securely store the email account credentials: diff --git a/docs/anomaly-detection.md b/docs/anomaly-detection.md index d6739be3b..0123a615c 100644 --- a/docs/anomaly-detection.md +++ b/docs/anomaly-detection.md @@ -1,4 +1,4 @@ -### Anomaly Detection +# Anomaly Detection Malcolm uses the Anomaly Detection plugins for [OpenSearch](https://github.com/opensearch-project/anomaly-detection) and [OpenSearch Dashboards](https://github.com/opensearch-project/anomaly-detection-dashboards-plugin) to identify anomalous log data in near real-time using the [Random Cut Forest](https://api.semanticscholar.org/CorpusID:927435) (RCF) algorithm. This can be paired with [Alerting](alerting.md#Alerting) to automatically notify when anomalies are found. See [Anomaly detection](https://opensearch.org/docs/latest/monitoring-plugins/ad/index/) in the OpenSearch documentation for usage instructions on how to create detectors for any of the many fields Malcolm supports. diff --git a/docs/api-aggregations.md b/docs/api-aggregations.md index b65f8b70b..a201aa76e 100644 --- a/docs/api-aggregations.md +++ b/docs/api-aggregations.md @@ -1,4 +1,4 @@ -#### Field Aggregations +# Field Aggregations `GET` or `POST` - /mapi/agg/`` diff --git a/docs/api-document-lookup.md b/docs/api-document-lookup.md index 3c6d8ec29..2fb0a5bb4 100644 --- a/docs/api-document-lookup.md +++ b/docs/api-document-lookup.md @@ -1,4 +1,4 @@ -#### Document Lookup +# Document Lookup `GET` or `POST` - /mapi/document diff --git a/docs/api-event-logging.md b/docs/api-event-logging.md index 2c92a9a96..4a6c70e66 100644 --- a/docs/api-event-logging.md +++ b/docs/api-event-logging.md @@ -1,4 +1,4 @@ -#### Event Logging +# Event Logging `POST` - /mapi/event diff --git a/docs/api-examples.md b/docs/api-examples.md index dc5b49847..f86026027 100644 --- a/docs/api-examples.md +++ b/docs/api-examples.md @@ -1,4 +1,4 @@ -#### Examples +# Examples Some security-related API examples: diff --git a/docs/api-fields.md b/docs/api-fields.md index 52634c2b9..00679476c 100644 --- a/docs/api-fields.md +++ b/docs/api-fields.md @@ -1,4 +1,4 @@ -#### Fields +# Fields `GET` - /mapi/fields diff --git a/docs/api-indices.md b/docs/api-indices.md index 1756e37fe..d3a2230f0 100644 --- a/docs/api-indices.md +++ b/docs/api-indices.md @@ -1,4 +1,4 @@ -#### Indices +# Indices `GET` - /mapi/indices diff --git a/docs/api-ping.md b/docs/api-ping.md index f6d88fa84..28fd70de1 100644 --- a/docs/api-ping.md +++ b/docs/api-ping.md @@ -1,4 +1,4 @@ -#### Ping +# Ping `GET` - /mapi/ping diff --git a/docs/api-version.md b/docs/api-version.md index 8b7946097..a0490300c 100644 --- a/docs/api-version.md +++ b/docs/api-version.md @@ -1,4 +1,4 @@ -#### Version Information +# Version Information `GET` - /mapi/version diff --git a/docs/api.md b/docs/api.md index a278f847c..f5434d71c 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1,4 +1,4 @@ -### API +# API * [Aggregations](api-aggregations.md) * [Document](api-document-lookup.md) diff --git a/docs/arkime.md b/docs/arkime.md index a821abab9..6dfa79a0c 100644 --- a/docs/arkime.md +++ b/docs/arkime.md @@ -1,4 +1,4 @@ -## Arkime +# Arkime * [Arkime](#Arkime) - [Zeek log integration](#ArkimeZeek) @@ -15,7 +15,7 @@ The Arkime interface will be accessible over HTTPS on port 443 at the docker hosts IP address (e.g., [https://localhost](https://localhost) if you are connecting locally). -### Zeek log integration +## Zeek log integration A stock installation of Arkime extracts all of its network connection ("session") metadata ("SPI" or "Session Profile Information") from full packet capture artifacts (PCAP files). Zeek (formerly Bro) generates similar session metadata, linking network events to sessions via a connection UID. Malcolm aims to facilitate analysis of Zeek logs by mapping values from Zeek logs to the Arkime session database schema for equivalent fields, and by creating new "native" Arkime database fields for all the other Zeek log values for which there is not currently an equivalent in Arkime: @@ -31,7 +31,7 @@ Click the icon of the owl **🦉** in the upper-left hand corner of to access th The values of records created from Zeek logs can be expanded and viewed like any native Arkime session by clicking the plus **➕** icon to the left of the record in the Sessions view. However, note that when dealing with these Zeek records the full packet contents are not available, so buttons dealing with viewing and exporting PCAP information will not behave as they would for records from PCAP files. Other than that, Zeek records and their values are usable in Malcolm just like native PCAP session records. -#### Correlating Zeek logs and Arkime sessions +### Correlating Zeek logs and Arkime sessions The Arkime interface displays both Zeek logs and Arkime sessions alongside each other. Using fields common to both data sources, one can [craft queries](queries-cheat-sheet.md#SearchCheatSheet) to filter results matching desired criteria. @@ -45,11 +45,11 @@ Filtering on community ID OR'ed with zeek UID (e.g., `network.community_id == "1 ![Correlating Arkime sessions and Zeek logs](./images/screenshots/arkime_correlate_communityid_uid.png) -### Help +## Help Click the icon of the owl 🦉 in the upper-left hand corner of to access the Arkime usage documentation (accessible at [https://localhost/help](https://localhost/help) if you are connecting locally), which includes such topics as [search syntax](https://localhost/help#search), the [Sessions view](https://localhost/help#sessions), [SPIView](https://localhost/help#spiview), [SPIGraph](https://localhost/help#spigraph), and the [Connections](https://localhost/help#connections) graph. -### Sessions +## Sessions The **Sessions** view provides low-level details of the sessions being investigated, whether they be Arkime sessions created from PCAP files or [Zeek logs mapped](#ArkimeZeek) to the Arkime session database schema. @@ -80,13 +80,13 @@ When viewing Arkime session details (ie., a session generated from a PCAP file), See also Arkime's usage documentation for more information on the [Sessions view](https://localhost/help#sessions). -#### PCAP Export +### PCAP Export Clicking the down arrow **▼** icon to the far right of the search bar presents a list of actions including **PCAP Export** (see Arkime's [sessions help](https://localhost/help#sessions) for information on the other actions). When full PCAP sessions are displayed, the **PCAP Export** feature allows you to create a new PCAP file from the matching Arkime sessions, including controls for which sessions are included (open items, visible items, or all matching items) and whether or not to include linked segments. Click **Export PCAP** button to generate the PCAP, after which you'll be presented with a browser download dialog to save or open the file. Note that depending on the scope of the filters specified this might take a long time (or, possibly even time out). ![Export PCAP](./images/screenshots/arkime_export_pcap.png) -### SPIView +## SPIView Arkime's **SPI** (**S**ession **P**rofile **I**nformation) **View** provides a quick and easy-to-use interface for exploring session/log metrics. The SPIView page lists categories for general session metrics (e.g., protocol, source and destination IP addresses, sort and destination ports, etc.) as well as for all of various types of network traffic understood by Malcolm. These categories can be expanded and the top *n* values displayed, along with each value's cardinality, for the fields of interest they contain. @@ -98,7 +98,7 @@ Note that because the SPIView page can potentially run many queries, SPIView lim See also Arkime's usage documentation for more information on [SPIView](https://localhost/help#spiview). -### SPIGraph +## SPIGraph Arkime's **SPI** (**S**ession **P**rofile **I**nformation) **Graph** visualizes the occurrence of some field's top *n* values over time, and (optionally) geographically. This is particularly useful for identifying trends in a particular type of communication over time: traffic using a particular protocol when seen sparsely at regular intervals on that protocol's date histogram in the SPIGraph may indicate a connection check, polling, or beaconing (for example, see the `llmnr` protocol in the screenshot below). @@ -108,7 +108,7 @@ Controls can be found underneath the time bounding controls for selecting the fi See also Arkime's usage documentation for more information on [SPIGraph](https://localhost/help#spigraph). -### Connections +## Connections The **Connections** page presents network communications via a force-directed graph, making it easy to visualize logical relationships between network hosts. @@ -127,7 +127,7 @@ or any other combination of these or other fields. See also Arkime's usage documentation for more information on the [Connections graph](https://localhost/help#connections). -### Hunt +## Hunt Arkime's **Hunt** feature allows an analyst to search within the packets themselves (including payload data) rather than simply searching the session metadata. The search string may be specified using ASCII (with or without case sensitivity), hex codes, or regular expressions. Once a hunt job is complete, matching sessions can be viewed in the [Sessions](#ArkimeSessions) view. @@ -157,7 +157,7 @@ The hunt feature is available only for sessions created from full packet capture See also Arkime's usage documentation for more information on the [hunt feature](https://localhost/help#hunt). -### Statistics +## Statistics Arkime provides several other reports which show information about the state of Arkime and the underlying OpenSearch database. @@ -175,9 +175,9 @@ The **History** view provides a historical list of queries issues to Arkime and See also Arkime's usage documentation for more information on the [Files list](https://localhost/help#files), [statistics](https://localhost/help#files), and [history](https://localhost/help#history). -### Settings +## Settings -#### General settings +### General settings The **Settings** page can be used to tweak Arkime preferences, defined additional custom views and column configurations, tweak the color theme, and more. diff --git a/docs/authsetup.md b/docs/authsetup.md index bd813e27d..4a0fe6a10 100644 --- a/docs/authsetup.md +++ b/docs/authsetup.md @@ -1,4 +1,4 @@ -### Configure authentication +# Configure authentication * [Configure authentication](#AuthSetup) - [Local account management](#AuthBasicAccountManagement) @@ -25,7 +25,7 @@ In either case, you **must** run `./scripts/auth_setup` before starting Malcolm * specify whether or not to [store the username/password](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#authenticate-sender-account) for [email alert senders](https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#create-destinations) * these parameters are stored securely in the OpenSearch keystore file `opensearch/opensearch.keystore` -##### Local account management +# Local account management [`auth_setup`](#AuthSetup) is used to define the username and password for the administrator account. Once Malcolm is running, the administrator account can be used to manage other user accounts via a **Malcolm User Management** page served over HTTPS on port 488 (e.g., [https://localhost:488](https://localhost:488) if you are connecting locally). @@ -33,7 +33,7 @@ Malcolm user accounts can be used to access the [interfaces](quickstart.md#UserI Users may change their passwords via the **Malcolm User Management** page by clicking **User Self Service**. A forgotten password can also be reset via an emailed link, though this requires SMTP server settings to be specified in `htadmin/config.ini` in the Malcolm installation directory. -#### Lightweight Directory Access Protocol (LDAP) authentication +## Lightweight Directory Access Protocol (LDAP) authentication The [nginx-auth-ldap](https://github.com/kvspb/nginx-auth-ldap) module serves as the interface between Malcolm's [Nginx](https://nginx.org/) web server and a remote LDAP server. When you run [`auth_setup`](#AuthSetup) for the first time, a sample LDAP configuration file is created at `nginx/nginx_ldap.conf`. @@ -74,7 +74,7 @@ Before starting Malcolm, edit `nginx/nginx_ldap.conf` according to the specifics The **Malcolm User Management** page described above is not available when using LDAP authentication. -##### LDAP connection security +# LDAP connection security Authentication over LDAP can be done using one of three ways, [two of which](https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/8e73932f-70cf-46d6-88b1-8d9f86235e81) offer data confidentiality protection: @@ -96,7 +96,7 @@ In addition to the `NGINX_BASIC_AUTH` environment variable being set to `false` For encrypted connections (whether using **StartTLS** or **LDAPS**), Malcolm will require and verify certificates when one or more trusted CA certificate files are placed in the `nginx/ca-trust/` directory. Otherwise, any certificate presented by the domain server will be accepted. -### TLS certificates +# TLS certificates When you [set up authentication](#AuthSetup) for Malcolm a set of unique [self-signed](https://en.wikipedia.org/wiki/Self-signed_certificate) TLS certificates are created which are used to secure the connection between clients (e.g., your web browser) and Malcolm's browser-based interface. This is adequate for most Malcolm instances as they are often run locally or on internal networks, although your browser will most likely require you to add a security exception for the certificate the first time you connect to Malcolm. diff --git a/docs/components.md b/docs/components.md index 21cf1fdf3..c885f477d 100644 --- a/docs/components.md +++ b/docs/components.md @@ -1,4 +1,4 @@ -## Components +# Components Malcolm leverages the following excellent open source tools, among others. diff --git a/docs/contributing-dashboards.md b/docs/contributing-dashboards.md index 25c487692..785c9fe2a 100644 --- a/docs/contributing-dashboards.md +++ b/docs/contributing-dashboards.md @@ -1,8 +1,8 @@ -## OpenSearch Dashboards +# OpenSearch Dashboards [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/index/) is an open-source fork of [Kibana](https://www.elastic.co/kibana/), which is [no longer open-source software](https://github.com/idaholab/Malcolm/releases/tag/v5.0.0). -### Adding new visualizations and dashboards +## Adding new visualizations and dashboards Visualizations and dashboards can be [easily created](dashboards.md#BuildDashboard) in OpenSearch Dashboards using its drag-and-drop WYSIWIG tools. Assuming you've created a new dashboard you wish to package with Malcolm, the dashboard and its visualization components can be exported using the following steps: @@ -34,7 +34,7 @@ Visualizations and dashboards can be [easily created](dashboards.md#BuildDashboa ``` 1. Include the new dashboard either by using a [bind mount](contributing-local-modifications.md#Bind) for the `./dashboards./dashboards/` directory or by [rebuilding](development.md#Build) the `dashboards-helper` Docker image. Dashboards are imported the first time Malcolm starts up. -### OpenSearch Dashboards plugins +## OpenSearch Dashboards plugins The [dashboards.Dockerfile](../Dockerfiles/dashboards.Dockerfile) installs the OpenSearch Dashboards plugins used by Malcolm (search for `opensearch-dashboards-plugin install` in that file). Additional Dashboards plugins could be installed by modifying this Dockerfile and [rebuilding](development.md#Build) the `dashboards` Docker image. diff --git a/docs/contributing-file-scanners.md b/docs/contributing-file-scanners.md index 1ca722a00..2118516d8 100644 --- a/docs/contributing-file-scanners.md +++ b/docs/contributing-file-scanners.md @@ -1,4 +1,4 @@ -## Carved file scanners +# Carved file scanners Similar to the [PCAP processing pipeline](contributing-pcap.md#PCAP) described above, new tools can plug into Malcolm's [automatic file extraction and scanning](file-scanning.md#ZeekFileExtraction) to examine file transfers carved from network traffic. diff --git a/docs/contributing-guide.md b/docs/contributing-guide.md index 59faa820f..47b984eeb 100644 --- a/docs/contributing-guide.md +++ b/docs/contributing-guide.md @@ -2,8 +2,7 @@ The purpose of this document is to provide some direction for those willing to modify Malcolm, whether for local customization or for contribution to the Malcolm project. -## Table of Contents - + * [Local modifications](contributing-local-modifications.md#LocalMods) + [Docker bind mounts](contributing-local-modifications.md#Bind) + [Building Malcolm's Docker images](contributing-local-modifications.md#ContribBuild) diff --git a/docs/contributing-local-modifications.md b/docs/contributing-local-modifications.md index b6f517ada..d69ee9674 100644 --- a/docs/contributing-local-modifications.md +++ b/docs/contributing-local-modifications.md @@ -1,8 +1,8 @@ -## Local modifications +# Local modifications There are several ways to customize Malcolm's runtime behavior via local changes to configuration files. Many commonly-tweaked settings are discussed in the project [README](README.md) (see [`docker-compose.yml` parameters](malcolm-config.md#DockerComposeYml) and [Customizing event severity scoring](severity.md#SeverityConfig) for some examples). -### Docker bind mounts +## Docker bind mounts Some configuration changes can be put in place by modifying local copies of configuration files and then use a [Docker bind mount](https://docs.docker.com/storage/bind-mounts/) to overlay the modified file onto the running Malcolm container. This is already done for many files and directories used to persist Malcolm configuration and data. For example, the default list of bind mounted files and directories for each Malcolm service is as follows: @@ -117,7 +117,7 @@ The change would take effect after stopping and starting Malcolm. See the documentation on [Docker bind mount](https://docs.docker.com/storage/bind-mounts/) for more information on this technique. -### Building Malcolm's Docker images +## Building Malcolm's Docker images Another method for modifying your local copies of Malcolm's services' containers is to [build your own](development.md#Build) containers with the modifications baked-in. diff --git a/docs/contributing-logstash.md b/docs/contributing-logstash.md index 9fb0a9fbe..610fda03e 100644 --- a/docs/contributing-logstash.md +++ b/docs/contributing-logstash.md @@ -1,6 +1,6 @@ -## Logstash +# Logstash -### Parsing a new log data source +## Parsing a new log data source Let's continue with the example of the `cooltool` service we added in the [PCAP processors](contributing-pcap.md#PCAP) section above, assuming that `cooltool` generates some textual log files we want to parse and index into Malcolm. @@ -30,7 +30,7 @@ So, in order to add a new **parse pipeline** for `cooltool` after tweaking [`fil Finally, in your `docker-compose` files, set a new `LOGSTASH_PARSE_PIPELINE_ADDRESSES` environment variable under `logstash-variables` to `cooltool-parse,zeek-parse,suricata-parse,beats-parse` (assuming you named the pipeline address from the previous step `cooltool-parse`) so that logs sent from `filebeat` to `logstash` are forwarded to all parse pipelines. -### Parsing new Zeek logs +## Parsing new Zeek logs The following modifications must be made in order for Malcolm to be able to parse new Zeek log files: @@ -42,10 +42,10 @@ The following modifications must be made in order for Malcolm to be able to pars 1. If necessary, define conversions for floating point or integer values in [`logstash/pipelines/zeek/11_zeek_logs.conf`](../logstash/pipelines/zeek/13_zeek_convert.conf) 1. Identify the new fields and add them as described in [Adding new log fields](contributing-new-log-fields.md#NewFields) -### Enrichments +## Enrichments Malcolm's Logstash instance will do a lot of enrichments for you automatically: see the [enrichment pipeline](../logstash/pipelines/enrichment), including MAC address to vendor by OUI, GeoIP, ASN, and a few others. In order to take advantage of these enrichments that are already in place, normalize new fields to use the same standardized field names Malcolm uses for things like IP addresses, MAC addresses, etc. You can add your own additional enrichments by creating new `.conf` files containing [Logstash filters](https://www.elastic.co/guide/en/logstash/7.10/filter-plugins.html) in the [enrichment pipeline](../logstash/pipelines/enrichment) directory and using either of the techniques in the [Local modifications](contributing-local-modifications.md#LocalMods) section to implement your changes in the `logstash` container -### Logstash plugins +## Logstash plugins The [logstash.Dockerfile](../Dockerfiles/logstash.Dockerfile) installs the Logstash plugins used by Malcolm (search for `logstash-plugin install` in that file). Additional Logstash plugins could be installed by modifying this Dockerfile and [rebuilding](development.md#Build) the `logstash` Docker image. \ No newline at end of file diff --git a/docs/contributing-new-image.md b/docs/contributing-new-image.md index 47e447bb0..e8c8fb566 100644 --- a/docs/contributing-new-image.md +++ b/docs/contributing-new-image.md @@ -1,4 +1,4 @@ -## Adding a new service (Docker image) +# Adding a new service (Docker image) A new service can be added to Malcolm by following the following steps: @@ -7,7 +7,7 @@ A new service can be added to Malcolm by following the following steps: 1. Add a new section for your service under `services:` in the `docker-compose.yml` and `docker-compose-standalone.yml` files 1. If you want to enable automatic builds for your service on GitHub, create a new [workflow](../.github/workflows/), using an existing workflow as an example -### Networking and firewall +## Networking and firewall If your service needs to expose a web interface to the user, you'll need to adjust the following files: diff --git a/docs/contributing-new-log-fields.md b/docs/contributing-new-log-fields.md index 98706d912..17d7c09bb 100644 --- a/docs/contributing-new-log-fields.md +++ b/docs/contributing-new-log-fields.md @@ -1,4 +1,4 @@ -## Adding new log fields +# Adding new log fields As several of the sections in this document will reference adding new data source fields, we'll cover that here at the beginning. diff --git a/docs/contributing-pcap.md b/docs/contributing-pcap.md index 3f0bde6f7..a896f19d8 100644 --- a/docs/contributing-pcap.md +++ b/docs/contributing-pcap.md @@ -1,4 +1,4 @@ -## PCAP processors +# PCAP processors When a PCAP is uploaded (either through Malcolm's [upload web interface](upload.md#Upload) or just copied manually into the `./pcap/upload/` directory), the `pcap-monitor` container has a script that picks up those PCAP files and publishes to a [ZeroMQ](https://zeromq.org/) topic that can be subscribed to by any other process that wants to analyze that PCAP. In Malcolm at the time of this writing (as of the [v5.0.0 release](https://github.com/idaholab/Malcolm/releases/tag/v5.0.0)), there are two of those: the `zeek` container and the `arkime` container. In Malcolm, they actually both share the [same script](../shared/bin/pcap_processor.py) to read from that topic and run the PCAP through Zeek and Arkime, respectively. If you're looking for an example to follow, the `zeek` container is the less complicated of the two. So, if you were looking to integrate a new PCAP processing tool into Malcolm (named `cooltool` for this example), the process would be something like: diff --git a/docs/contributing-style.md b/docs/contributing-style.md index 7fdcb819c..9211b23e1 100644 --- a/docs/contributing-style.md +++ b/docs/contributing-style.md @@ -1,5 +1,5 @@ -## Style +# Style -### Python +## Python For Python code found in Malcolm, the author uses [Black: The uncompromising Python code formatter](https://github.com/psf/black) with the options `--line-length 120 --skip-string-normalization`. \ No newline at end of file diff --git a/docs/contributing-zeek.md b/docs/contributing-zeek.md index 68aa3ff94..9564681de 100644 --- a/docs/contributing-zeek.md +++ b/docs/contributing-zeek.md @@ -1,15 +1,15 @@ -## Zeek +# Zeek -### `local.zeek` +## `local.zeek` Some Zeek behavior can be tweaked without having to manually edit configuration files through the use of environment variables: search for `ZEEK` in the [`docker-compose.yml` parameters](malcolm-config.md#DockerComposeYml) section of the documentation. Other changes to Zeek's behavior could be made by modifying [local.zeek](../zeek/config/local.zeek) and either using a [bind mount](contributing-local-modifications.md#Bind) or [rebuilding](development.md#Build) the `zeek` Docker image with the modification. See the [Zeek documentation](https://docs.zeek.org/en/master/quickstart.html#local-site-customization) for more information on customizing a Zeek instance. Note that changing Zeek's behavior could result in changes to the format of the logs Zeek generates, which could break Malcolm's parsing of those logs, so exercise caution. -### Adding a new Zeek package +## Adding a new Zeek package The easiest way to add a new Zeek package to Malcolm is to add the git URL of that package to the `ZKG_GITHUB_URLS` array in [zeek_install_plugins.sh](../shared/bin/zeek_install_plugins.sh) script and then [rebuilding](development.md#Build) the `zeek` Docker image. This will cause your package to be installed (via the [`zkg`](https://docs.zeek.org/projects/package-manager/en/stable/zkg.html) command-line tool). See [Parsing new Zeek logs](contributing-logstash.md#LogstashZeek) on how to process any new `.log` files if your package generates them. -### Zeek Intelligence Framework +## Zeek Intelligence Framework See [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) in the Malcolm README for information on how to use Zeek's [Intelligence Framework](https://docs.zeek.org/en/master/frameworks/intel.html) with Malcolm. \ No newline at end of file diff --git a/docs/cyberchef.md b/docs/cyberchef.md index 3f8c7c98e..c9a6b2aeb 100644 --- a/docs/cyberchef.md +++ b/docs/cyberchef.md @@ -1,4 +1,4 @@ -### CyberChef +# CyberChef Malcolm provides an instance of [CyberChef](https://github.com/gchq/CyberChef), the "Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis." CyberChef is available at at [https://localhost/cyberchef.html](https://localhost/cyberchef.html) if you are connecting locally. diff --git a/docs/dashboards.md b/docs/dashboards.md index 76038c199..7a5f5fc00 100644 --- a/docs/dashboards.md +++ b/docs/dashboards.md @@ -1,4 +1,4 @@ -## OpenSearch Dashboards +# OpenSearch Dashboards * [OpenSearch Dashboards](#Dashboards) - [Discover](#Discover) @@ -15,14 +15,14 @@ The OpenSearch Dashboards container can be accessed at [https://localhost/dashbo OpenSearch Dashboards has several components for data searching and visualization: -### Discover +## Discover The **Discover** view enables you to view events on a record-by-record basis (similar to a *session* record in Arkime or an individual line from a Zeek log). See the official [Kibana User Guide](https://www.elastic.co/guide/en/kibana/7.10/index.html) (OpenSearch Dashboards is an open-source fork of Kibana, which is no longer open-source software) for information on using the Discover view: * [Discover](https://www.elastic.co/guide/en/kibana/7.10/discover.html) * [Searching Your Data](https://www.elastic.co/guide/en/kibana/7.10/search.html) -#### Screenshots +### Screenshots ![Discover view](./images/screenshots/dashboards_discover.png) @@ -34,15 +34,15 @@ The **Discover** view enables you to view events on a record-by-record basis (si ![Opening a previously-saved search](./images/screenshots/dashboards_open_search.png) -### Visualizations and dashboards +## Visualizations and dashboards -#### Prebuilt visualizations and dashboards +### Prebuilt visualizations and dashboards Malcolm comes with dozens of prebuilt visualizations and dashboards for the network traffic represented by each of the Zeek log types. Click **Dashboard** to see a list of these dashboards. As is the case with all OpenSearch Dashboards visualizations, all of the charts, graphs, maps, and tables are interactive and can be clicked on to narrow or expand the scope of the data you are investigating. Similarly, click **Visualize** to explore the prebuilt visualizations used to build the dashboards. Many of Malcolm's prebuilt visualizations for Zeek logs were originally inspired by the excellent [Kibana Dashboards](https://github.com/Security-Onion-Solutions/securityonion-elastic/tree/master/kibana/dashboards) that are part of [Security Onion](https://securityonion.net/). -##### Screenshots +#### Screenshots ![The Security Overview highlights security-related network events](./images/screenshots/dashboards_security_overview.png) @@ -86,7 +86,7 @@ Many of Malcolm's prebuilt visualizations for Zeek logs were originally inspired ![S7comm is a Siemens proprietary protocol that runs between programmable logic controllers (PLCs) of the Siemens family](./images/screenshots/dashboards_s7comm.png) -#### Building your own visualizations and dashboards +### Building your own visualizations and dashboards See the official [Kibana User Guide](https://www.elastic.co/guide/en/kibana/7.10/index.html) and [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/index/) (OpenSearch Dashboards is an open-source fork of Kibana, which is no longer open-source software) documentation for information on creating your own visualizations and dashboards: @@ -94,6 +94,6 @@ See the official [Kibana User Guide](https://www.elastic.co/guide/en/kibana/7.10 * [Kibana Dashboards](https://www.elastic.co/guide/en/kibana/7.10/dashboard.html) * [TimeLine](https://www.elastic.co/guide/en/kibana/7.12/timelion.html) -##### Screenshots +#### Screenshots ![OpenSearch dashboards boasts many types of visualizations for displaying your data](./images/screenshots/dashboards_new_visualization.png) \ No newline at end of file diff --git a/docs/development.md b/docs/development.md index d1f3e9dcd..06e3c77ef 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,4 +1,4 @@ -## Development +# Development * [Development](#Development) - [Building from source](#Build) @@ -46,7 +46,7 @@ and the following files of special note: * `docker-compose.yml` - the configuration file used by `docker-compose` to build, start, and stop an instance of the Malcolm appliance * `docker-compose-standalone.yml` - similar to `docker-compose.yml`, only used for the ["packaged"](#Packager) installation of Malcolm -### Building from source +## Building from source Building the Malcolm docker images from scratch requires internet access to pull source files for its components. Once internet access is available, execute the following command to build all of the Docker images used by the Malcolm appliance: @@ -79,9 +79,9 @@ Then, go take a walk or something since it will be a while. When you're done, yo Alternately, if you have forked Malcolm on GitHub, [workflow files](./.github/workflows/) are provided which contain instructions for GitHub to build the docker images and [sensor](live-analysis.md#Hedgehog) and [Malcolm](malcolm-iso.md#ISO) installer ISOs. The resulting images are named according to the pattern `ghcr.io/owner/malcolmnetsec/image:branch` (e.g., if you've forked Malcolm with the github user `romeogdetlevjr`, the `arkime` container built for the `main` would be named `ghcr.io/romeogdetlevjr/malcolmnetsec/arkime:main`). To run your local instance of Malcolm using these images instead of the official ones, you'll need to edit your `docker-compose.yml` file(s) and replace the `image:` tags according to this new pattern, or use the bash helper script `./shared/bin/github_image_helper.sh` to pull and re-tag the images. -## Pre-Packaged installation files +# Pre-Packaged installation files -### Creating pre-packaged installation files +## Creating pre-packaged installation files `scripts/malcolm_appliance_packager.sh` can be run to package up the configuration files (and, if necessary, the Docker images) which can be copied to a network share or USB drive for distribution to non-networked machines. For example: @@ -149,7 +149,7 @@ total 2.0G -rw-r--r-- 1 user user 183k May 13 11:32 malcolm_20190513_101117_f0d052c.tar.gz ``` -### Installing from pre-packaged installation files +## Installing from pre-packaged installation files If you have obtained pre-packaged installation files to install Malcolm on a non-networked machine via an internal network share or on a USB key, you likely have the following files: diff --git a/docs/file-scanning.md b/docs/file-scanning.md index 914894eb3..6d6396a9e 100644 --- a/docs/file-scanning.md +++ b/docs/file-scanning.md @@ -1,4 +1,4 @@ -### Automatic file extraction and scanning +# Automatic file extraction and scanning Malcolm can leverage Zeek's knowledge of network protocols to automatically detect file transfers and extract those files from PCAPs as Zeek processes them. This behavior can be enabled globally by modifying the `ZEEK_EXTRACTOR_MODE` [environment variable in `docker-compose.yml`](malcolm-config.md#DockerComposeYml), or on a per-upload basis for PCAP files uploaded via the [browser-based upload form](upload.md#Upload) when **Analyze with Zeek** is selected. diff --git a/docs/hardening.md b/docs/hardening.md index 793a1df04..71fa0996b 100644 --- a/docs/hardening.md +++ b/docs/hardening.md @@ -1,4 +1,4 @@ -### Hardening +# Hardening * [Hardening](#Hardening) - [Compliance Exceptions](#ComplianceExceptions) @@ -9,7 +9,7 @@ The Malcolm aggregator base operating system uses the [harbian-audit](https://gi * [DISA STIG (Security Technical Implementation Guides) for RHEL 7](https://www.stigviewer.com/stig/red_hat_enterprise_linux_7/) v2r5 Ubuntu v1r2 [adapted](https://github.com/hardenedlinux/STIG-OS-mirror/blob/master/redhat-STIG-DOCs/U_Red_Hat_Enterprise_Linux_7_V2R5_STIG.zip) for a Debian operating system * Additional recommendations from [cisecurity.org](https://www.cisecurity.org/) -#### Compliance Exceptions +## Compliance Exceptions [Currently](https://github.com/hardenedlinux/harbian-audit/tree/master/bin/hardening) there are 274 checks to determine compliance with the [harbian-audit](https://github.com/hardenedlinux/harbian-audit) benchmark. diff --git a/docs/hedgehog-config-root.md b/docs/hedgehog-config-root.md index 5a6ab068d..357672e23 100644 --- a/docs/hedgehog-config-root.md +++ b/docs/hedgehog-config-root.md @@ -1,6 +1,6 @@ -## Interfaces, hostname, and time synchronization +# Interfaces, hostname, and time synchronization -### Hostname +## Hostname The first step of sensor configuration is to configure the network interfaces and sensor hostname. Clicking the **Configure Interfaces and Hostname** toolbar icon (or, if you are at a command line prompt, running `configure-interfaces`) will prompt you for the root password you created during installation, after which the configuration welcome screen is shown. Select **Continue** to proceed. @@ -12,7 +12,7 @@ Selecting **Hostname**, you will be presented with a summary of the current sens ![Specifying a new sensor hostname](./docs/images/hostname_setting.png) -### Interfaces +## Interfaces Returning to the configuration mode selection, choose **Interface**. You will be prompted if you would like help identifying network interfaces. If you select **Yes**, you will be prompted to select a network interface, after which that interface's link LED will blink for 10 seconds to help you in its identification. This network interface identification aid will continue to prompt you to identify further network interfaces until you select **No**. @@ -30,7 +30,7 @@ If you select static, you will be prompted to enter the IP address, netmask, and In either case, upon selecting **OK** the network interface will be brought down, configured, and brought back up, and the result of the operation will be displayed. You may choose **Quit** upon returning to the configuration tool's welcome screen. -### Time synchronization +## Time synchronization Returning to the configuration mode selection, choose **Time Sync**. Here you can configure the sensor to keep its time synchronized with either an NTP server (using the NTP protocol) or a local [Malcolm](https://github.com/idaholab/Malcolm) aggregator or another HTTP/HTTPS server. On the next dialog, choose the time synchronization method you wish to configure. diff --git a/docs/hedgehog-config-user.md b/docs/hedgehog-config-user.md index d39e44137..af1bae0af 100644 --- a/docs/hedgehog-config-user.md +++ b/docs/hedgehog-config-user.md @@ -1,10 +1,10 @@ -## Capture, forwarding, and autostart services +# Capture, forwarding, and autostart services Clicking the **Configure Capture and Forwarding** toolbar icon (or, if you are at a command prompt, running `configure-capture`) will launch the configuration tool for capture and forwarding. The root password is not required as it was for the interface and hostname configuration, as sensor services are run under the non-privileged sensor account. Select **Continue** to proceed. You may select from a list of configuration options. ![Select configuration mode](./docs/images/capture_config_main.png) -### Capture +## Capture Choose **Configure Capture** to configure parameters related to traffic capture and local analysis. You will be prompted if you would like help identifying network interfaces. If you select **Yes**, you will be prompted to select a network interface, after which that interface's link LED will blink for 10 seconds to help you in its identification. This network interface identification aid will continue to prompt you to identify further network interfaces until you select **No**. @@ -20,7 +20,7 @@ Next you must specify the paths where captured PCAP files and logs will be store ![Specify capture paths](./docs/images/capture_paths.png) -#### Automatic file extraction and scanning +### Automatic file extraction and scanning Hedgehog Linux can leverage Zeek's knowledge of network protocols to automatically detect file transfers and extract those files from network traffic as Zeek sees them. @@ -47,7 +47,7 @@ Files which are flagged as potentially malicious will be logged as Zeek `signatu Finally, you will be presented with the list of configuration variables that will be used for capture, including the values which you have configured up to this point in this section. Upon choosing **OK** these values will be written back out to the sensor configuration file located at `/opt/sensor/sensor_ctl/control_vars.conf`. It is not recommended that you edit this file manually. After confirming these values, you will be presented with a confirmation that these settings have been written to the configuration file, and you will be returned to the welcome screen. -### Forwarding +## Forwarding Select **Configure Forwarding** to set up forwarding logs and statistics from the sensor to an aggregator server, such as [Malcolm](https://github.com/idaholab/Malcolm). @@ -55,7 +55,7 @@ Select **Configure Forwarding** to set up forwarding logs and statistics from th There are five forwarder services used on the sensor, each for forwarding a different type of log or sensor metric. -### capture: Arkime session forwarding +## capture: Arkime session forwarding [capture](https://github.com/arkime/arkime/tree/master/capture) is not only used to capture PCAP files, but also the parse raw traffic into sessions and forward this session metadata to an [OpenSearch](https://opensearch.org/) database so that it can be viewed in [Arkime viewer](https://arkime.com/), whether standalone or as part of a [Malcolm](https://github.com/idaholab/Malcolm) instance. If you're using Hedgehog Linux with Malcolm, please read [Correlating Zeek logs and Arkime sessions](https://github.com/idaholab/Malcolm#ZeekArkimeFlowCorrelation) in the Malcolm documentation for more information. @@ -79,7 +79,7 @@ Finally, you'll be given the opportunity to review the all of the Arkime `captur ![capture settings confirmation](./docs/images/arkime_confirm.png) -### filebeat: Zeek and Suricata log forwarding +## filebeat: Zeek and Suricata log forwarding [Filebeat](https://www.elastic.co/products/beats/filebeat) is used to forward [Zeek](https://www.zeek.org/) and [Suricata](https://suricata.io/) logs to a remote [Logstash](https://www.elastic.co/products/logstash) instance for further enrichment prior to insertion into an [OpenSearch](https://opensearch.org/) database. @@ -109,13 +109,13 @@ Once you have specified all of the filebeat parameters, you will be presented wi ![Confirm filebeat settings](./docs/images/filebeat_confirm.png) -### miscbeat: System metrics forwarding +## miscbeat: System metrics forwarding The sensor uses [Fluent Bit](https://fluentbit.io/) to gather miscellaneous system resource metrics (CPU, network I/O, disk I/O, memory utilization, temperature, etc.) and the [Beats](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-tcp.html) protocol to forward these metrics to a remote [Logstash](https://www.elastic.co/products/logstash) instance for further enrichment prior to insertion into an [OpenSearch](https://opensearch.org/) database. Metrics categories can be enabled/disabled as described in the [autostart services](#HedgehogConfigAutostart) section of this document. This forwarder's configuration is almost identical to that of [filebeat](#Hedgehogfilebeat) in the previous section. Select `miscbeat` from the forwarding configuration mode options and follow the same steps outlined above to set up this forwarder. -### Autostart services +## Autostart services Once the forwarders have been configured, the final step is to **Configure Autostart Services**. Choose this option from the configuration mode menu after the welcome screen of the sensor configuration tool. diff --git a/docs/hedgehog-config-zeek-intel.md b/docs/hedgehog-config-zeek-intel.md index 5a10f9660..cef702d53 100644 --- a/docs/hedgehog-config-zeek-intel.md +++ b/docs/hedgehog-config-zeek-intel.md @@ -1,4 +1,4 @@ -### Zeek Intelligence Framework +# Zeek Intelligence Framework To quote Zeek's [Intelligence Framework](https://docs.zeek.org/en/master/frameworks/intel.html) documentation, "The goals of Zeek’s Intelligence Framework are to consume intelligence data, make it available for matching, and provide infrastructure to improve performance and memory utilization. Data in the Intelligence Framework is an atomic piece of intelligence such as an IP address or an e-mail address. This atomic data will be packed with metadata such as a freeform source field, a freeform descriptive field, and a URL which might lead to more information about the specific item." Zeek [intelligence](https://docs.zeek.org/en/master/scripts/base/frameworks/intel/main.zeek.html) [indicator types](https://docs.zeek.org/en/master/scripts/base/frameworks/intel/main.zeek.html#type-Intel::Type) include IP addresses, URLs, file names, hashes, email addresses, and more. diff --git a/docs/hedgehog.md b/docs/hedgehog.md index 14bebcf28..e735cd785 100644 --- a/docs/hedgehog.md +++ b/docs/hedgehog.md @@ -1,5 +1,6 @@ -# Hedgehog Linux -## Network Traffic Capture Appliance +# Hedgehog Linux + +**Network Traffic Capture Appliance** ![](./docs/logo/hedgehog-color-w-text.png) @@ -12,8 +13,7 @@ Hedgehog Linux is a Debian-based operating system built to ![sensor-iso-build-docker-wrap-push-ghcr](https://github.com/idaholab/Malcolm/workflows/sensor-iso-build-docker-wrap-push-ghcr/badge.svg) -### Table of Contents - + * [Sensor installation](hedgehog-installation.md#HedgehogInstallation) - [Image boot options](hedgehog-installation.md#HedgehogBootOptions) - [Installer](hedgehog-installation.md#HedgehogInstaller) diff --git a/docs/host-and-subnet-mapping.md b/docs/host-and-subnet-mapping.md index fec2cd2b0..c220a1622 100644 --- a/docs/host-and-subnet-mapping.md +++ b/docs/host-and-subnet-mapping.md @@ -1,4 +1,4 @@ -### Automatic host and subnet name assignment +# Automatic host and subnet name assignment * [Automatic host and subnet name assignment](host-and-subnet-mapping.md#HostAndSubnetNaming) - [IP/MAC address to hostname mapping via `host-map.txt`](host-and-subnet-mapping.md#HostNaming) @@ -6,7 +6,7 @@ - [Defining hostname and CIDR subnet names interface](host-and-subnet-mapping.md#NameMapUI) - [Applying mapping changes](host-and-subnet-mapping.md#ApplyMapping) -#### IP/MAC address to hostname mapping via `host-map.txt` +## IP/MAC address to hostname mapping via `host-map.txt` The `host-map.txt` file in the Malcolm installation directory can be used to define names for network hosts based on IP and/or MAC addresses in Zeek logs. The default empty configuration looks like this: ``` @@ -35,7 +35,7 @@ As Zeek logs are processed into Malcolm's OpenSearch instance, the log's source `source.hostname` and `destination.hostname` may each contain multiple values. For example, if both a host's source IP address and source MAC address were matched by two different lines, `source.hostname` would contain the hostname values from both matching lines. -#### CIDR subnet to network segment name mapping via `cidr-map.txt` +## CIDR subnet to network segment name mapping via `cidr-map.txt` The `cidr-map.txt` file in the Malcolm installation directory can be used to define names for network segments based on IP addresses in Zeek logs. The default empty configuration looks like this: ``` @@ -69,7 +69,7 @@ If both `source.segment` and `destination.segment` are added to a log, and if th ![Cross-segment traffic in Connections](./images/screenshots/arkime_connections_segments.png) -#### Defining hostname and CIDR subnet names interface +## Defining hostname and CIDR subnet names interface As an alternative to manually editing `cidr-map.txt` and `host-map.txt`, a **Host and Subnet Name Mapping** editor is available at [https://localhost/name-map-ui/](https://localhost/name-map-ui/) if you are connecting locally. Upon loading, the editor is populated from `cidr-map.txt`, `host-map.txt` and `net-map.json`. @@ -87,7 +87,7 @@ This editor provides the following controls: ![Host and Subnet Name Mapping Editor](./images/screenshots/malcolm_name_map_ui.png) -#### Applying mapping changes +## Applying mapping changes When changes are made to either `cidr-map.txt`, `host-map.txt` or `net-map.json`, Malcolm's Logstash container must be restarted. The easiest way to do this is to restart malcolm via `restart` (see [Stopping and restarting Malcolm](running.md#StopAndRestart)) or by clicking the 🔁 **Restart Logstash** button in the [name mapping interface](#NameMapUI) interface. diff --git a/docs/host-config-linux.md b/docs/host-config-linux.md index 517c5331a..db5c65db7 100644 --- a/docs/host-config-linux.md +++ b/docs/host-config-linux.md @@ -1,6 +1,6 @@ -#### Linux host system configuration +# Linux host system configuration -##### Installing Docker +## Installing Docker Docker installation instructions vary slightly by distribution. Please follow the links below to docker.com to find the instructions specific to your distribution: @@ -21,11 +21,11 @@ Docker starts automatically on DEB-based distributions. On RPM-based distributio You can test docker by running `docker info`, or (assuming you have internet access), `docker run --rm hello-world`. -##### Installing docker-compose +## Installing docker-compose Please follow [this link](https://docs.docker.com/compose/install/) on docker.com for instructions on installing docker-compose. -##### Operating system configuration +## Operating system configuration The host system (ie., the one running Docker) will need to be configured for the [best possible OpenSearch performance](https://www.elastic.co/guide/en/elasticsearch/reference/master/system-config.html). Here are a few suggestions for Linux hosts (these may vary from distribution to distribution): diff --git a/docs/host-config-macos.md b/docs/host-config-macos.md index 74b63ef91..74e277887 100644 --- a/docs/host-config-macos.md +++ b/docs/host-config-macos.md @@ -1,10 +1,10 @@ -#### macOS host system configuration +# macOS host system configuration -##### Automatic installation using `install.py` +## Automatic installation using `install.py` The `install.py` script will attempt to guide you through the installation of Docker and Docker Compose if they are not present. If that works for you, you can skip ahead to **Configure docker daemon option** in this section. -##### Install Homebrew +## Install Homebrew The easiest way to install and maintain docker on Mac is using the [Homebrew cask](https://brew.sh). Execute the following in a terminal. @@ -14,7 +14,7 @@ $ brew install cask $ brew tap homebrew/cask-versions ``` -##### Install docker-edge +## Install docker-edge ``` $ brew cask install docker-edge @@ -25,7 +25,7 @@ $ brew cask upgrade --no-quarantine docker-edge ``` You can now run docker from the Applications folder. -##### Configure docker daemon option +## Configure docker daemon option Some changes should be made for performance ([this link](http://markshust.com/2018/01/30/performance-tuning-docker-mac) gives a good succinct overview). diff --git a/docs/host-config-windows.md b/docs/host-config-windows.md index 428786fc1..a12716b33 100644 --- a/docs/host-config-windows.md +++ b/docs/host-config-windows.md @@ -1,6 +1,6 @@ -#### Windows host system configuration +# Windows host system configuration -##### Installing and configuring Docker Desktop for Windows +## Installing and configuring Docker Desktop for Windows Installing and configuring [Docker to run under Windows](https://docs.docker.com/desktop/windows/wsl/) must be done manually, rather than through the `install.py` script as is done for Linux and macOS. @@ -11,6 +11,6 @@ Installing and configuring [Docker to run under Windows](https://docs.docker.com 1. Reboot 1. Open the WSL distribution's terminal and run run `docker info` to make sure Docker is running -##### Finish Malcolm's configuration +## Finish Malcolm's configuration Once Docker is installed, configured and running as described in the previous section, run [`./scripts/install.py --configure`](malcolm-config.md#ConfigAndTuning) to finish configuration of the local Malcolm installation. Malcolm will be controlled and run from within your WSL distribution's terminal environment. \ No newline at end of file diff --git a/docs/host-config.md b/docs/host-config.md index 81e5802e1..f92375141 100644 --- a/docs/host-config.md +++ b/docs/host-config.md @@ -1,4 +1,4 @@ -## Platform-specific Configuration +# Platform-specific Configuration * [Linux host system configuration](host-config-linux.md#HostSystemConfigLinux) * [macOS host system configuration](host-config-macos.md#HostSystemConfigMac) diff --git a/docs/ics-best-guess.md b/docs/ics-best-guess.md index 6969706b3..7e6eb61cc 100644 --- a/docs/ics-best-guess.md +++ b/docs/ics-best-guess.md @@ -1,4 +1,4 @@ -### "Best Guess" Fingerprinting for ICS Protocols +# "Best Guess" Fingerprinting for ICS Protocols There are many ICS (industrial control systems) protocols. While Malcolm's collection of [protocol parsers](protocols.md#Protocols) includes a number of them, many, particularly those that are proprietary or less common, are unlikely to be supported with a full parser in the foreseeable future. diff --git a/docs/index-management.md b/docs/index-management.md index ee6ed7f76..2277fbd07 100644 --- a/docs/index-management.md +++ b/docs/index-management.md @@ -1,4 +1,4 @@ -### OpenSearch index management +# OpenSearch index management Malcolm releases prior to v6.2.0 used environment variables to configure OpenSearch [Index State Management](https://opensearch.org/docs/latest/im-plugin/ism/index/) [policies](https://opensearch.org/docs/latest/im-plugin/ism/policies/). diff --git a/docs/live-analysis.md b/docs/live-analysis.md index 40941d193..11d046ec1 100644 --- a/docs/live-analysis.md +++ b/docs/live-analysis.md @@ -1,11 +1,11 @@ -## Live analysis +# Live analysis * [Live analysis](#LiveAnalysis) - [Using a network sensor appliance](#Hedgehog) - [Monitoring local network interfaces](#LocalPCAP) - [Manually forwarding logs from an external source](#ExternalForward) -### Using a network sensor appliance +## Using a network sensor appliance A dedicated network sensor appliance is the recommended method for capturing and analyzing live network traffic when performance and throughput is of utmost importance. [Hedgehog Linux](hedgehog.md) is a custom Debian-based operating system built to: @@ -16,7 +16,7 @@ A dedicated network sensor appliance is the recommended method for capturing and Please see the [Hedgehog Linux README](hedgehog.md) for more information. -### Monitoring local network interfaces +## Monitoring local network interfaces Malcolm's `pcap-capture`, `suricata-live` and `zeek-live` containers can monitor one or more local network interfaces, specified by the `PCAP_IFACE` environment variable in [`docker-compose.yml`](malcolm-config.md#DockerComposeYml). These containers are started with additional privileges (`IPC_LOCK`, `NET_ADMIN`, `NET_RAW`, and `SYS_ADMIN`) to allow opening network interfaces in promiscuous mode for capture. @@ -28,7 +28,7 @@ These various options for monitoring traffic on local network interfaces can als Note that currently Microsoft Windows and Apple macOS platforms run Docker inside of a virtualized environment. Live traffic capture and analysis on those platforms would require additional configuration of virtual interfaces and port forwarding in Docker which is outside of the scope of this document. -### Manually forwarding logs from an external source +## Manually forwarding logs from an external source Malcolm's Logstash instance can also be configured to accept logs from a [remote forwarder](https://www.elastic.co/products/beats/filebeat) by running [`./scripts/install.py --configure`](malcolm-config.md#ConfigAndTuning) and answering "yes" to "`Expose Logstash port to external hosts?`." Enabling encrypted transport of these logs files is discussed in [Configure authentication](authsetup.md#AuthSetup) and the description of the `BEATS_SSL` environment variable in the [`docker-compose.yml`](malcolm-config.md#DockerComposeYml) file. diff --git a/docs/malcolm-config.md b/docs/malcolm-config.md index 8f932c31a..839bd7caf 100644 --- a/docs/malcolm-config.md +++ b/docs/malcolm-config.md @@ -1,4 +1,4 @@ -### Malcolm Configuration +# Malcolm Configuration If you already have Docker and Docker Compose installed, the `install.py` script can still help you tune system configuration and `docker-compose.yml` parameters for Malcolm. To run it in "configuration only" mode, bypassing the steps to install Docker and Docker Compose, run it like this: ``` @@ -7,7 +7,7 @@ If you already have Docker and Docker Compose installed, the `install.py` script Although `install.py` will attempt to automate many of the following configuration and tuning parameters, they are nonetheless listed in the following sections for reference: -#### `docker-compose.yml` parameters +## `docker-compose.yml` parameters Edit `docker-compose.yml` and search for the `OPENSEARCH_JAVA_OPTS` key. Edit the `-Xms4g -Xmx4g` values, replacing `4g` with a number that is half of your total system memory, or just under 32 gigabytes, whichever is less. So, for example, if I had 64 gigabytes of memory I would edit those values to be `-Xms31g -Xmx31g`. This indicates how much memory can be allocated to the OpenSearch heaps. For a pleasant experience, I would suggest not using a value under 10 gigabytes. Similar values can be modified for Logstash with `LS_JAVA_OPTS`, where using 3 or 4 gigabytes is recommended. diff --git a/docs/malcolm-iso.md b/docs/malcolm-iso.md index b558fdaf6..0b192818a 100644 --- a/docs/malcolm-iso.md +++ b/docs/malcolm-iso.md @@ -1,4 +1,4 @@ -## Malcolm installer ISO +# Malcolm installer ISO * [Malcolm installer ISO](#ISO) - [Installation](#ISOInstallation) @@ -10,7 +10,7 @@ Malcolm's Docker-based deployment model makes Malcolm able to run on a variety o Malcolm can be packaged into an installer ISO based on the current [stable release](https://wiki.debian.org/DebianStable) of [Debian](https://www.debian.org/). This [customized Debian installation](https://wiki.debian.org/DebianLive) is preconfigured with the bare minimum software needed to run Malcolm. -### Generating the ISO +## Generating the ISO Official downloads of the Malcolm installer ISO are not provided: however, it can be built easily on an internet-connected Linux host with Vagrant: @@ -56,7 +56,7 @@ A system installed from the resulting ISO will load the Malcolm Docker images up Alternately, if you have forked Malcolm on GitHub, [workflow files](./.github/workflows/) are provided which contain instructions for GitHub to build the docker images and [sensor](live-analysis.md#Hedgehog) and [Malcolm](#ISO) installer ISOs, specifically [`malcolm-iso-build-docker-wrap-push-ghcr.yml`](./.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml) for the Malcolm ISO. You'll need to run the workflows to build and push your fork's Malcolm docker images before building the ISO. The resulting ISO file is wrapped in a Docker image that provides an HTTP server from which the ISO may be downloaded. -### Installation +## Installation The installer is designed to require as little user input as possible. For this reason, there are NO user prompts and confirmations about partitioning and reformatting hard disks for use by the operating system. The installer assumes that all non-removable storage media (eg., SSD, HDD, NVMe, etc.) are available for use and ⛔🆘😭💀 ***will partition and format them without warning*** 💀😭🆘⛔. @@ -78,7 +78,7 @@ At the end of the installation process, you will be prompted with a few self-exp Following these prompts, the installer will reboot and the Malcolm base operating system will boot. -### Setup +## Setup When the system boots for the first time, the Malcolm Docker images will load if the installer was built with pre-packaged installation files as described above. Wait for this operation to continue (the progress dialog will disappear when they have finished loading) before continuing the setup. diff --git a/docs/malcolm-preparation.md b/docs/malcolm-preparation.md index 6ed65b6b2..6e0861c60 100644 --- a/docs/malcolm-preparation.md +++ b/docs/malcolm-preparation.md @@ -1,4 +1,4 @@ -### Configuration +# Configuration * [Configuration](#Configuration) - [Recommended system requirements](system-requirements.md#SystemRequirements) diff --git a/docs/malcolm-upgrade.md b/docs/malcolm-upgrade.md index fa071f8b5..8984164e5 100644 --- a/docs/malcolm-upgrade.md +++ b/docs/malcolm-upgrade.md @@ -1,14 +1,14 @@ -## Upgrading Malcolm +# Upgrading Malcolm At this time there is not an "official" upgrade procedure to get from one version of Malcolm to the next, as it may vary from platform to platform. However, the process is fairly simple can be done by following these steps: -### Update the underlying system +## Update the underlying system You may wish to get the official updates for the underlying system's software packages before you proceed. Consult the documentation of your operating system for how to do this. If you are upgrading an Malcolm instance installed from [Malcolm installation ISO](malcolm-iso.md#ISOInstallation), follow scenario 2 below. Due to the Malcolm base operating system's [hardened](hardening.md#Hardening) configuration, when updating the underlying system, temporarily set the umask value to Debian default (`umask 0022` in the root shell in which updates are being performed) instead of the more restrictive Malcolm default. This will allow updates to be applied with the right permissions. -### Scenario 1: Malcolm is a GitHub clone +## Scenario 1: Malcolm is a GitHub clone If you checked out a working copy of the Malcolm repository from GitHub with a `git clone` command, here are the basic steps to performing an upgrade: @@ -29,7 +29,7 @@ If you checked out a working copy of the Malcolm repository from GitHub with a ` 9. you may be prompted to [configure authentication](authsetup.md#AuthSetup) if there are new authentication-related files that need to be generated * you probably do not need to re-generate self-signed certificates -### Scenario 2: Malcolm was installed from a packaged tarball +## Scenario 2: Malcolm was installed from a packaged tarball If you installed Malcolm from [pre-packaged installation files](https://github.com/idaholab/Malcolm#Packager), here are the basic steps to perform an upgrade: @@ -54,9 +54,9 @@ If you installed Malcolm from [pre-packaged installation files](https://github.c 9. you may be prompted to [configure authentication](authsetup.md#AuthSetup) if there are new authentication-related files that need to be generated * you probably do not need to re-generate self-signed certificates -### Post-upgrade +## Post-upgrade -#### Monitoring Malcolm +### Monitoring Malcolm If you are technically-minded, you may wish to follow the debug output provided by `./scripts/start` (or `./scripts/logs` if you need to re-open the log stream after you've closed it), although there is a lot there and it may be hard to distinguish whether or not something is okay. @@ -64,10 +64,10 @@ Running `docker-compose ps -a` should give you a good idea if all of Malcolm's D After upgrading following one of the previous outlines, give Malcolm several minutes to get started. Once things are up and running, open one of Malcolm's [web interfaces](quickstart.md#UserInterfaceURLs) to verify that things are working. -#### Loading new OpenSearch Dashboards visualizations +### Loading new OpenSearch Dashboards visualizations Once the upgraded instance Malcolm has started up, you'll probably want to import the new dashboards and visualizations for OpenSearch Dashboards. You can signal Malcolm to load the new visualizations by opening OpenSearch Dashboards, clicking **Management** → **Index Patterns**, then selecting the `arkime_sessions3-*` index pattern and clicking the delete **🗑** button near the upper-right of the window. Confirm the **Delete index pattern?** prompt by clicking **Delete**. Close the OpenSearch Dashboards browser window. After a few minutes the missing index pattern will be detected and OpenSearch Dashboards will be signalled to load its new dashboards and visualizations. -### Major releases +## Major releases The Malcolm project uses [semantic versioning](https://semver.org/) when choosing version numbers. If you are moving between major releases (e.g., from v4.0.1 to v5.0.0), you're likely to find that there are enough major backwards compatibility-breaking changes that upgrading may not be worth the time and trouble. A fresh install is strongly recommended between major releases. \ No newline at end of file diff --git a/docs/netbox.md b/docs/netbox.md index 5694ec3a4..2ffd84120 100644 --- a/docs/netbox.md +++ b/docs/netbox.md @@ -1,4 +1,4 @@ -### Asset Management with NetBox +# Asset Management with NetBox Malcolm provides an instance of [NetBox](https://netbox.dev/), an open-source "solution for modeling and documenting modern networks." The NetBox web interface is available at at [https://localhost/netbox/](https://localhost/netbox/) if you are connecting locally. diff --git a/docs/opensearch-instances.md b/docs/opensearch-instances.md index 9d98f16e9..c77ab38c3 100644 --- a/docs/opensearch-instances.md +++ b/docs/opensearch-instances.md @@ -1,4 +1,4 @@ -### OpenSearch instances +# OpenSearch instances * [OpenSearch instances](#OpenSearchInstance) - [Authentication and authorization for remote OpenSearch clusters](#OpenSearchAuth) @@ -40,7 +40,7 @@ You must run auth_setup after install.py to store OpenSearch connection credenti … ``` -#### Authentication and authorization for remote OpenSearch clusters +## Authentication and authorization for remote OpenSearch clusters In addition to setting the environment variables in [`docker-compose.yml`](malcolm-config.md#DockerComposeYml) as described above, you must provide Malcolm with credentials for it to be able to communicate with remote OpenSearch instances. These credentials are stored in the Malcolm installation directory as `.opensearch.primary.curlrc` and `.opensearch.secondary.curlrc` for the primary and secondary OpenSearch connections, respectively, and are bind mounted into the Docker containers which need to communicate with OpenSearch. These [cURL-formatted](https://everything.curl.dev/cmdline/configfile) config files can be generated for you by the [`auth_setup`](authsetup.md#AuthSetup) script as illustrated: diff --git a/docs/protocols.md b/docs/protocols.md index c02ff690a..7d687e416 100644 --- a/docs/protocols.md +++ b/docs/protocols.md @@ -1,4 +1,4 @@ -## Supported Protocols +# Supported Protocols Malcolm uses [Zeek](https://docs.zeek.org/en/stable/script-reference/proto-analyzers.html) and [Arkime](https://github.com/arkime/arkime/tree/master/capture/parsers) to analyze network traffic. These tools provide varying degrees of visibility into traffic transmitted over the following network protocols: diff --git a/docs/queries-cheat-sheet.md b/docs/queries-cheat-sheet.md index 1dcce2802..751c4a65c 100644 --- a/docs/queries-cheat-sheet.md +++ b/docs/queries-cheat-sheet.md @@ -1,4 +1,4 @@ -## Search Queries in Arkime and OpenSearch Dashboards +# Search Queries in Arkime and OpenSearch Dashboards OpenSearch Dashboards supports two query syntaxes: the legacy [Lucene](https://www.elastic.co/guide/en/kibana/current/lucene-query.html) syntax and [Dashboards Query Language (DQL)](https://opensearch.org/docs/1.2/dashboards/dql/), both of which are somewhat different than Arkime's query syntax (see the help at [https://localhost/help#search](https://localhost/help#search) if you are connecting locally). The Arkime interface is for searching and visualizing both Arkime sessions and Zeek logs. The prebuilt dashboards in the OpenSearch Dashboards interface are for searching and visualizing Zeek logs, but will not include Arkime sessions. Here are some common patterns used in building search query strings for Arkime and OpenSearch Dashboards, respectively. See the links provided for further documentation. diff --git a/docs/quickstart.md b/docs/quickstart.md index b741efdc6..6abeae160 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,28 +1,28 @@ -## Quick start +# Quick start * [Quick start](#QuickStart) - [Getting Malcolm](#GetMalcolm) - [User interface](#UserInterfaceURLs) -### Getting Malcolm +## Getting Malcolm For a `TL;DR` example of downloading, configuring, and running Malcolm on a Linux platform, see [Installation example using Ubuntu 22.04 LTS](ubuntu-install-example.md#InstallationExample). The scripts to control Malcolm require Python 3. The [`install.py`](malcolm-config.md#ConfigAndTuning) script requires the [requests](https://docs.python-requests.org/en/latest/) module for Python 3, and will make use of the [pythondialog](https://pythondialog.sourceforge.io/) module for user interaction (on Linux) if it is available. -#### Source code +### Source code The files required to build and run Malcolm are available on its [GitHub page](https://github.com/idaholab/Malcolm/tree/main). Malcolm's source code is released under the terms of a permissive open source software license (see see `License.txt` for the terms of its release). -#### Building Malcolm from scratch +### Building Malcolm from scratch The `build.sh` script can build Malcolm's Docker images from scratch. See [Building from source](development.md#Build) for more information. -#### Initial configuration +### Initial configuration You must run [`auth_setup`](authsetup.md#AuthSetup) prior to pulling Malcolm's Docker images. You should also ensure your system configuration and `docker-compose.yml` settings are tuned by running `./scripts/install.py` or `./scripts/install.py --configure` (see [System configuration and tuning](malcolm-config.md#ConfigAndTuning)). -#### Pull Malcolm's Docker images +### Pull Malcolm's Docker images Malcolm's Docker images are periodically built and hosted on [Docker Hub](https://hub.docker.com/u/malcolmnetsec). If you already have [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/), these prebuilt images can be pulled by navigating into the Malcolm directory (containing the `docker-compose.yml` file) and running `docker-compose pull` like this: ``` @@ -75,16 +75,16 @@ malcolmnetsec/suricata 6.4.0 x malcolmnetsec/zeek 6.4.0 xxxxxxxxxxxx 3 days ago 1GB ``` -#### Import from pre-packaged tarballs +### Import from pre-packaged tarballs Once built, the `malcolm_appliance_packager.sh` script can be used to create pre-packaged Malcolm tarballs for import on another machine. See [Pre-Packaged Installation Files](development.md#Packager) for more information. -### Starting and stopping Malcolm +## Starting and stopping Malcolm Use the scripts in the `scripts/` directory to start and stop Malcolm, view debug logs of a currently running instance, wipe the database and restore Malcolm to a fresh state, etc. -### User interface +## User interface A few minutes after starting Malcolm (probably 5 to 10 minutes for Logstash to be completely up, depending on the system), the following services will be accessible: diff --git a/docs/running.md b/docs/running.md index 8473dc353..a2db0d40d 100644 --- a/docs/running.md +++ b/docs/running.md @@ -1,4 +1,4 @@ -## Running Malcolm +# Running Malcolm * [Running Malcolm](#Running) - [OpenSearch instances](opensearch-instances.md#OpenSearchInstance) @@ -8,7 +8,7 @@ - [Clearing Malcolm's data](#Wipe) - [Temporary read-only interface](#ReadOnlyUI) -### Starting Malcolm +## Starting Malcolm [Docker compose](https://docs.docker.com/compose/) is used to coordinate running the Docker containers. To start Malcolm, navigate to the directory containing `docker-compose.yml` and run: ``` @@ -20,17 +20,17 @@ $ ./scripts/logs ``` You can also use `docker stats` to monitor the resource utilization of running containers. -### Stopping and restarting Malcolm +## Stopping and restarting Malcolm You can run `./scripts/stop` to stop the docker containers and remove their virtual network. Alternatively, `./scripts/restart` will restart an instance of Malcolm. Because the data on disk is stored on the host in docker volumes, doing these operations will not result in loss of data. Malcolm can be configured to be automatically restarted when the Docker system daemon restart (for example, on system reboot). This behavior depends on the [value](https://docs.docker.com/config/containers/start-containers-automatically/) of the [`restart:`](https://docs.docker.com/compose/compose-file/#restart) setting for each service in the `docker-compose.yml` file. This value can be set by running [`./scripts/install.py --configure`](malcolm-config.md#ConfigAndTuning) and answering "yes" to "`Restart Malcolm upon system or Docker daemon restart?`." -### Clearing Malcolm's data +## Clearing Malcolm's data Run `./scripts/wipe` to stop the Malcolm instance and wipe its OpenSearch database (**including** [index snapshots and management policies](index-management.md#IndexManagement) and [alerting configuration](alerting.md#Alerting)). -### Temporary read-only interface +## Temporary read-only interface To temporarily set the Malcolm user interaces into a read-only configuration, run the following commands from the Malcolm installation directory. diff --git a/docs/severity.md b/docs/severity.md index 676c6a169..2d8da26f0 100644 --- a/docs/severity.md +++ b/docs/severity.md @@ -1,4 +1,4 @@ -### Event severity scoring +# Event severity scoring * [Event severity scoring](#Severity) - [Customizing event severity scoring](#SeverityConfig) @@ -30,7 +30,7 @@ When a Zeek log satisfies more than one of these conditions its severity scores ![The Severity dashboard](./images/screenshots/dashboards_severity.png) -#### Customizing event severity scoring +## Customizing event severity scoring These categories' severity scores can be customized by editing `logstash/maps/malcolm_severity.yaml`: diff --git a/docs/system-requirements.md b/docs/system-requirements.md index b25d80c47..e789504c1 100644 --- a/docs/system-requirements.md +++ b/docs/system-requirements.md @@ -1,4 +1,4 @@ -### Recommended system requirements +# Recommended system requirements Malcolm runs on top of [Docker](https://www.docker.com/) which runs on recent releases of Linux, Apple macOS and Microsoft Windows 10. diff --git a/docs/third-party-logs.md b/docs/third-party-logs.md index 7cf923c69..804e8f66c 100644 --- a/docs/third-party-logs.md +++ b/docs/third-party-logs.md @@ -14,8 +14,7 @@ Malcolm uses [OpenSearch](https://opensearch.org/) and [OpenSearch Dashboards](h The types of third-party logs and metrics discussed in this document are *not* the same as the network session metadata provided by Arkime, Zeek and Suricata. Please refer to the [Malcolm Contributor Guide](contributing-guide.md) for information on integrating a new network traffic analysis provider. -## Table of Contents - + * [Configuring Malcolm](#Malcolm) - [Secure communication](#MalcolmTLS) * [Fluent Bit](#FluentBit) diff --git a/docs/time-sync.md b/docs/time-sync.md index aeaf6de37..f4c3500fc 100644 --- a/docs/time-sync.md +++ b/docs/time-sync.md @@ -1,4 +1,4 @@ -### Time synchronization +# Time synchronization If you wish to set up time synchronization via [NTP](http://www.ntp.org/) or `htpdate`, open a terminal and run `sudo configure-interfaces.py`. Select **Continue**, then choose **Time Sync**. Here you can configure the operating system to keep its time synchronized with either an NTP server (using the NTP protocol), another Malcolm instance, or another HTTP/HTTPS server. On the next dialog, choose the time synchronization method you wish to configure. diff --git a/docs/ubuntu-install-example.md b/docs/ubuntu-install-example.md index fb8bedbbd..c61cfaac4 100644 --- a/docs/ubuntu-install-example.md +++ b/docs/ubuntu-install-example.md @@ -1,4 +1,4 @@ -## Installation example using Ubuntu 22.04 LTS +# Installation example using Ubuntu 22.04 LTS Here's a step-by-step example of getting [Malcolm from GitHub](https://github.com/idaholab/Malcolm/tree/main), configuring your system and your Malcolm instance, and running it on a system running Ubuntu Linux. Your mileage may vary depending on your individual system configuration, but this should be a good starting point. diff --git a/docs/upload.md b/docs/upload.md index f5c15d9af..a0a4e5fe1 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -1,4 +1,4 @@ -## Capture file and log archive upload +# Capture file and log archive upload * [Capture file and log archive upload](#Upload) - [Tagging](#Tagging) @@ -19,12 +19,12 @@ The types of files supported are: Files uploaded via these methods are monitored and moved automatically to other directories for processing to begin, generally within one minute of completion of the upload. -### Tagging +## Tagging In addition to be processed for uploading, Malcolm events will be tagged according to the components of the filenames of the PCAP files or Zeek log archives files from which the events were parsed. For example, records created from a PCAP file named `ACME_Scada_VLAN10.pcap` would be tagged with `ACME`, `Scada`, and `VLAN10`. Tags are extracted from filenames by splitting on the characters `,` (comma), `-` (dash), and `_` (underscore). These tags are viewable and searchable (via the `tags` field) in Arkime and OpenSearch Dashboards. This behavior can be changed by modifying the `AUTO_TAG` [environment variable in `docker-compose.yml`](malcolm-config.md#DockerComposeYml). Tags may also be specified manually with the [browser-based upload form](#Upload). -### Processing uploaded PCAPs with Zeek and Suricata +## Processing uploaded PCAPs with Zeek and Suricata The **Analyze with Zeek** and **Analyze with Suricata** checkboxes may be used when uploading PCAP files to cause them to be analyzed by Zeek and Suricata, respectively. This is functionally equivalent to the `ZEEK_AUTO_ANALYZE_PCAP_FILES` and `SURICATA_AUTO_ANALYZE_PCAP_FILES` environment variables [described above](malcolm-config.md#DockerComposeYml), only on a per-upload basis. Zeek can also automatically carve out files from file transfers; see [Automatic file extraction and scanning](file-scanning.md#ZeekFileExtraction) for more details. diff --git a/docs/zeek-intel.md b/docs/zeek-intel.md index ac86699d6..f60744dd2 100644 --- a/docs/zeek-intel.md +++ b/docs/zeek-intel.md @@ -1,4 +1,4 @@ -### Zeek Intelligence Framework +# Zeek Intelligence Framework * [Zeek Intelligence Framework](#ZeekIntel) - [STIX™ and TAXII™](#ZeekIntelSTIX) @@ -18,7 +18,7 @@ docker-compose exec --user $(id -u) zeek /usr/local/bin/entrypoint.sh true For a public example of Zeek intelligence files, see Critical Path Security's [repository](https://github.com/CriticalPathSecurity/Zeek-Intelligence-Feeds) which aggregates data from various other threat feeds into Zeek's format. -#### STIX™ and TAXII™ +## STIX™ and TAXII™ In addition to loading Zeek intelligence files, on startup Malcolm will [automatically generate](shared/bin/zeek_intel_from_threat_feed.py) a Zeek intelligence file for all [Structured Threat Information Expression (STIX™)](https://oasis-open.github.io/cti-documentation/stix/intro.html) [v2.0](https://docs.oasis-open.org/cti/stix/v2.0/stix-v2.0-part1-stix-core.html)/[v2.1](https://docs.oasis-open.org/cti/stix/v2.1/stix-v2.1.html) JSON files found under `./zeek/intel/STIX`. @@ -40,7 +40,7 @@ Malcolm will attempt to query the TAXII feed(s) for `indicator` STIX objects and Note that only **indicators** of [**cyber-observable objects**](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr) matched with the **equals (`=`)** [comparison operator](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_t11hn314cr7w) against a **single value** can be expressed as Zeek intelligence items. More complex STIX indicators will be silently ignored. -#### MISP +## MISP In addition to loading Zeek intelligence files, on startup Malcolm will [automatically generate](shared/bin/zeek_intel_from_threat_feed.py) a Zeek intelligence file for all [Malware Information Sharing Platform (MISP)](https://www.misp-project.org/datamodels/) JSON files found under `./zeek/intel/MISP`.