Malcolm v23.01.0

Malcolm v23.01.0 is a feature release with new features and enhancements, component version updates and bug fixes.

v6.4.3...v23.01.0

• New features

  • Enrich network traffic metadata via NetBox lookups (idaholab/Malcolm#132)

• Enhancements

  • Switched from semantic versioning (semver) to calendar versioning (calver) (idaholab/Malcolm#139)
  • Added bartblaze/Yara-rules as a YARA rule source
  • Support new fields in EtherNet/IP / CIP parser

• Component version updates

  • OpenSearch and OpenSearch Dashboards v2.4.1
  • Beats to v8.5.3
  • NetBox to v3.4.2
  • docker-compose on ISO now uses the compose plugin

• Fixes

  • when using custom locations, pcap/upload and pcap/processed directories don't get created correctly after wipe (idaholab/Malcolm#140)
  • one Malcolm instance forwarding to another secondary tier Malcolm instance continually imports opensearch index templates (idaholab/Malcolm#142)
  • Updated source code copyright dates from 2022 to 2023

• Deprecated

  • Removed host-map.txt and cidr-map.txt for host and subnet name assignment (use net-map.json file or NetBox now)
  • MAC address to host name mapping for host and subnet name assignment (MAC address is too inconsistent to use as an identifier for a host as network captures may not show the actual MAC address for a given host's communication)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v6.4.3

Malcolm v6.4.3 is a minor release containing enhancements, component version updates and bug fixes.

v6.4.2...v6.4.3

• Enhancements

  • Import the NetBox Device Type Library on NetBox first run to populate manufacturers, device types, models and modules
  • idaholab/Malcolm#127 have install.py --configure ask about other storage locations for PCAP, Zeek logs and OpenSearch indices
  • idaholab/Malcolm#128 have install.py --configure prompt for Arkime to manage uploaded PCAP files or not

• Component version updates

  • Alpine Linux to v3.17 for some Docker containers' base images
  • Filebeat to v8.5.2
  • NetBox to v3.3.9
  • Zeek to v5.0.4
  • opensearch-py to v2.0.1
  • Fluent Bit to v2.0.6

• Fixes

  • Fix some bad links in the documentation and other minor documentation improvements
  • Fix idaholab/Malcolm#126, suricata logs show up in Arkime as "notip" for the protocol
  • Fix idaholab/Malcolm#129, filtering by rootId in Arkime returns no results
  • Fix Docker health checks for NetBox and supporting containers
  • Fix "read-only" version of nginx.conf
  • Tweaks to install.py memory recommendations

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v6.4.2

Malcolm v6.4.2 is a minor release containing a few component version updates (some addressing component vulnerabilities) and other improvements.

v6.4.1...v6.4.2

• Component version updates
  • Zeek to v5.0.3 (this release fixes several security vulnerabilities in Zeek itself)
  • OpenSearch and OpenSearch Dashboards to v2.4.0
  • Logstash to v8.4.0
  • FileBeat to v8.5.1
  • NetBox to v3.3.8
• Bug Fixes
  • Fix unhandled exceptions in API when certain API calls are made before data is indexed
• Improvements
  • Added Zeek plugin to detect vulnerability to and exploitation attempts of CVE-2022-3602
  • Minor documentation fixes
  • Minor improvements to Docker container debug logging
  • Implemented caching of entropy calculations for DNS requests and TLS hostnames

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v6.4.1

Malcolm v6.4.1 is a minor release containing a few bug fixes, component version updates and other improvements.

v6.4.0...v6.4.1

• Bug fixes
  • Zeek log files that have been renamed and are in the process of moving not caught correctly by Logstash (idaholab#121)
  • Hedgehog Arkime viewer node should use TLS (idaholab#122)
  • Recent changes to Elastic Common Schema needed adjustment (map number data type to long)
• Component version updates
  • Arkime v4.0.2
  • NetBox v3.3.7
• Improvements
  • On Hedgehog Linux, allow configuration of Arkime capture to use PCAP compression if desired
  • Changes to GitHub Docker image and ISO workflows, updating deprecated actions and features
  • Create corresponding net-map.json/Host and Subnet Name Mapping items in NetBox on when applicable
  • Remove unnecessary linux-headers- package from Zeek Docker image

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v6.4.0

Malcolm v6.4.0 features refactored documentation, the initial integration of NetBox (a network infrastructure resource modeling tool), several component version updates and other improvements and bug fixes.

Note that some changes involved in this release require some modifications to files used by docker-compose. Please run ./scripts/auth_setup and ./scripts/install.py --configure to ensure the appropriate new environment variables are set.

v6.3.0...v6.4.0

• New features
  • initial NetBox integration (development ongoing, see idaholab#17)
• Improvements
  • Documentation reformat/refactor
  • Use tini for Docker image init
  • Added support for s7comm_upload_download.log
  • Surface more options in install.py --configure, as well as minor tweaks
  • Update documentation report for results of ISO hardening
• Component version updates
  • Arkime v4.0.1
    • Allow (optional) PCAP compression on Hedgehog
  • OpenSearch and OpenSearch Dashboards v2.3.0
  • Fluent Bit v1.9.9
  • Zeek v5.0.2
• Bug fixes
  • verify capa signature hits are still being parsed/inserted correctly (idaholab#120)
  • Handle long integers in parsing bacnet_discovery and bacnet_property
  • Better enrichment of network.direction based on source and destination IP addresses

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v6.3.0

Malcolm v6.3.0 is a feature release with a number of new features, bug fixes and improvements. Of particular note is Malcolm's ability to now use another OpenSearch instance or cluster in lieu of its own local instance.

Note that the changes involved in idaholab#10 require modifications to files used by docker-compose. Please run ./scripts/auth_setup and ./scripts/install.py --configure to ensure the appropriate new environment variables are set.

v6.2.0...v6.3.0

• New Features

  • Support remote OpenSearch instance/cluster as alternative to local containerized instance (idaholab#10)
  • Documentation and convenience scripts for configuring ingestion of third-party logs, and basic parsing/normalizing of Fluent-Bit's Windows event logs
  • S7comm Plus support and replaced Amazon S7comm parser with icsnpp-s7comm (idaholab#99)

• Version Bumps

  • OpenSearch and OpenSearch Dashboards to v2.2.1
  • Zeek to v5.0.1
  • Spicy to v1.5.1
  • spicy-plugin to v1.3.17
  • YARA to v4.2.3
  • Capa to v4.0.1

• Improvements

  • Major improvements to OPC UA Binary parser and supporting dashboards
  • Ensure that all containers are provided the same information about trusted CA certificates
  • changed list of "sensitive countries" to match U.S. Department of Energy Sensitive Country List
  • Increased maximum fields from 3,000 to 5,000
  • Standardized configuration and authentication for primary and secondary remote OpenSearch instances, and make sure that index templates are created on secondary remote OpenSearch instances
  • Expand and fix normalization of network.direction in lieu of using tags
  • Various tweaks and improvements to the install.py script for enabling/disabling some features

• Bugs Fixed

  • fields could be missing in Arkime due to a large number of concurrent requests (idaholab#115)
  • mapper_parsing_exception, TCP flag parsing problem (#214)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v6.2.0

Malcolm v6.2.0 is a feature release with a number of bug fixes and improvements. Of particular note is a major reworking of how a standalone instance of Malcolm (i.e., when not receiving traffic from a network sensor) analyzes "live" traffic. See the README for more information.

Note that the changes around idaholab#109 and idaholab#110 require changes to the files used by docker-compose. Please run ./scripts/auth_setup and ./scripts/install.py --configure to ensure the appropriate new environment variables are set.

v6.1.0...v6.2.0

• Improvements

  • idaholab#109: break Zeek/Suricata into two containers: one for "live" capture and one for uploaded PCAPs
    • give option to disable capture interface hardware offloading and adjust ring buffer sizes for standalone Malcolm capture
    • Zeek and Suricata images are now configured to not drop privileges at init (in order to be able to set permissions for network capture), but supervisord will drop privileges for all of its child processes before they execute to maintain security posture
    • include headers needed to build Zeek af_packet plugin in Zeek docker container
    • updated README to describe methods for capturing local traffic with standalone Malcolm
    • same images will be used for zeek and zeek-live containers, as well as for suricata and suricata-live containers, respectively
    • use the same scripts zeekdeploy.sh to configure and run Zeek on both Hedgehog and in the Malcolm zeek docker images
    • prevent "live" and "non-live" Zeek containers from both trying to update intel indicators at the same time
  • Speed up build time by getting official Debian suricata packages from backports rather than building from source
  • Added Suricata rule update cron jobs
  • Added documentation (in the form of comments) to all docker-compose file variables

• Bugs

  • Fix idaholab#107: expand action/result meaning in DNP3 (and other?) dashboards
    • Clean up some Nul values that could appear in Zeek logs
    • improve mapping of BACnet actions
  • Fix idaholab#108: export PCAP not working from Arkime sessions without "Arkime Sessions"
  • Fix idaholab#110: SFTP upload broken due to dollar sign(s) in openssl-encrypted password
    • prompt in install.py --configure whether or not to expose this port to external hosts
  • Fix an issue that could prevent some zeek logs from being parsed correctly if they contained non-ASCII charactters

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v6.1.0

Malcolm v6.1.0 is a feature release with a number of updates and improvements.

v6.0.1...v6.1.0

• Bugs fixed

  • Zeek logs get reingested after container restart - (idaholab#101)
  • Added IPsec fields that were not being parsed
  • Fixed some dashboards that should have been using ECS field names
  • Split the STUN attribute type field on comma during stun.log parsing

• Improvements

  • Malcolm's OpenSearch index template is now composed upon initialization with elements from the latest Elastic Common Schema release.
  • Replaced most instances of beats on Hedgehog Linux (with the exception of the Apache-licensed 7.10.2 filebeat which is compatible with OpenSearch) with Fluent Bit (see idaholab#102) for resource utilization monitoring, etc. and recreated dashboards referencing these metrics
  • Replaced Auditbeat file integrity checking module with AIDE for Hedgehog Linux
  • Added an optionally exposed (disabled by default) a TCP input endpoint to Malcolm to allow easier ingestion of other third-party logs not natively supported by Malcolm
  • Improvements to APIs for listing fields and indices
  • Removed old environment variable-configured Index State Management code as the new OpenSearch v2.1.0 release has nice UIs for both index state management and snapshot management

• Version bumps of note

  • Supercronic to v0.2.1
  • OpenSearch and OpenSearch Dashboards to v2.1.0 (incorporating changes from v2.0.0, v2.0.1 and v2.1.0)
  • Zeek to v5.0.0 with built-in Spicy and Spicy Zeek plugin
  • YARA to v4.2.2

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v6.0.1

Malcolm v6.0.1 is a minor release updating some of Malcolm's core components and adding a couple of Zeek plugins for detecting recent CVEs.

v6.0.0...v6.0.1

• Added Zeek plugins

  • Corelight's DCE/RPC remote code execution vulnerability (CVE-2022-26809) plugin
  • Corelight's VMware Workspace ONE Access and Identity Manager RCE vulnerability (CVE-2022-22954) plugin

• Bugs fixed

  • Fixed an issue where user-supplied trusted CA certificates might not be added to the OpenSearch container's trust store

• Version bumps

  • OpenSearch to v1.3.2
  • OpenSearch Dashboards to v1.3.2
  • Alpine base Docker image to v3.16
  • Docker Compose to v2.5.0 (as installed by install.py and in the Malcolm ISO installer)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v6.0.0

Malcolm v6.0.0 is a major release which incorporates Suricata as a data source for network traffic analysis in Malcolm alongside Zeek and Arkime. A team at BYU (@piercema, @aglad-eng, @Jarscott1, @n8hacks) recently completed their work on Suricata integration for their capstone project. This release includes their changes as well as some additional work by Malcolm's developer in integrating Suricata in other ways not covered in the scope of their project. This release also includes other bug fixes and improvements.

v5.2.11...v6.0.0

As the Malcolm project uses semantic versioning when choosing version numbers. This release required some pretty extensive remapping of Zeek fields in order for Zeek and Suricata to target the same naming conventions for common fields. This backwards-compatibility breaking change is the reason for bumping the major version number from 5 to 6. It is not recommended to attempt an upgrade from a previous release; a fresh install is strongly encouraged.

• Features

  • Incorporate Suricata as a data source for network traffic analysis in both Malcolm and Hedgehog Linux
  • Added support for the GENISYS protocol

• Improvements

  • Minor tweaks to the GitHub workflows for building the Malcolm installer ISO
  • Better fingerprinting of events during Logstash parsing in order to create a unique but reproducible hash for events in the case that duplicate data is indexed into Malcolm
  • All data sources (Arkime, Zeek and Suricata) now specify the data source (stored as event.provider, arkime, zeek and suricata, respectively) and the log type (stored as event.dataset, e.g., session, conn, alert, etc.) in order to facilitate filtering among various types of network metadata
  • The Malcolm REST API was improved to support POST operations for all of the calls which can accept a filter argument to allow for easier representation of filters as JSON objects
  • Reworked several dashboards, including the Overview, Security Overview, Zeek Notices and Signatures dashboards
  • Leave packages in place on the ISO-installed Malcolm and Hedgehog Linux environments in order to support mounting SMB shares from the Thunar GUI

• Bug fixes

  • Fix idaholab#94: docker-compose | "function" has no attribute "get" (ubuntu 20.04 install)
  • Fix idaholab#96: DNP3 dashboard has invalid saved search syntax
  • Fix idaholab#97: virustotal file scanning broken (AttributeError: 'Namespace' object has no attribute 'vtotReqLimit')
  • Fix idaholab#98: BSAP RDB data parsed incorrectly

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.