Malcolm v24.01.0

Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.01.0

• Features and enhancements
  • new Malcolm instance landing page (idaholab#252)
  • file carve download with password-protected .zip file (idaholab#288)
  • new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (idaholab#290)
  • allow customizing indexes for logs written to OpenSearch/Elasticsearch (idaholab#313)
  • more consistently differentiate between uploaded and live-captured traffic (idaholab#321)
  • make download extracted file context item from Arkime smarter (idaholab#330)
  • improve netbox device type library import by using "official" import script (idaholab#384)
• Component version updates
  • Alpine Linux to v3.19 as the base for some Docker images
  • Fluent Bit to v2.2.2
  • Beats to v8.11.4
  • LogStash to v8.11.4
• Bug fixes
  • Suricata Alerts dashboard "Alerts - Tags" visualization is useless (idaholab#314)
  • third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (idaholab#318)
  • update document lookup APIs to search either network or host data (idaholab#322)
  • suricata rule update is broken (idaholab#323)
  • time sync from hedgehog to Malcolm opensearch instance not working (idaholab#324)
  • fix issue specifying database mode via command-line
  • have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
• Configuration changes (in environment variables in ./config/)
  • added the following variables with relation to idaholab#313
    • added ARKIME_ROTATE_INDEX to arkime.env with default value of daily (see Arkime docs on rotateIndex)
    • added the following variables and defaults to opensearch.env:

    # OpenSearch index patterns and timestamp fields
    # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
    MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
    # Default time field to use for network traffic logs in Logstash and Dashboards
    MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
    # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
    MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
    # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
    MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
    # Default time field to use for other logs in Logstash and Dashboards
    MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
    # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
    MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
    # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
    ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
    # Default time field used by for sessions in Arkime viewer
    ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
    

  • changed default for EXTRACTED_FILE_HTTP_SERVER_KEY to infected in zeek-secret.env
  • added EXTRACTED_FILE_HTTP_SERVER_ZIP with default value of false in zeek.env, see (idaholab#288)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.