Malcolm v24.02.0
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.
- Features and enhancements
- Hedgehog Linux SD card image for Raspberry Pi (idaholab#250; special thanks to @aut0exec for his work on this)
- allow configuration of Arkime's ILM/ISM settings (idaholab#300)
- add option for customizing which log types get NetBox enrichment (idaholab#316)
- improve the extracted_files download page (idaholab#329)
- include missing aggregations in API bucket queries (idaholab#386)
- more intelligent .env file checking on startup (idaholab#387)
- Malcolm report to itself on capture statistics (idaholab#395)
- link to Dashboards/Arkime from NetBox devices view (idaholab#410)
- changed default PCAP storage format to zstd(3) for new installations
- various documentation updates and improvements
- changed back to using official Zeek .deb files rather than building from source to reduce build times
- Component version updates
- Bug fixes
- pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335)
- redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403)
- allow netbox-restore and netbox-backup to specify container name (idaholab#337)
- fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and updated documentation)
- source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401)
- event.severity_tags is not being assigned correctly based on rule.category (idaholab#402)
- basic authentication breaks with special characters (idaholab#404)
- changed some Logstash Ruby variables from global (
$
) to instance (@
) (see "avoiding concurrency issues")
- Configuration changes (in environment variables in
./config/
)- these variables in
arkime.env
to allow configuration of Arkime's ILM/ISM settings (idaholab#300)
# These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
- these variables in
dashboards.env
to override the values automatically configured for pivot links (idaholab#335) and/dashboard/
redirect (idaholab#403) for Elasticsearch backend
# These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS=
- these variables in
logstash.env
for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
# Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
# Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
- these variables in
netbox-common.env
for adjusting matching device manufacturers to OUIs in NetBox autopopulation
# Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
- these variables in suricata-live.env and zeek-live.env that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395)
# Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false
# Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true
- this variable in zeek.env related to the improvements to the extracted_files download page (idaholab#329)
# Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
- these variables in
Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.