Malcolm v24.06.0 #318
Merged
Malcolm v24.06.0 #318
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…into v24.02.0_merge_idaholab
…into v24.02.0_merge_idaholab
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates. idaholab/Malcolm@v24.01.0...v24.02.0 * Features and enhancements - [Hedgehog Linux SD card image for Raspberry Pi](https://idaholab.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this) - allow configuration of Arkime's ILM/ISM settings (idaholab#300) - add option for customizing which log types get NetBox enrichment (idaholab#316) - improve the extracted_files download page (idaholab#329) - include missing aggregations in API bucket queries (idaholab#386) - more intelligent .env file checking on startup (idaholab#387) - Malcolm report to itself on capture statistics (idaholab#395) - link to Dashboards/Arkime from NetBox devices view (idaholab#410) - changed default PCAP storage format to zstd(3) for new installations - various documentation updates and improvements - changed back to using official Zeek .deb files rather than building from source to reduce build times * Component version updates - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130) - Capa to [v7.0.1](https://github.com/mandiant/capa/releases) - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases) - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html) - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html) - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1) * Bug fixes - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335) - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403) - allow netbox-restore and netbox-backup to specify container name (idaholab#337) - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch)) - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401) - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402) - basic authentication breaks with special characters (idaholab#404) - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency)) * Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.02.0/config)) * these variables in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300) ``` # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm) # Whether or not Arkime should perform index management INDEX_MANAGEMENT_ENABLED=false # Time in hours/days before moving to warm and force merge (number followed by h or d) INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d # Time in hours/days before deleting index (number followed by h or d) INDEX_MANAGEMENT_RETENTION_TIME=90d # Number of replicas for older sessions indices INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0 # Number of weeks of history to retain INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13 # Number of segments to optimize sessions for INDEX_MANAGEMENT_SEGMENTS=1 # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index) INDEX_MANAGEMENT_HOT_WARM_ENABLED=false ``` * these variables in [`dashboards.env`](https://github.com/idaholab/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend ``` # These values are used to handle the Arkime value actions to pivot from Arkime # to Dashboards. The nginx-proxy container's entrypoint will try to formulate # them automatically, but they may be specified explicitly here. NGINX_DASHBOARDS_PREFIX= NGINX_DASHBOARDS_PROXY_PASS= ``` * these variables in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash ``` # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs) LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird ``` ``` # Zeek log types that will be ignored (dropped) by LogStash LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout ``` * these variables in [`netbox-common.env`](https://github.com/idaholab/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation ``` # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env) NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95 ``` * these variables in [suricata-live.env](https://github.com/idaholab/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395) ``` # Whether or not enable capture statistics and include them in eve.json SURICATA_STATS_ENABLED=false SURICATA_STATS_EVE_ENABLED=false SURICATA_STATS_INTERVAL=30 SURICATA_STATS_DECODER_EVENTS=false ``` ``` # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log ZEEK_DISABLE_STATS=true ``` * this variable in [zeek.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329) ``` # Whether or not to use libmagic to show MIME types for Zeek-extracted files served EXTRACTED_FILE_HTTP_SERVER_MAGIC=false ```
Logout page
…into v24.03.0_merge_idaholab
…into v24.03.0_merge_idaholab
…into v24.03.0_merge_idaholab
…into v24.03.0_merge_idaholab
…into v24.03.0_merge_idaholab
Malcolm v24.03.0
…into v24.03.1_merge_idaholab
…into v24.03.1_merge_idaholab
Malcolm v24.03.1
…into v24.04.0_merge_idaholab
Malcolm v24.04.0
* Features and enhancements
- Zeek-extracted files scanned and preserved on a [Hedgehog Linux](https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#HedgehogZeekFileExtraction) sensor can now be accessed via [the extracted files download user interface](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtractionUI) (idaholab#331).
- Improvements to creation of index templates, dashboards, and other saved objects on startup (idaholab#208) to ensure that saved objects get created correctly upon upgrade (see [this comment](idaholab#208 (comment)) for more details on this feature).
- [Populating the NetBox inventory via passively-gathered network traffic metadata](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassive) now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (idaholab#415). Autopopulated devices now have their *status* field set to `Active` rather than `Stage`, and uses *tags* instead to indicated that they were created through autopopulation.
- Users can now specify pruning thresholds for [carved files](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtraction) so that old files are deleted in order to avoid filling available storage (idaholab#453). See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about this and similar settings.
- Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (idaholab#455).
- The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with [category fields for high cardinality](https://opensearch.org/docs/latest/observing-your-data/ad/index/#optional-set-category-fields-for-high-cardinality) to allow for better breakdown of contributing values to anomalies discovered (idaholab#464).
- Include [JA4+ plugin in Arkime](https://arkime.com/settings#ja4plus). See idaholab#419 for status on upcoming full JA4+ support in Malcolm.
- Hedgehog Linux sensors can now [periodically refresh](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/hedgehog-iso/interface/sensor_ctl/control_vars.conf#L75) their [Zeek inteligence files](https://idaholab.github.io/Malcolm/docs/hedgehog-config-zeek-intel.html#HedgehogZeekIntel).
+ **NOTE**: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line `export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel` to `/opt/sensor/sensor_ctl/control_vars.conf` and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
- Assorted documentation improvements.
* Component version updates
- Arkime to [v5.1.2](https://github.com/arkime/arkime/blob/bcd9d7e68be8e4a52a17c35211c5d5a7fdcc1a1c/CHANGELOG#L36-L41)
- OpenSearch and OpenSearch Dashboards to [v2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md)
- Beats to [v8.13.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.13.2.html)
- Logstash to [v8.13.2](https://www.elastic.co/guide/en/logstash/current/logstash-8-13-2.html)
- gunicorn to v22.0.0 to address [CVE-2024-1135](GHSA-w3h3-4rj7-4ph4).
- elasticsearch-dsl to [v8.13.0](https://github.com/elastic/elasticsearch-dsl-py/releases/tag/v8.13.0)
- elasticsearch-py to [v8.13.0](https://github.com/elastic/elasticsearch-py/releases/tag/v8.13.0)
- idna to v3.7 to address [CVE-2024-3651](GHSA-jjg7-2v4v-x38h)
- Fluent Bit to [v3.0.3](https://fluentbit.io/announcements/v3.0.3/)
* Bug fixes
- The documentation for [Windows host system configuration](https://idaholab.github.io/Malcolm/docs/host-config-windows.html#HostSystemConfigWindows) was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (idaholab#421).
- An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (idaholab#426).
- The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of `zeek-live` containers (idaholab#456). See [this comment](idaholab#456 (comment)) for more details.
- Removed the version top-level element from `docker-compose.yml` files as it is [now obsolete](https://docs.docker.com/compose/compose-file/04-version-and-name/) and caused a warning message that sometimes was not handled correctly.
- Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
- Restart live Zeek instances with `zeekctl deploy` instead of `zeekctl restart`.
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.04.0/config))
- `ARKIME_QUERY_ALL_INDICES` in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/arkime.env.example#L9-L11) can be set to control the [`queryAllIndices` setting](https://arkime.com/settings#queryAllIndices) in Arkime's `config.ini`.
- `DASHBOARDS_PREFIX` in [`dashboards-helper.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/dashboards-helper.env.example#L3C1-L4C19) has been added for idaholab#455 (see above in **Features and Enhancements**).
- `LOGSTASH_NETBOX_ENRICHMENT_DATASETS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L13) has been changed to include `zeek.dhcp`, `zeek.dns`, and `zeek.ntlm` to support idaholab#415 (see above in **Features and Enhancements**).
- `LOGSTASH_ZEEK_IGNORED_LOGS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L15) has been changed to remove `capture_loss` and `stats` so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
- `ZEEK_CRON` has been removed from [`zeek-live.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-live.env.example) and `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` was removed from [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example) and moved to the "offline" version of the container in [`zeek-offline.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-offline.env.example#L17-L19) for idaholab#456.
- `EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE`, `EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT`, and `EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS` were added to [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example#L32-L37) for idaholab#453. See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about these and similar settings.
…into v24.05.0_merge_idaholab
…into v24.05.0_merge_idaholab
Ja4 integration
…into v24.06.0_merge_cisagov
…into v24.06.0_merge_cisagov
…into v24.06.0_merge_cisagov
…into v24.06.0_merge_cisagov
…into v24.06.0_merge_cisagov
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
./config/*.envenvironment variable files in Malcolm v24.06.0