Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

event.severity_tags is not being assigned correctly based on rule.category #402

Closed
mmguero opened this issue Feb 5, 2024 · 0 comments
Closed

event.severity_tags is not being assigned correctly based on rule.category #402

mmguero opened this issue Feb 5, 2024 · 0 comments
Assignees
@mmguero
Labels
bug Something isn't working logstash Relating to Malcolm's use of Logstash regression It worked at one point...
Milestone
v24.02.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Feb 5, 2024

The event.severity_tags field was not being assigned correctly for notice.log documents.

There were a few issues here:

  1. at this point in the filters, zeek.notice.category has already been renamed to rule.category
  2. rule.category is an array, not a scalar, so the == comparison didn't work
@mmguero mmguero added bug Something isn't working logstash Relating to Malcolm's use of Logstash regression It worked at one point... labels Feb 5, 2024
@mmguero mmguero added this to the v24.02.0 milestone Feb 5, 2024
@mmguero mmguero self-assigned this Feb 5, 2024
@mmguero mmguero added this to Malcolm Feb 5, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 5, 2024
@mmguero
fix bug idaholab#402, event.severity_tags is not being assigned corre…
dfa6409 
…ctly based on rule.category
@mmguero mmguero moved this to Done in Malcolm Feb 5, 2024
@mmguero mmguero closed this as completed Feb 5, 2024
This was referenced Feb 14, 2024
Merged
Merged
mmguero added a commit that referenced this issue Feb 15, 2024
@mmguero
Merge pull request #412 from idaholab/v24.02.0_merge_idaholab
a885008 
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

* Features and enhancements
    - [Hedgehog Linux SD card image for Raspberry Pi](https://idaholab.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (#250; special thanks to @aut0exec for his work on this)
    - allow configuration of Arkime's ILM/ISM settings (#300)
    - add option for customizing which log types get NetBox enrichment (#316)
    - improve the extracted_files download page (#329)
    - include missing aggregations in API bucket queries (#386)
    - more intelligent .env file checking on startup (#387)
    - Malcolm report to itself on capture statistics (#395)
    - link to Dashboards/Arkime from NetBox devices view (#410)
    - changed default PCAP storage format to zstd(3) for new installations
    - various documentation updates and improvements
    - changed back to using official Zeek .deb files rather than building from source to reduce build times
* Component version updates
    - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130)
    - Capa to [v7.0.1](https://github.com/mandiant/capa/releases)
    - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases)
    - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html)
    - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html)
    - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1)
* Bug fixes
    - pivot links from Arkime to Kibana in external elasticsearch are not working (#335)
    - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (#403)
    - allow netbox-restore and netbox-backup to specify container name (#337)
    - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (#393) (and [updated documentation](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch))
    - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (#401)
    - event.severity_tags is not being assigned correctly based on rule.category (#402)
    - basic authentication breaks with special characters (#404)
    - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency))
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.02.0/config))
    * these variables in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (#300)
    ```
    # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
    # Whether or not Arkime should perform index management
    INDEX_MANAGEMENT_ENABLED=false
    # Time in hours/days before moving to warm and force merge (number followed by h or d)
    INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
    # Time in hours/days before deleting index (number followed by h or d)
    INDEX_MANAGEMENT_RETENTION_TIME=90d
    # Number of replicas for older sessions indices
    INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
    # Number of weeks of history to retain
    INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
    # Number of segments to optimize sessions for
    INDEX_MANAGEMENT_SEGMENTS=1
    # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
    INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
    ```
    * these variables in [`dashboards.env`](https://github.com/idaholab/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (#335) and `/dashboard/` redirect (#403) for Elasticsearch backend
    ```
    # These values are used to handle the Arkime value actions to pivot from Arkime
    #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
    #   them automatically, but they may be specified explicitly here.
    NGINX_DASHBOARDS_PREFIX=
    NGINX_DASHBOARDS_PROXY_PASS=
    ```
    * these variables in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
    ```
    # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
    LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
    ```
    ```
    # Zeek log types that will be ignored (dropped) by LogStash
    LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
    ```
    * these variables in [`netbox-common.env`](https://github.com/idaholab/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation
    ```
    # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
    NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
    NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
    ```
    * these variables in [suricata-live.env](https://github.com/idaholab/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (#395)
    ```
    # Whether or not enable capture statistics and include them in eve.json
    SURICATA_STATS_ENABLED=false
    SURICATA_STATS_EVE_ENABLED=false
    SURICATA_STATS_INTERVAL=30
    SURICATA_STATS_DECODER_EVENTS=false
    ```
    ```
    # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
    ZEEK_DISABLE_STATS=true
    ```
    * this variable in [zeek.env](https://github.com/idaholab/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (#329)
    ```
    # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
    EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
    ```
mmguero added a commit to cisagov/Malcolm that referenced this issue Feb 15, 2024
@mmguero
Merge pull request #297 from cisagov/v24.02.0_merge_cisagov
98ac2fb 
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

* Features and enhancements
    - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this)
    - allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    - add option for customizing which log types get NetBox enrichment (idaholab#316)
    - improve the extracted_files download page (idaholab#329)
    - include missing aggregations in API bucket queries (idaholab#386)
    - more intelligent .env file checking on startup (idaholab#387)
    - Malcolm report to itself on capture statistics (idaholab#395)
    - link to Dashboards/Arkime from NetBox devices view (idaholab#410)
    - changed default PCAP storage format to zstd(3) for new installations
    - various documentation updates and improvements
    - changed back to using official Zeek .deb files rather than building from source to reduce build times
* Component version updates
    - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130)
    - Capa to [v7.0.1](https://github.com/mandiant/capa/releases)
    - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases)
    - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html)
    - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html)
    - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1)
* Bug fixes
    - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335)
    - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403)
    - allow netbox-restore and netbox-backup to specify container name (idaholab#337)
    - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch))
    - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401)
    - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402)
    - basic authentication breaks with special characters (idaholab#404)
    - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency))
* Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config))
    * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    ```
    # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
    # Whether or not Arkime should perform index management
    INDEX_MANAGEMENT_ENABLED=false
    # Time in hours/days before moving to warm and force merge (number followed by h or d)
    INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
    # Time in hours/days before deleting index (number followed by h or d)
    INDEX_MANAGEMENT_RETENTION_TIME=90d
    # Number of replicas for older sessions indices
    INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
    # Number of weeks of history to retain
    INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
    # Number of segments to optimize sessions for
    INDEX_MANAGEMENT_SEGMENTS=1
    # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
    INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
    ```
    * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend
    ```
    # These values are used to handle the Arkime value actions to pivot from Arkime
    #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
    #   them automatically, but they may be specified explicitly here.
    NGINX_DASHBOARDS_PREFIX=
    NGINX_DASHBOARDS_PROXY_PASS=
    ```
    * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
    ```
    # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
    LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
    ```
    ```
    # Zeek log types that will be ignored (dropped) by LogStash
    LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
    ```
    * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation
    ```
    # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
    NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
    NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
    ```
    * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395)
    ```
    # Whether or not enable capture statistics and include them in eve.json
    SURICATA_STATS_ENABLED=false
    SURICATA_STATS_EVE_ENABLED=false
    SURICATA_STATS_INTERVAL=30
    SURICATA_STATS_DECODER_EVENTS=false
    ```
    ```
    # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
    ZEEK_DISABLE_STATS=true
    ```
    * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329)
    ```
    # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
    EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
    ```
@mmguero mmguero moved this from Done to Released in Malcolm Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
bug Something isn't working logstash Relating to Malcolm's use of Logstash regression It worked at one point...
Projects
Malcolm
Status: Released
Milestone
v24.02.0
Development

No branches or pull requests
1 participant
@mmguero