Adding additional fields to ElastAlert documents.

[jeff-bb]

Given this sample yaml file below, I'm curious how to get various key/values/arrays into the elastalert as new fields.

For example, name maps to rule_name description maps to description, elastic query results map to match_body... but what about fields defined in the yaml that I'd like to push forward into the alert. For example, status, severity, references, and mitre?

I had the thought of modifying the elastaery.py self.add_metadata_alert and ruletypes.py init sections but that felt dirty to insert like that.

Is this something I'd leverage a custom rule type for to insert this additional data using add_data? I've done some amount of searching, but I can't find other references to this. I've found some references to using this data in the alert_text, but I want to get it stored with the document.

name: Rulename
status: stable
severity: medium
priority: 3
description: Words
references:
    - https://google.com
mitre:
    - TA0001
        -T1659
    - TA0002
        -T1651
        -T1059

type: any
index: logstash-*

buffer_time:
    days: 7
realert:
    hours: 1

query_key: [host.name]

filter:
    - query:
        query_string:
            query: 'event.code:1'

alert_subject: "host {}"

alert_subject_args:
    - host.name

alert_text: |-
    Event Activity!
    Occurred at: {}
    Host: {}

alert_text_args:
    - "@timestamp"
    - host.name

alert_text_type: alert_text_only

alert:
    - "email"

email:
    - "to@domain"
from_addr: "from@domain"
smtp_host: "1.1.1.1"

[jertel]

I'm not following. If you want to push static data, like you have in the rule above (ex: status = stable) why wouldn't you just type that into the alert_text value. Ex:

alert_text: |-
  Event Activity!
  Occured At: {}
  Host: {}
  Status: Stable

[jeff-bb]

That will exist in the alert, in this example the email. It will not exist in the document written to elastic. I'm asking how to get additional values, from the yaml file, into elastic. Similar to how name becomes rule_name or description becomes description.

[jertel]

I see. You can add an enhancement to grab those params out of the rule dict, and push them into the match dictionary. That match dictionary is what lands inside the match_body.* fields, stored in the elastalert index.

Ex:

My Rule:

type: frequency

foo: bar

index: logs-*
name: foo

num_events: 10
timeframe:
  minutes: 5

filter:
- query_string:
    analyze_wildcard: true
    query: 'level: (ERROR OR WARN) AND NOT message: ("test msg" OR "testing rule")'

alert:
  - debug

match_enhancements:
- "mod.test.MyEnhancement"

My Custom Enhancement, placed in a new file mod/test.py

from elastalert.enhancements import BaseEnhancement

class MyEnhancement(BaseEnhancement):
    def process(self, match):
        match["custom_field"] = self.rule["foo"]

Results in a new field called custom_field getting written to the ElastAlert index with my rule's configured value of bar:

[jeff-bb]

Thank you for the assistance. I'd have preferred to put them as part of the 'top level object', and I can see where I would make those changes in code. For now, I'm using the enhancement process to at least get them in at all.