[FR] Add support for investigation_fields

[Mikaayenson]

Issues

Resolves #3135
Resolves #3224

Summary

• Add's support for the new field investigation_fields completed in 8.11 with min_compat set to 8.11
• Original feature was added here in 8.10 but updated later here in 8.11
• Added @yctercero who implemented the feature upstream for review
• Didn't add any downgrade functionality since we didn't implement the breaking change included in 8.10
• Note: I did not add validation on the field values. I suspect this will be expensive to do in the data class validation. We could potentially add a unit test that loads all the schemas, but we may run into the same issue here.

Testing

• Unit test should all pass
• Made sure imports/exports of the new field worked.
• Tested the test-cli makefile command make_test_cli.txt
• This file was exported from Kibana and tested with the importer: rules_export-16.ndjson.txt

Import Testing

Sample NDJSON from TOML

{"actions": [{"action_type_id": ".email", "frequency": {"notifyWhen": "onActiveAlert", "summary": true}, "group": "default", "id": "elastic-cloud-email", "params": {"message": "Rule {{context.rule.name}} generated {{state.signals_count}} alerts. my double space newline here ### My header newline More stuff here.", "subject": "my test subject", "to": ["[email protected]"]}, "uuid": "6a2023af-5854-4dae-86dd-4c26b78315cb"}], "author": ["me"], "data_view_id": "logs-*", "description": "ere add aims aewerdas", "filters": [{"$state": {"store": "appState"}, "meta": {"disabled": false, "field": "Effective_process.pid", "index": "logs-*", "key": "Effective_process.pid", "negate": false, "params": {"query": "0"}, "type": "phrase"}, "query": {"match_phrase": {"Effective_process.pid": "0"}}}], "from": "now-360s", "interval": "5m", "investigation_fields": {"field_names": ["Effective_process.pid", "Responses.action.key.values.actions", "Target.process.Ext.dll.code_signature.subject_name"]}, "language": "eql", "max_signals": 100, "meta": {"from": "1m", "kibana_siem_app_url": ""}, "name": "test investigation_fields", "query": "process where true\n", "risk_score": 21, "rule_id": "8f6eb3b6-e9f2-4c10-a72b-cf48b4e90c2e", "severity": "low", "to": "now", "type": "eql", "version": 1}

Sample TOML

I added additional metadata needed to pass unit tests.

[metadata]
creation_date = "2024/03/29"
maturity = "production"
updated_date = "2024/03/29"
min_stack_comments = "New fields added: investigation_fields"
min_stack_version = "8.11.0"

[rule]
author = ["Elastic"]
data_view_id = "logs-*"
description = "ere add aims aewerdas"
from = "now-360s"
interval = "5m"
language = "eql"
license = "Elastic License v2"
max_signals = 100
name = "test investigation_fields"
risk_score = 21
rule_id = "8f6eb3b6-e9f2-4c10-a72b-cf48b4eeec2e"
severity = "low"
tags = ["Domain: Endpoint", "OS: Windows", "Use Case: Threat Detection", "Tactic: Privilege Escalation"]
to = "now"
type = "eql"
timestamp_override = "event.ingested"
query = '''
process where true and host.os.type: "windows"
'''

[rule.investigation_fields]
field_names = [
    "Effective_process.pid",
    "Responses.action.key.values.actions",
    "Target.process.Ext.dll.code_signature.subject_name",
]

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1134"
name = "Access Token Manipulation"
reference = "https://attack.mitre.org/techniques/T1134/"


[rule.threat.tactic]
id = "TA0004"
name = "Privilege Escalation"
reference = "https://attack.mitre.org/tactics/TA0004/"

[brokensound77]

LGTM

[yctercero]

LGTM and confirming that no backwards compatibility logic is needed if it's being added post the breaking changes.

[eric-forte-elastic]

🟢 Manual review and testing, looks good to me! 👍

Test results