FR] Add Core Support for ES|QL Rule Type

[Mikaayenson]

Issues

https://github.com/elastic/ia-trade-team/issues/163

Summary

• Adds support for the ES|QL Rule type based on the kibana implementation
• Adds base stub classes to validate esql rules

Testing

1. Unit tests should pass with the sample esql rule locally.
2. Manual testing with CLI commands.

Sample Rule

Use the following rule, and run view-rule command to ensure the esql rule loads successfully, skipping validation.

[metadata]
creation_date = "2023/02/27"
integration = ["endpoint"]
maturity = "production"
min_stack_comments = "New fields added: required_fields, related_integrations, setup"
min_stack_version = "8.11.0"
updated_date = "2023/06/22"

[rule]
author = ["Elastic"]
description = """
Identifies the execution of the unshadow utility which is part of John the Ripper,
a password-cracking tool on the host machine. Malicious actors can use the utility to retrieve
the combined contents of the '/etc/shadow' and '/etc/password' files.
Using the combined file generated from the utility, the malicious threat actors can use them as input
for password-cracking utilities or prepare themselves for future operations by gathering
credential information of the victim.
"""
from = "now-9m"
language = "esql"
license = "Elastic License v2"
name = "ESQL Potential Linux Credential Dumping via Unshadow"
references = [
    "https://www.cyberciti.biz/faq/unix-linux-password-cracking-john-the-ripper/",
]
risk_score = 47
rule_id = "77af47d0-5bd4-11ee-8f6d-f661ea17fbce"
severity = "medium"
tags = ["Data Source: Elastic Endgame", "Domain: Endpoint", "OS: Linux", "Use Case: Threat Detection", "Tactic: Credential Access", "Data Source: Elastic Defend"]
timestamp_override = "event.ingested"
type = "esql"
query = '''
from logs-endpoint.events.*, logs-endpoint.events.process*
    | keep host.os.type, process.name, process.working_directory, event.type, event.action
    | where host.os.type == "linux" and process.name == "unshadow" and event.type == "start" and event.action in ("exec", "exec_event")
'''


[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1003"
name = "OS Credential Dumping"
reference = "https://attack.mitre.org/techniques/T1003/"
[[rule.threat.technique.subtechnique]]
id = "T1003.008"
name = "/etc/passwd and /etc/shadow"
reference = "https://attack.mitre.org/techniques/T1003/008/"



[rule.threat.tactic]
id = "TA0006"
name = "Credential Access"
reference = "https://attack.mitre.org/tactics/TA0006/"

CLI Testing Commands

python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection
-rules/rules/linux/credential_access_credential_dumping_via_unshadow.toml --api-format 

Or run:

python -m detection_rules validate-rule /Users/stryker/workspace/Elastic/detection
-rules/rules/linux/credential_access_credential_dumping_via_unshadow.toml

Output should look similar to the following:

Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "Identifies the execution of the unshadow utility which is part of John the Ripper, a password-cracking tool on the host machine. Malicious actors can use the utility to retrieve the combined contents of the '/etc/shadow' and '/etc/password' files. Using the combined file generated from the utility, the malicious threat actors can use them as input for password-cracking utilities or prepare themselves for future operations by gathering credential information of the victim.",
  "from": "now-9m",
  "language": "esql",
  "license": "Elastic License v2",
  "name": "ESQL Potential Linux Credential Dumping via Unshadow",
  "query": "from logs-endpoint.events.*, logs-endpoint.events.process*\n    | keep host.os.type, process.name, process.working_directory, event.type, event.action\n    | where host.os.type == \"linux\" and process.name == \"unshadow\" and event.type == \"start\" and event.action in (\"exec\", \"exec_event\")\n",
  "references": [
    "https://www.cyberciti.biz/faq/unix-linux-password-cracking-john-the-ripper/"
  ],
  "risk_score": 47,
  "rule_id": "77af47d0-5bd4-11ee-8f6d-f661ea17fbce",
  "severity": "medium",
  "tags": [
    "Data Source: Elastic Endgame",
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Credential Access",
    "Data Source: Elastic Defend"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0006",
        "name": "Credential Access",
        "reference": "https://attack.mitre.org/tactics/TA0006/"
      },
      "technique": [
        {
          "id": "T1003",
          "name": "OS Credential Dumping",
          "reference": "https://attack.mitre.org/techniques/T1003/",
          "subtechnique": [
            {
              "id": "T1003.008",
              "name": "/etc/passwd and /etc/shadow",
              "reference": "https://attack.mitre.org/techniques/T1003/008/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "esql",
  "version": 1
}

Additional Information

• ESQL query validation is not available with this PR. A warning is provided to the user notifying them for each ESQL rule.
• In the interim, users can read our Blog for more details on ES|QL validation.

[terrancedejesus]

LGTM for base ES|QL rule support.

[eric-forte-elastic]

Suggested change

	if data.type != "esql":
	if getattr(data, 'type', '') != "esql":

Maybe we should use .get in the event that the type is not set? It looks like we do that in our other checks here.