[FR] [DAC] Add exceptions importing from ndjson

[eric-forte-elastic]

Note

This PR is an old PR that is now replaced by: #3869

Issues

#3674

Summary

This PR adds support for exporting exception lists from Kibana that are attached to rules that we export. This is accomplished by adding a small method export_exception_list to our Kibana library's resource.py file containing the definition of a rules resource. At this time we are not supporting managing exceptions/exception lists outside of rules. Meaning, every exception list needs to be tied to a rule. The purpose of this feature is to add support for importing and exporting rules that make use of exception lists to Kibana instances that do not have those lists already available.

This PR is the export part of that functionality.
Note: the prior exception list logic does not support importing to Kibana. This PR contains only the export logic, a separate PR will be needed to support importing into Kibana.

Details

Current output if you attempt to import rules with exceptions list (the exceptions list does not get moved with the rule).

2 rule(s) failed to import!
 - 794d2fc0-ecd0-4963-99da-fd587666b80d: (400) Rule with rule_id: "794d2fc0-ecd0-4963-99da-fd587666b80d" references a non existent exception list of list_id: "1ce8efca-877d-4d4a-a069-6d41aa77e0d9". Reference has been removed.
 - 7c22a9d2-5910-4da2-92af-7ff7481bd0f7: (400) Rule with rule_id: "7c22a9d2-5910-4da2-92af-7ff7481bd0f7" references a non existent exception list of list_id: "ad78032a-6730-44c1-8ec3-129ff1dc2ad9". Reference has been removed.

Additional note: while it may be desirable to also provide support for pulling exception lists when one has a .ndjson export from Kibana, this may be blocked by #3863.

Kibana API Docs

https://www.elastic.co/guide/en/security/current/exceptions-api-get-item.html
https://www.elastic.co/guide/en/security/current/exceptions-api-create-container.html

Testing

To test this code:

1. Create a rule with one or more exceptions in the Kibana API.

Details

2. Next, run the export rules command with the new -e flag to also export the exceptions.
   Example Command: python -m detection_rules kibana export-rules -d custom_rules/rules -s -sv -e
   Note: Depending on how many custom rules you have in your Kibana instance, your numbers may be slightly different.

Expected Output

detection-rules on  3674-frdac-add-exceptions-importing-from-ndjson [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 2s
❯ python -m detection_rules kibana export-rules -d custom_rules/rules -s -sv -e
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

8 rules exported
8 rules converted
2 exceptions exported
8 saved to custom_rules/rules

3. Check to see if your exception toml file exists in the exceptions/ folder in your custom rules directory.

Example Exception File Contents

[[exceptions]]
description = "Exception list item"
id = "8d1c6de2-12bf-442d-9b52-00bc99bfcea2"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
name = "FakeRoot"
os_types = []
namespace_type = "single"
_version = "WzQ3NTY0LDJd"
tags = []
type = "simple"
comments = []
created_at = "2024-07-01T17:50:11.181Z"
created_by = "3610252053"
updated_at = "2024-07-01T17:50:11.181Z"
updated_by = "3610252053"
item_id = "d6a0e21c-bf41-4758-a522-cca5df3a2332"
tie_breaker_id = "e1e84f62-b36f-4608-9778-1e4ca29539ae"
[[exceptions.entries]]
field = "process.name"
operator = "included"
type = "match"
value = "FakeRoot"


[[exceptions]]
description = "Exception list item"
id = "49f9966c-9fb4-4d8a-8bed-8e7bfcdafbc5"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
name = "Pid not One"
os_types = []
namespace_type = "single"
_version = "WzQ3NTY1LDJd"
tags = []
type = "simple"
comments = []
created_at = "2024-07-01T19:35:20.071Z"
created_by = "3610252053"
updated_at = "2024-07-01T19:35:20.071Z"
updated_by = "3610252053"
item_id = "970945dd-71d5-4128-89a8-7e8689326a19"
tie_breaker_id = "db323af7-5564-4a42-8d8e-81f933c5cef1"
[[exceptions.entries]]
field = "Effective_process.pid"
operator = "included"
type = "match"
value = "1"


[metadata]
creation_date = "2024/07/03"
rule_id = "7c22a9d2-5910-4da2-92af-7ff7481bd0f7"
rule_name = "Test Exception List"
updated_date = "2024/07/03"

Example Rule File Toml (in rules directory of custom rules dir) that is attached to the above exception list

[metadata]
creation_date = "2024/07/03"
maturity = "production"
updated_date = "2024/07/03"

[rule]
actions = []
author = ["Elastic"]
description = "Test Exception List"
enabled = true
false_positives = []
filters = []
from = "now-18060s"
index = [
    "apm-*-transaction*",
    "auditbeat-*",
    "endgame-*",
    "filebeat-*",
    "logs-*",
    "packetbeat-*",
    "traces-apm*",
    "winlogbeat-*",
    "-*elastic-cloud-logs-*",
]
interval = "5h"
language = "eql"
license = ""
max_signals = 100
name = "Test Exception List"
references = []
related_integrations = []
required_fields = []
risk_score = 21
risk_score_mapping = []
rule_id = "7c22a9d2-5910-4da2-92af-7ff7481bd0f7"
setup = ""
severity = "low"
severity_mapping = []
tags = []
threat = []
to = "now"
type = "eql"

query = '''
process where true
'''


[[rule.exceptions_list]]
id = "222e1466-6dee-49ed-bb40-b7791891dc90"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
namespace_type = "single"
type = "rule_default"

[rule.meta]
from = "1m"
kibana_siem_app_url = "https://dev-deployment-2c684a.kb.us-central1.gcp.cloud.es.io:9243/app/security"

[eric-forte-elastic]

Note, I expect we may want to have a docs update with this PR.

[Mikaayenson]

👋 @eric-forte-elastic We can setup some time to go over these questions, but here are a few solely based on the PR summary.

1. IINM if you have a custom rule and exception list in the custom exception.toml file, you should be able to link the exception to the rule by rule id. Did you link the two this way before trying to upload the rule/exception combination back to kibana? (again this should work prior to any changes)
2. I dont believe we need to specify the exceptions in the custom rule toml files (not sure if it's a problem either without looking closer). We may not need the rule_default updates if we're not trying to also link this way.
3. Regarding the exceptions schemas, can you confirm that the kibana form and our schemas match. They look slight different from what I thought was correct a while back when it was being implemented. It would be good to double check what it actually should be and update the documentation. To minimize import/export, id make them align as close as possible.

One other note: If we updating the kibana lib dont forget we'll need to bump the version.

[traut]

I would advise against generating TIME_NOW on the module level and using it as a default variable. Often, it's ok to do module-level constants, but the benefit should be significant (usually, we're using it as a simplistic "caching" mechanism). As a rule of thumb with module-level vars - we can't guarantee that the module will always be imported in the way we expected it to be, so we need to be mindful of the lifecycle and the value the var. Especially when the variable is as as dynamic as current datetime (even though we serialize it into a date string).

The cleaner approach would be to make creation_date and updated_date either optional (or maybe even do not allow them to be set from the outside the method), and populate them with the current time inside the function with

creation_date = creation_date or datetime.now(timezone.utc).strftime("%Y/%m/%d")

I'm using datetime.now(timezone.utc) here instead of time to avoid all those corner cases when local non-UTC timezone can mess up the current date. These issues are so so difficult to debug when they happen.

[traut]

should we use click.echo here as well, for consistency?

[traut]

maybe these can be just module-level Kibana path constants?

[eric-forte-elastic]

No longer needed, alternative PR merged. #3870