[Security Solution] Unable to Save Edited ESQL Rule Due to Query Validation Error

[pborgonovi]

Description:

When editing the prebuilt rule “AWS IAM User Created Access Keys For Another User” (type: ESQL) and attempting to save changes, the system throws a validation error for the query field. The error states:
“Queries that don’t use the STATS…BY function (non-aggregating queries) must include the ‘metadata _id, _version, _index’ operator after the source command. For example: FROM logs metadata _id, _version, _index.”*
This prevents the rule from being saved even if no changes are made to the query.

Kibana/Elasticsearch Stack version:

8.x

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Prebuilt Rules Customization

Pre requisites:

1. prebuiltRulesCustomizationEnabled flag is enabled
2. Prebuilt rules are installed
3. "AWS IAM User Created Access Keys For Another User" rule is available

Steps to reproduce:

1. Navigate to the Rules Management page.
2. Locate the prebuilt rule “AWS IAM User Created Access Keys For Another User” (type: ESQL).
3. Click Edit Rule Settings.
4. Make any change to the rule settings (or none at all).
5. Click Save.

Current behavior:

• The system throws a query validation error:
  “Queries that don’t use the STATS…BY function (non-aggregating queries) must include the ‘metadata _id, _version, _index’ operator after the source command. For example: FROM logs metadata _id, _version, _index.”*
• Saving the rule fails, and the rule remains in the editing state.

Expected behavior:

• The rule should be saved successfully if the query is unaltered and valid

Evidences:

Screen.Recording.2024-12-09.at.2.25.02.PM.mov

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[maximpn]

Hi @pborgonovi,

What rule version do you observe this version on?

I tried to install AWS IAM User Created Access Keys For Another User rule locally from 8.17.1-beta.1 security_detection_engine package with version 4 and don't observe this problem. Though validation complains on the last comma which is another problem.

You may notice ES|QL query has metadata_id. On the video you attached I don't see neither metadata _id nor stats ... by. Could you attach an ES|QL query to the ticket description as well? There is a screenshot from the video you attached

[maximpn]

@banderror It seems there are prebuilt rules with queries not passing form validation. I got another form validation error OOTB for AWS IAM User Created Access Keys For Another User. It complains on the last comma

And it affects the rule execution. Rule fails all the time (at first locally it complains on missing index but when it's there the rules fails with ES|QL parsing error)

[shashank-elastic]

The trailing comma is removed via - elastic/detection-rules#4292

We are rebuilding the release versions with this fix

[banderror]

@pborgonovi The rule has been fixed in the detection-rules repo and released in the latest v8.17.1 package. Could you please validate?

[pborgonovi]

Validated the rule version 5 with latest 8.17.1 package and it looks good:

Screen.Recording.2024-12-13.at.11.52.49.AM.mov