[FEATURE] Custom Rule field mappings should be provided while configuring detector

[jinalp-crest]

Is your feature request related to a problem?
Custom Rule Mappings are not provided out-of-the-box while configuring field mappings for the detectors.

While creating detectors, after the selection of log type for a particular detector i.e. Cloudtrail, and configuring all Sigma rules, we are provided with the field mappings screen. In field mappings, on the left-hand side, it gives all fields as per the chosen log category, and on the right-hand side, it gives a list of fields in the ingested index to choose from. But on the selection of custom rules, it does not recognize its fields for the field mappings and hence, we are not able to map custom rule fields properly.

Also, due to ongoing issues of "." to "_" conversion #375 in custom rule queries, it does not even matches the exact field that we use for rule specification.

What solution would you like?
Based on the Sigma or custom rules selection, we should be prompted with necessary fields as per the chosen log category, which allows us to map our ingested fields correctly.

What alternatives have you considered?
As fields were not getting mapped automatically, we tried to put alias mappings to the respective index manually. That will map the necessary fields while running detector queries on the index.

Custom rule specification

Custom rule detector query

API Request:

API Response:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.605183,
    "hits": [
      {
        "_index": ".opensearch-sap-custom-rules-config",
        "_id": "GUYlVocB-q1Aw1948wE9",
        "_score": 1.605183,
        "_source": {
          "rule": {
            "category": "cloudtrail",
            "title": "custom-assumerole",
            "log_source": "cloudtrail",
            "description": "custom-assumerole",
            "references": [
              {
                "value": ""
              }
            ],
            "tags": [],
            "level": "low",
            "false_positives": [
              {
                "value": ""
              }
            ],
            "author": "jinalp",
            "status": "experimental",
            "last_update_time": "2023-04-06T00:00:00.000Z",
            "queries": [
              {
                "value": """(user_type: "AssumedRole") AND (user_sessionContext_sessionIssuer_type: "Role")"""
              }
            ],
            "query_field_names": [
              {
                "value": "user_sessionContext_sessionIssuer_type"
              },
              {
                "value": "user_type"
              }
            ],
            "aggregationQueries": [],
            "rule": """id: 25b9c01c-350d-4b95-bed1-836d04a4f324
title: custom-assumerole
description: custom-assumerole
status: experimental
author: jinalp
date: 2023/04/06
modified: 2023/04/06
logsource:
  category: cloudtrail
level: low
detection:
  selection:
    user.type: AssumedRole
    user.sessionContext.sessionIssuer.type: Role
  condition: selection
references:
  - ''
falsepositives:
  - ''
"""
          }
        }
      }
    ]
  }
}

Field alias mapping specification on index

After manually mapping the fields, we were able to generate the relevant findings.

[praveensameneni]

Looking into the issue

[amsiglan]

This has been fixed for 2.9 and beyond. We will work on backporting the fixes to earlier versions