[Security Solutions][Detection Engine] Adds ability to ignore fields during alert indexing and a workaround for an EQL bug

[FrankHassanabad]

Summary

Adds a workaround for EQL bug: elastic/elasticsearch#77152
Adds the safety feature mentioned here: #110802

Adds the ability to ignore particular fields when the field is merged with _source. Also fixes an EQL bug where EQL is introducing the meta field of _ignored within the fields and causing documents to not be indexable when we merge with the fields from EQL.

Alerting document creation uses the fields API to get runtime field, constant keyword, etc... that are only available within the fields API and then merges the field values not found within the _source document with the _source document and then finally indexes this merged document as an alert document.

This fix/ability is a "safety feature" in that if a problematic runtime field, constant keyword is discovered or another bug along the stack we can set a kibana.yml key/value pair to ignore the problematic field.

This WILL NOT remove problematic fields from the _source document. This will only ignore problematic constant keyword, runtime fields, aliases, or anything else found in the fields API that is causing merge issues.

This PR:

• Adds a alertIgnoreFields kibana.yml array key with a default of an empty array if not specified.
• Plumbs the alertIgnoreFields through the stack and into the fields/_source merge strategies of missingFields and allFields
• Adds a temporary isEqlBug77152 where it hard codes an ignore of _ignored until the EQL problem is fixed and then we will remove the workaround
• Adds unit tests
• Adds e2e tests which covers the described use cases above.

The alertIgnoreFields key/value within kibana.yml if set should be an array of strings of each field you want to ignore. This can also contain regular expressions as long as they are of the form, "/regex/" in the array.

Example if you want to ignore fields that are problematic called "host.name" and then one in which you want to ignore all fields that start with "user." using a regular expression:

xpack.securitySolution.alertIgnoreFields: ['host.name', '/user\..*/']

Although there are e2e tests which exercise the use cases...

If you want to manual test the EQL bug fix you would add these documents in dev tools:

# Delete and add a mapping with a small ignore_above.
DELETE eql-issue-ignore-fields-delme
PUT eql-issue-ignore-fields-delme
{
  "mappings" : {
    "dynamic": "strict",
    "properties" : {
      "@timestamp": {
        "type": "date"
      },
      "some_keyword" : {
        "ignore_above": 5, 
        "type" : "keyword"
      },
      "other_keyword" : {
        "ignore_above": 10, 
        "type" : "keyword"
      }
    }
  }
}

# Add a single document with one field that will be truncated and a second that will not.
PUT eql-issue-ignore-fields-delme/_doc/1
{
  "@timestamp": "2021-09-02T04:13:05.626Z",
  "some_keyword": "longer than normal",
  "other_keyword": "normal"
}

Then create an alert which queries everything from it:

and ensure signals are created:

To test the manual exclusions of any other problematic fields, create any index which has runtime fields or constant keywords but does not have anything within the _source document using dev tools. For example you can use constant keyword like so

PUT constant-keywords-deleme
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "testing_ignored": {
        "properties": {
          "constant": {
            "type": "constant_keyword",
            "value": "constant_value"
          }
        }
      },
      "testing_regex": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "normal_constant": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "small_field": {
        "type": "keyword",
        "ignore_above": 10
      }
    }
  }
}

PUT constant-keywords-deleme/_doc/1
{
  "@timestamp": "2021-09-02T04:20:01.760Z"
}

Set in your kibana.yml the key/value of:

xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/']

Setup a rule to run:

Once it runs you should notice that the constant values for testing are not on the signals table since it only typically exists in the fields API:

But the normal one still exists:

If you change the xpack.securitySolution.alertIgnoreFields by removing it and re-generate the signals you will see these values added back.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[tylersmalley]

Docker changes LGTM

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 1132b33
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💔 Build #150436 failed 42d0a9e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @FrankHassanabad

[jonathan-buttner]

Code changes look good! Just left a few comments about invalid regular expressions

[FrankHassanabad]

@elasticmachine merge upstream

[rylnd]

I had one incredibly minor nit about an incorrect comment in a test. Otherwise, this LGTM!

[spong]

For posterity, ES fieldnames can match RE syntax, but in testing I wasn't able to break anything here, so more of a FunFact™ than anything...

[FrankHassanabad]

Thanks, I think if you really needed it you could escape them like /\/test\// but you could only use RegEx to match those strings. So they're not blocked, they just would very annoyingly have to use reg ex to match slashes like that.

Really though, the hope is we never have to use this feature other than extreme emergencies and even with that use case we are hopeful we don't have to use it at all.

[spong]

Checked out, ran through test scenarios locally, and reviewed code. Appreciate the thoroughness here and all the updated tests! LGTM! 👍

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: f214f9c
• Storybooks Preview
• Documentation Changes

Metrics [docs]

✅ unchanged

History

• 💛 Build #150747 was flaky c3ba0d0
• 💔 Build #150729 failed 828fa74
• 💔 Build #150684 failed e755c3e
• 💚 Build #150448 succeeded 1132b33
• 💔 Build #150436 failed 42d0a9e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @FrankHassanabad

[kibanamachine]

💔 Backport failed

Status	Branch	Result
❌	7.15	Commit could not be cherrypicked due to conflicts
✅	7.x
❌	7.14	Commit could not be cherrypicked due to conflicts

Successful backport PRs will be merged automatically after passing CI.

To backport manually run:
node scripts/backport --pr 110927

[ghost]

Hi @MadameSheema ,

We have validated this ticket on 7.14.2 BC1 build and observed that issue is "Fixed". Please find the below observations:

Build Details:

Version:7.14.2 BC6
Commit:6b971d6410ca4a99271f2abf9d3de68b7a3653a5
Build:42959

Observation 1:

PUT eql-issue-ignore-fields-delme

{
  "mappings" : {
    "dynamic": "strict",
    "properties" : {
      "@timestamp": {
        "type": "date"
      },
      "some_keyword" : {
        "ignore_above": 5, 
        "type" : "keyword"
      },
      "other_keyword" : {
        "ignore_above": 10, 
        "type" : "keyword"
      }
    }
  }
}

eql-issue-ignore-fields-delme/_doc/1

{
  "@timestamp": "2021-09-02T04:13:05.626Z",
  "some_keyword": "longer than normal",
  "other_keyword": "normal"
}

1. Alerts are triggered when create the EQL rule with above index.

Screenshot:

Observation 2:

PUT constant-keywords-deleme

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "testing_ignored": {
        "properties": {
          "constant": {
            "type": "constant_keyword",
            "value": "constant_value"
          }
        }
      },
      "testing_regex": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "normal_constant": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "small_field": {
        "type": "keyword",
        "ignore_above": 10
      }
    }
  }
}

constant-keywords-deleme/_doc/1

{
  "@timestamp": "2021-09-02T04:20:01.760Z"
}

2. Alerts are triggered when create the Custom rule with above index without adding the xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/']

Screenshot:

Observation 3:

PUT constant-keywords-deleme

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "testing_ignored": {
        "properties": {
          "constant": {
            "type": "constant_keyword",
            "value": "constant_value"
          }
        }
      },
      "testing_regex": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "normal_constant": {
        "type": "constant_keyword",
        "value": "constant_value"
      },
      "small_field": {
        "type": "keyword",
        "ignore_above": 10
      }
    }
  }
}

constant-keywords-deleme/_doc/1

{
  "@timestamp": "2021-09-02T04:20:01.760Z"
}

3. Alerts are triggered when create the Custom rule with above index adding the xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/'] [We have test this scenario on on-prem setup]

Screenshot:

Please let us know if anything else is need to be test from our end.

Thanks!!