Add custom threshold rule

[mdbirnstiehl]

This PR closes Issues 3031 and 3134.

Preview HERE

[github-actions]

A documentation preview will be available soon:

• 📚 HTML diff
• 📙 Observability Guide

[mergify]

This pull request does not have a backport label. Could you fix it @mdbirnstiehl? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[lcawl]

FYI In my local tests, this new rule is also affected by elastic/kibana#162605. That is to say, it's got a "role visibility" field that you must set when you create the rule, and you can pick either "metrics" or "logs". In case it's helpful, I updated the ES query rule in the Kibana docs recently for the same new feature: https://www.elastic.co/guide/en/kibana/master/rule-type-es-query.html#_define_the_conditions_2

You must select a scope value, which affects the Kibana feature privileges that are required to access the rule. For example when it’s set to Stack Rules, you must have the appropriate Management > Stack Rules feature privileges to view or edit the rule.

I think the same is likely true for this new rule (except "Stack rules" isn't one of the options here).

[mdbirnstiehl]

FYI In my local tests, this new rule is also affected by elastic/kibana#162605. That is to say, it's got a "role visibility" field that you must set when you create the rule, and you can pick either "metrics" or "logs". In case it's helpful, I updated the ES query rule in the Kibana docs recently for the same new feature: https://www.elastic.co/guide/en/kibana/master/rule-type-es-query.html#_define_the_conditions_2

You must select a scope value, which affects the Kibana feature privileges that are required to access the rule. For example when it’s set to Stack Rules, you must have the appropriate Management > Stack Rules feature privileges to view or edit the rule.

I think the same is likely true for this new rule (except "Stack rules" isn't one of the options here).

Thanks, Lisa! I'll update accordingly.

[maryam-saeidi]

Looks good already, added some comments for clarification.

[maryam-saeidi]

Great document and thanks for taking care of comments :)

[colleenmcginnis]

Just a few little suggestions below!

[colleenmcginnis]

✨