Collect new alerting telemetry data from Kibana index

[ersin-erdal]

Resolves #138996

This PR covers only the telemetry data that we can aggregate from the Kibana index (numbers 1-6 in the issue). The telemetry data from event-log index (numbers 7-9) will be covered by another follow up PR.

To Verify:

• Create some rules with tags, some without
• Use different notify when options (Every time alert is active, On status change etc)
• Snooze one of the rules
• Mute (AKA infinitive snooze) one of the rules
• Mute one of the alerts

Change telemetry task run interval to something very short time (e.g. 1 min)

kibana/x-pack/plugins/alerting/server/usage/task.ts

Line 197 in 01ecbd4

return moment().add(1, 'd').startOf('d').toDate();

Expect to see below data on https://localhost:5601/api/stats?extended=true&legacy=true
count_rules_by_execution_status,
count_rules_with_tags,
count_rules_by_notify_when,
count_rules_snoozed,
count_rules_muted,
count_rules_with_muted_alerts

[ersin-erdal]

Not sure about this DYNAMIC_KEY usage...

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

Verified new telemetry is being generated as expected. Nice job! Left a question about whether we should be explicitly mapping all the possible connector types.

[ymao1]

This is going to look for a field DYNAMIC_KEY and apply the mapping to that, so the mapping will not be applied to data like siem.__email or alerts.__email. With other items that we break down by rule type id, we have to explicitly map each possible value.

However, since we plan to copy these to the custom response ops index via the indexer, maybe having this mapping doesn't matter as much anymore? WDYT @mikecote

[mikecote]

From what I recall, the introduction of DYNAMIC_KEY was to allow telemetry to be indexed without failures (using strict mappings?). I wonder if this has changed now where they just store the data within the _source unmapped.

@elastic/kibana-core can you weigh in on whether or not we still need to use DYNAMIC_KEY to make such data accessible in our custom telemetry indexer?

[ersin-erdal]

maybe we should ping @elastic/kibana-core one more time :)

[pgayvallet]

cc @afharo @Bamieh

[afharo]

Thank you for the ping! I think there is some confusion between the schema and the mappings:

Kibana describes the schema as a way to declare a contract of the format of the document we ship. This may or may not result in a direct mapping approach.

---

Re DYNAMIC_KEY, it was introduced to allow Kibana developers to specify tree-shaped structures where the keys are dynamic/arbitrarily built. Whether we use flattened type, use dynamic templates or add the known keys to the mappings, it's up to the analysis end to decide. The important thing is that, from the Kibana POV, it's sending an object with an arbitrary set of keys.

---

Re the keys including a dot inside, that's a good point, @ymao1. We allow shipping objects like:

{
  "siem.__prop1": "some value",
  "siem.__prop2": "some other value"
}

Technically speaking, it adheres to the DYNAMIC_KEY contract, although I agree that indexing that into ES might end up as:

{
  "siem": {
    "__prop1": "some value",
    "__prop2": "some other value"
  }
}

FWIW, I don't think it's such an issue:

• If we declare count_connector_types_by_consumers as type: "flattened", it won't break.
• If we declare dynamic_templates with match_path: "count_connector_types_by_consumers.*, it won't break.
• If we explicitly declare their mappings, it won't break either.

The only potential error would be if we also report the "siem" key on its own. Do we?

[ersin-erdal]

The only potential error would be if we also report the "siem" key on its own. Do we?

No, this is not possible.

And thank you for clarifying DYNAMIC_KEY usage :)

[ymao1]

LGTM

[afharo]

LGTM

[mikecote]

Changes LGTM! Tested locally and saw the telemetry reflect the state of my rules.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 9377d19

Failed CI Steps

• FTR Configs #50

Test Failures

• [job] [logs] FTR Configs #50 / apis UnifiedFieldList field stats apis field distribution should return examples for non-aggregatable fields

Metrics [docs]

✅ unchanged

History

• 💚 Build #71422 succeeded 5077714
• 💚 Build #70112 succeeded 4862fa2
• 💚 Build #69405 succeeded 7cd445a
• 💔 Build #69361 failed cc8e835
• 💛 Build #69283 was flaky 96bcc65
• 💔 Build #69264 failed 0f7a7a4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ersin-erdal