[Security Solution][Detections] No results in Rule Preview

[banderror]

Summary

I have a test index for source events with custom non-ECS fields. If I create a rule that targets this index, I can't see any results in the Rule Preview, but the rule is able to find the events and generate alerts if you save and run it.

Repro steps

Create a custom index for source events and write a test object with a timestamp as of today:

PUT /sdh-247
{
  "mappings": {
    "dynamic": "true",
    "properties": {
      "@timestamp": { "type": "date" },
      "deviceCustomDateLabel": { "type": "keyword" },
      "deviceCustomDate_Date": { "type": "date" },
      "deviceCustomDate_Long": { "type": "long" },
      "deviceCustomDate_DateNanos": { "type": "date_nanos" }
    }
  },
  "settings": {
    "index": {
      "number_of_shards": 1
    }
  }
}

PUT /sdh-247/_doc/1
{
  "@timestamp": "2021-11-18T12:10:30Z",
  "deviceCustomDateLabel": "TRENUTAN_DATUM_UO",
  "deviceCustomDate_Date": "2021-11-18T12:10:30Z",
  "deviceCustomDate_Long": 1074640200000,
  "deviceCustomDate_DateNanos": 1074640200000
}

Create a Custom query rule that should generate an alert based off of it:

Observe no results in the Rule Preview:

Set a sufficient look-back time so the rule could find the source event (e.g. 48 hours). Save and run this rule, and it should find the event and generate an alert from it:

Environment

Local dev environment
Branch: main
Kibana config:

elasticsearch:
  username: 'kibana_system'
  password: 'changeme'
  hosts: 'http://localhost:9200'

xpack.ruleRegistry.write.enabled: true
xpack.ruleRegistry.unsafe.indexUpgrade.enabled: true
xpack.ruleRegistry.unsafe.legacyMultiTenancy.enabled: true
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[ecezalp]

inspecting the rule preview query

adding "event.category": {"type": "keyword"} mapping to the sdh-247 index, and creating a new event

PUT /sdh-247/_doc/2
{
  "@timestamp": "2021-11-18T12:10:30Z",
  "deviceCustomDateLabel": "TRENUTAN_DATUM_UO",
  "deviceCustomDate_Date": "2021-11-18T12:10:30Z",
  "deviceCustomDate_Long": 1074640200000,
  "deviceCustomDate_DateNanos": 1074640200000,
  "event": {
    "category": "test"
  }
}

the new event with the category shows up on the preview

[ecezalp]

the problem can be fixed by adding a value for the missing parameter to the matrix histogram query.

[ecezalp]

#119253