Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solutions][Detection Engine] Throws date time format errors on custom mappings #79865

Closed
FrankHassanabad opened this issue Oct 7, 2020 · 0 comments · Fixed by #79911
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Security Solution rules and Detection Engine v7.10.0 v7.11.0 v8.0.0

Comments

@FrankHassanabad
Copy link
Contributor

Kibana version:
7.9.0+

Describe the bug:
When you have a custom timestamp mapping and run the detection engine against that mapping it can produce errors or it can not detect signals depending on what the timestamp happens to be.

Solution is to use format everywhere in ranges:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html

Such as here:
https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_events_query.ts#L40-L46

Likewise docValues for when we pull the @timestamp out:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-request-docvalue-fields.html

Steps to reproduce:
Add a epoch mapping and then get a valid epoch from here:
https://www.epochconverter.com/

PUT /frank-test-delme
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
        "@timestamp": {
            "type": "date",
            "format": "epoch_millis"
        }
    }
  }
}

PUT frank-test-delme/_doc/1
{
  "@timestamp": 1602085855941
}

Then create a detection rule against the index and you will see:
Screen Shot 2020-10-07 at 10 02 26 AM

Errors:

erver    log   [09:52:52.147] [error][plugins][plugins][securitySolution][securitySolution] Bulk Indexing of signals failed: [parse_exception] failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]: [failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]] :: {"path":"/frank-test-delme/_search","query":{"allow_no_indices":true,"size":100,"ignore_unavailable":true},"body":"{\"query\":{\"bool\":{\"filter\":[{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}}],\"should\":[],\"must_not\":[]}},{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"range\":{\"@timestamp\":{\"gte\":\"2020-10-07T15:46:51.254Z\"}}}],\"minimum_should_match\":1}},{\"bool\":{\"should\":[{\"range\":{\"@timestamp\":{\"lte\":\"2020-10-07T15:52:51.254Z\"}}}],\"minimum_should_match\":1}}]}},{\"match_all\":{}}]}},\"sort\":[{\"@timestamp\":{\"order\":\"asc\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]: [failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]]\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"frank-test-delme\",\"node\":\"UmFIfD_iQfepBl6sUyEk2g\",\"reason\":{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]: [failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [2020-10-07T15:46:51.254Z] with format [epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"date_time_parse_exception: Failed to parse with all enclosed parsers\"}}}}]},\"status\":400}"} name: "host create after" id: "02cbbb9f-4759-4fcd-a615-adf6bbfd1019" rule id: "b4cf64d9-ce06-45a5-98ce-2f2b1354c830" signals index: ".siem-signals-hassanabad4-default"

Expected behavior:
No errors, the rule as expected if the custom mapping is Epoch

@FrankHassanabad FrankHassanabad self-assigned this Oct 7, 2020
@FrankHassanabad FrankHassanabad added v7.10.0 v7.11.0 v8.0.0 Feature:Detection Rules Security Solution rules and Detection Engine bug Fixes for quality problems that affect the customer experience labels Oct 7, 2020
FrankHassanabad added a commit that referenced this issue Oct 8, 2020
…79911)

## Summary

Fixes #79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this issue Oct 8, 2020
…lastic#79911)

## Summary

Fixes elastic#79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this issue Oct 8, 2020
…lastic#79911)

## Summary

Fixes elastic#79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this issue Oct 8, 2020
…79911) (#79965)

## Summary

Fixes #79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this issue Oct 8, 2020
…79911) (#79964)

## Summary

Fixes #79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Security Solution rules and Detection Engine v7.10.0 v7.11.0 v8.0.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant