-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution][Alerts] Use dynamic mappings to create smaller templates & mappings #100884
Comments
Pinging @elastic/kibana-security (Team:Security) |
@tsg did you mean to use the |
Pinging @elastic/siem (Team:SIEM) |
Argh, yes, sorry. |
FYI on another evolution on the Integration side: elastic/package-spec#178 The approach above doesn't have the same problem for us, because the top-level fields are always well-known. |
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
Hey everyone, FYI ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area ( |
This issue primarily affects security solution alerts indices since those indices are currently the only rule registry-based indices that use the full ECS mappings. |
The Fleet and Elasticsearch teams have optimized the data ingestion mappings to use dynamic mappings as much as possible: elastic/elasticsearch#64978 Here is the current version of the template: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/resources/data-streams-mappings.json
We should consider doing the same for alerts as data.
The idea is to take advantage of the following observations:
*.ip
) and text fields (*.message
)In addition to the benefit of having smaller templates, this means that all the fields that don't show up explicitly (keywords) are zero-cost when they are not used. This reduces the cost of assuming the whole of ECS is the base of the Alert schema.
We need to be careful, though, to not cause mapping conflicts, because we're copying data from disparate indices that can be in conflict with one another. I think the following approach would be feasible:
alert.original_event
field is set to be not indexed at all or flattenedalert.original_event
.alert.original_event
where it can't cause conflicts.The text was updated successfully, but these errors were encountered: