Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

Merged
merged 34 commits into from
Jan 27, 2023
Merged

[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

merged 34 commits into from
Jan 27, 2023

Conversation

jpdjere
Copy link
Contributor

@jpdjere jpdjere commented Dec 5, 2022 •
edited by banderror
Loading

Addresses: #130966
Based on: #135127

Summary

This PR deprecates the Sidecar SO of type siem-detection-engine-rule-execution-info in favour of storing Rule Execution Logging data within the rule itself, making use of the work previously done in the Alerting Framework:

Work done:

  • Pass execution statuses and metrics from rule executors to the Framework: through the use of RuleMonitoringService and RuleResultService from within the rule execution log client for executor. x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts
  • Fetch execution statuses and metrics from rules themselves instead of the sidecar siem-detection-engine-rule-execution-info saved objects: through the use of the new function createRuleExecutionSummary in x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/create_rule_execution_summary.ts, which extracts last execution information from the rule itself.
  • Remove the siem-detection-engine-rule-execution-info saved objects type from the codebase. Mark it as deleted in Kibana Core: added siem-detection-engine-rule-execution-info to packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core/unused_types.ts; and got rid of the related Saved Object client.
  • Make sure to keep backward compatibility in the Detection API endpoints and rule execution events we write into the Event Log: API compatibility is maintained. No breaking changes.

Checklist

@jpdjere jpdjere force-pushed the expand-rule-monitoring-security-solution branch from 931279d to c0407b1 Compare December 5, 2022 19:21
@jpdjere jpdjere added the ci:no-auto-commit Disable auto-committing changes on CI label Dec 6, 2022
@jpdjere jpdjere force-pushed the expand-rule-monitoring-security-solution branch from fb7c12f to 604ec4a Compare December 6, 2022 11:29
@jpdjere jpdjere mentioned this pull request Dec 7, 2022
Closed
@jpdjere jpdjere force-pushed the expand-rule-monitoring-security-solution branch from c09ed3e to 1ca9b7f Compare December 14, 2022 12:34
@jpdjere jpdjere force-pushed the expand-rule-monitoring-security-solution branch 2 times, most recently from 31a7159 to 9718858 Compare December 22, 2022 15:01
@jpdjere jpdjere changed the title [Security Solution] Expand RuleMonitoringService [Security Solution] Write and read Rule Execution Logs from rule instead of saved object Dec 29, 2022
@jpdjere jpdjere force-pushed the expand-rule-monitoring-security-solution branch from 7239852 to 2928e87 Compare December 29, 2022 16:29
@jpdjere jpdjere removed the ci:no-auto-commit Disable auto-committing changes on CI label Dec 29, 2022
@maximpn maximpn force-pushed the expand-rule-monitoring-security-solution branch from 8318023 to 95dc24d Compare January 16, 2023 16:31
@maximpn maximpn force-pushed the expand-rule-monitoring-security-solution branch from a38fcd4 to 9d60a59 Compare January 20, 2023 14:28
@maximpn maximpn marked this pull request as ready for review January 22, 2023 13:53
@maximpn maximpn requested review from a team as code owners January 22, 2023 13:53
@maximpn maximpn requested a review from spong January 22, 2023 13:53
@maximpn maximpn added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Detection Alerts Security Solution Detection Alerts Feature Team:Detection Rule Management Security Detection Rule Management Team v8.7.0 labels Jan 22, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@maximpn maximpn added backport:skip This commit does not require backporting release_note:feature Makes this part of the condensed release notes labels Jan 22, 2023
@maximpn maximpn force-pushed the expand-rule-monitoring-security-solution branch from 24c831c to 01b52fe Compare January 27, 2023 11:30
banderror
banderror approved these changes Jan 27, 2023
View reviewed changes
Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks for the fixes @maximpn! 🚀
Just one nit here for your consideration: #147035 (comment)

@maximpn maximpn force-pushed the expand-rule-monitoring-security-solution branch from 53466b1 to 8f0a58c Compare January 27, 2023 12:29
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Security Solution Tests #3 / Timelines Creates a timeline by clicking untitled timeline from bottom bar can be added notes

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 12.8MB 12.8MB +364.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id before after diff
siem-detection-engine-rule-execution-info 11 - -11
Unknown metric groups

References to deprecated APIs

id before after diff
securitySolution 386 384 -2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maximpn @jpdjere

@maximpn maximpn merged commit 6cc9855 into elastic:main Jan 27, 2023
@banderror banderror mentioned this pull request Jan 27, 2023
Closed
4 tasks
@maximpn maximpn mentioned this pull request Feb 1, 2023
Merged
@spong spong mentioned this pull request Feb 2, 2023
Merged
7 tasks
maximpn added a commit that referenced this pull request Feb 2, 2023
@maximpn
[Security Solution] Avoid exporting execution_summary field (#150097)
3c14f00 
## Summary

It fixes a problem of exporting `execution_summary` field while exporting detection rules which was introduce in #147035. Presence of that field make importing of just exported rule failing.

Tests to cover this fix will come in a separate PR.
kqualters-elastic pushed a commit to kqualters-elastic/kibana that referenced this pull request Feb 6, 2023
@jpdjere @kqualters-elastic
[Security Solution] Write and read Rule Execution Logs from rule inst…
fbd7f56 
…ead of saved object (elastic#147035)

**Addresses:** elastic#130966
**Based on:** elastic#135127

## Summary

This PR deprecates the Sidecar SO of type `siem-detection-engine-rule-execution-info` in favour of storing Rule Execution Logging data within the rule itself, making use of the work previously done in the Alerting Framework:
- elastic#140882
- elastic#147278

Work done:
- **Pass execution statuses and metrics from rule executors to the Framework:** through the use of `RuleMonitoringService` and `RuleResultService` from within the rule execution log client for executor. `x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts`
- **Fetch execution statuses and metrics from rules themselves instead of the sidecar `siem-detection-engine-rule-execution-info` saved objects**: through the use of the new function `createRuleExecutionSummary` in `x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/create_rule_execution_summary.ts`, which extracts last execution information from the rule itself.
- **Remove the siem-detection-engine-rule-execution-info saved objects type from the codebase. Mark it as deleted in Kibana Core:** added `siem-detection-engine-rule-execution-info` to `packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core/unused_types.ts`; and got rid of the related Saved Object client.
- **Make sure to keep backward compatibility in the Detection API endpoints and rule execution events we write into the Event Log**: API compatibility is maintained. No breaking changes.


### Checklist

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@spong spong mentioned this pull request Apr 24, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maximpn maximpn maximpn left review comments

@banderror banderror banderror approved these changes

@spong spong spong approved these changes

@e40pud e40pud e40pud approved these changes

@pgayvallet pgayvallet pgayvallet approved these changes

Assignees

@maximpn maximpn

@jpdjere jpdjere

Labels
backport:skip This commit does not require backporting Feature:Rule Monitoring Security Solution Detection Rule Monitoring area release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@jpdjere @elasticmachine @spong @maximpn @kibana-ci @pgayvallet @e40pud @banderror @kibanamachine