[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation #108115

banderror · 2021-08-10T21:28:23Z

Addresses: #106421, #106428, #102089, #106433

Summary

This PR focuses on consolidation of indexing implementations in rule_registry (#101016). It addresses some of the sub-tasks of the parent ticket.

Encapsulate index bootstrapping logic in a new improved API exposed by RuleDataService.
Enforce allowed values for the datasetSuffix on the API level.
Migrate plugins using the existing RuleDataService API to the improved one.
Make sure index names comply with design architecture.
- [RAC] Make sure index names comply with design architecture #102089
Improve the API of RuleDataClient.
Enhance index bootstrapping: support custom ILM policy per index ({registrationContext}.{datasetSuffix}).
Enhance index bootstrapping: create index template per namespace and support rollovers properly
- based on [Draft][Rule Registry] Create index template per namespace #107700
Enhance index bootstrapping: support secondary aliases
- based on [Draft][Rule Registry] Create index template per namespace #107700
Remove EventLogService implementation
- [RAC] Remove EventLogService implementation from rule_registry #106433

This will be addressed in follow-up PRs:

Enhance index bootstrapping: implement suggestions for backwards compatibility (naming scheme for alias and backing indices; versioning).
Enhance index bootstrapping: implement upgrades of existing index templates.
Make index bootstrapping logic more robust. This is partially addressed in this PR, but more improvements are needed.
Change the way index prefix works. UPD: [RAC] Disable RAC multi-tenancy #108506
Add support for optional TS schema (static typing).
Update README in rule_registry.

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2021-08-11T01:45:29Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2021-08-11T01:48:37Z

Pinging @elastic/security-solution (Team: SecuritySolution)

x-pack/plugins/infra/server/services/rules/rule_data_client.ts

elasticmachine · 2021-08-11T01:59:36Z

Pinging @elastic/uptime (Team:uptime)

elasticmachine · 2021-08-11T01:59:36Z

Pinging @elastic/apm-ui (Team:apm)

banderror · 2021-08-11T02:00:31Z

x-pack/plugins/infra/server/services/rules/rule_data_client.ts

+    componentTemplates: [
+      {
+        name: 'mappings',
+        version: 0,


Proper versioning according to the recent discussions will be implemented in a follow-up PR.

banderror · 2021-08-11T02:04:17Z

x-pack/plugins/observability/server/plugin.ts

+    const { ruleDataService } = plugins.ruleRegistry;
+    const ruleDataClient = ruleDataService.initializeIndex({
+      feature: 'observability',
+      registrationContext: 'observability',
+      dataset: Dataset.alerts,
+      componentTemplateRefs: [],
+      componentTemplates: [],
+      indexTemplate: {
+        version: 0,
+      },
+    });


This replaces the previous implementation:

const ruleDataClient = plugins.ruleRegistry.ruleDataService.getRuleDataClient( 'observability', plugins.ruleRegistry.ruleDataService.getFullAssetName(), () => Promise.resolve() );

Seems like it's a stub. Should I keep it? Why it's needed?

The observability plugin apparently needs a ruleDataClient for use in some route handlers without intending to write any alerts itself. Does this initializeIndex create any assets which the previous implementation didn't?

Does this initializeIndex create any assets which the previous implementation didn't?

It will see if it should bootstrap any resources for .alerts-observability. Specifically:

ILM policy if it's provided. In this case, it's not.

Component templates if they are provided. In this case, they're not.

And then it will update mappings of all concrete indices matching .alerts-observability-*. This pattern shouldn't match any indices, I guess, because you're saying this ruleDataClient is not used for writing alerts? Other indices containing alerts are named like .alerts-observability.apm-* etc.

Is this ruleDataClient used for reading all the observability alerts - from .alerts-observability.apm-*, .alerts-observability.logs-* and other indices?

It seems for this use case it would be useful to expose some kind of getRuleDataReader API in RuleDataPluginService that skips all the index management steps and just returns an IRuleDataReader. If I understand correctly the registrationContext would be observability.*.

I was also thinking about that

I created a ticket for that: #111173

marshallmain

I went through and did a first pass of the changes here. I found a few ways that I think we can simplify the implementation further, especially regarding signalling when asynchronous template installation functions are complete and the next operations can proceed.

But, very exciting to see the new usage of ruleDataService.initializeIndex! It's a huge improvement to move the initialization and signalling logic into a single place rather than having each plugin implement it separately.

x-pack/plugins/rule_registry/server/rule_data_plugin_service/rule_data_plugin_service.ts

x-pack/plugins/rule_registry/server/rule_data_client/rule_data_client.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/rule_data_plugin_service.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_names.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/rule_data_plugin_service.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/index_names.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/rule_data_plugin_service.ts

…alerts

…ing implementation (elastic#108115) **Addresses:** elastic#106421, elastic#106428, elastic#102089, elastic#106433 ## Summary This PR focuses on consolidation of indexing implementations in `rule_registry` (elastic#101016). It addresses some of the sub-tasks of the parent ticket. - [x] Encapsulate index bootstrapping logic in a new improved API exposed by `RuleDataService`. - [x] Enforce allowed values for the `datasetSuffix` on the API level. - [x] Migrate plugins using the existing `RuleDataService` API to the improved one. - [x] Make sure index names comply with design architecture. - elastic#102089 - [x] Improve the API of `RuleDataClient`. - [x] Enhance index bootstrapping: support custom ILM policy per index (`{registrationContext}.{datasetSuffix}`). - [x] Enhance index bootstrapping: create index template per namespace and support rollovers properly - based on elastic#107700 - [x] Enhance index bootstrapping: support secondary aliases - based on elastic#107700 - [x] Remove `EventLogService` implementation - elastic#106433 This will be addressed in follow-up PRs: - [ ] Enhance index bootstrapping: implement suggestions for backwards compatibility (naming scheme for alias and backing indices; versioning). - [ ] Enhance index bootstrapping: implement upgrades of existing index templates. - [ ] Make index bootstrapping logic more robust. This _is partially addressed_ in this PR, but more improvements are needed. - [ ] Change the way index prefix works. - [ ] Add support for optional TS schema (static typing). - [ ] Update `README` in `rule_registry`. ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

kibanamachine · 2021-08-15T12:55:56Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…ing implementation (#108115) (#108638) **Addresses:** #106421, #106428, #102089, #106433 ## Summary This PR focuses on consolidation of indexing implementations in `rule_registry` (#101016). It addresses some of the sub-tasks of the parent ticket. - [x] Encapsulate index bootstrapping logic in a new improved API exposed by `RuleDataService`. - [x] Enforce allowed values for the `datasetSuffix` on the API level. - [x] Migrate plugins using the existing `RuleDataService` API to the improved one. - [x] Make sure index names comply with design architecture. - #102089 - [x] Improve the API of `RuleDataClient`. - [x] Enhance index bootstrapping: support custom ILM policy per index (`{registrationContext}.{datasetSuffix}`). - [x] Enhance index bootstrapping: create index template per namespace and support rollovers properly - based on #107700 - [x] Enhance index bootstrapping: support secondary aliases - based on #107700 - [x] Remove `EventLogService` implementation - #106433 This will be addressed in follow-up PRs: - [ ] Enhance index bootstrapping: implement suggestions for backwards compatibility (naming scheme for alias and backing indices; versioning). - [ ] Enhance index bootstrapping: implement upgrades of existing index templates. - [ ] Make index bootstrapping logic more robust. This _is partially addressed_ in this PR, but more improvements are needed. - [ ] Change the way index prefix works. - [ ] Add support for optional TS schema (static typing). - [ ] Update `README` in `rule_registry`. ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Georgii Gorbachev <[email protected]>

kibanamachine · 2021-08-16T11:08:06Z

💚 Build Succeeded

Metrics [docs]

Unknown metric groups

API count

id	before	after	diff
`ruleRegistry`	95	114	+19

API count missing comments

id	before	after	diff
`ruleRegistry`	95	94	-1

Non-exported public API item count

id	before	after	diff
`ruleRegistry`	11	4	-7

History

💚 Build #145516 succeeded 077ca24
💚 Build #145302 succeeded adfe31038620ac11007891c7d33a3daa1655a398
💛 Build #145172 was flaky adfe31038620ac11007891c7d33a3daa1655a398
💚 Build #145027 succeeded 05dc4fa
💚 Build #144998 succeeded 495440360807431cfc2e43f4fca2a0e37386f80b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @banderror

banderror added Team:Detections and Resp Security Detection Response Team Theme: rac label obsolete labels Aug 10, 2021

banderror self-assigned this Aug 10, 2021

banderror force-pushed the improve-rule-data-service-and-client branch from a48f27d to a6fbb74 Compare August 10, 2021 21:29

banderror requested a review from marshallmain August 10, 2021 22:01

banderror changed the title ~~[RAC] Improve RuleDataService API and index bootstrapping implementation~~ [RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation Aug 10, 2021

banderror force-pushed the improve-rule-data-service-and-client branch from d799fa8 to 8d75711 Compare August 11, 2021 01:00

banderror marked this pull request as ready for review August 11, 2021 01:45

banderror requested review from a team as code owners August 11, 2021 01:45

banderror requested a review from weltenwort August 11, 2021 01:45

banderror added release_note:skip Skip the PR/issue when compiling release notes v7.15.0 v8.0.0 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Aug 11, 2021

banderror commented Aug 11, 2021

View reviewed changes

x-pack/plugins/infra/server/services/rules/rule_data_client.ts Outdated Show resolved Hide resolved

botelastic bot added Team:APM All issues that need APM UI Team support Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability labels Aug 11, 2021

banderror commented Aug 11, 2021

View reviewed changes

banderror added the auto-backport Deprecated - use backport:version if exact versions are needed label Aug 11, 2021

marshallmain reviewed Aug 11, 2021

View reviewed changes

jasonrhodes mentioned this pull request Aug 11, 2021

[RAC] 7.15 Observability Checklist #107933

Closed

52 tasks

banderror force-pushed the improve-rule-data-service-and-client branch from 6895ff9 to f625838 Compare August 11, 2021 16:29

banderror and others added 13 commits August 15, 2021 12:13

Old consolidation draft (TEMP, TO BE REMOVED)

ce85666

Improve the API of RuleDataService: draft of the new API

bac3d92

Improve the API of RuleDataService: clean up

7f53c4c

Improve the API of RuleDataService: migrate plugins to the new API

e83207a

Add TODO comments

591dd18

Index bootstrapping: create index template per namespace

a061a4e

Index bootstrapping: add secondary alias (.siem-signals) to security.…

461db8d

…alerts

Improve the API of RuleDataClient

6ccccbb

Fix: const enums are not supported

dccd175

Remove EventLogService implementation

4ecf2e3

Clean up

926cada

Address code review feedback

826276b

Swap right and left, wrap initializeService result

077ca24

banderror force-pushed the improve-rule-data-service-and-client branch from adfe310 to 077ca24 Compare August 15, 2021 10:13

banderror merged commit 2ee11db into elastic:master Aug 15, 2021

banderror deleted the improve-rule-data-service-and-client branch August 15, 2021 12:53

kibanamachine mentioned this pull request Aug 15, 2021

[7.x] [RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation (#108115) #108638

Merged

weltenwort mentioned this pull request Aug 19, 2021

[RAC] Observability alerts table reads from incorrect indices #109317

Closed

marshallmain mentioned this pull request Aug 25, 2021

[RAC][Rule Registry] Namespaces that are prefixes of other namespaces can cause unexpected behaviors #107704

Closed

This was referenced Sep 3, 2021

[RAC][Rule Registry] UI/UX around timeouts and errors during index bootstrapping #111170

Open

[RAC][Rule Registry] Implement RuleDataService.getReader() that skips index bootstrapping #111173

Closed

[RAC][Rule Registry] Small clean-ups #111470

Closed

ymao1 mentioned this pull request Oct 12, 2022

[Response Ops][Alerting] Research best practices for bootstrapping alerts as data indices #141146

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation #108115

[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation #108115

banderror commented Aug 10, 2021 •

edited

Loading

elasticmachine commented Aug 11, 2021

elasticmachine commented Aug 11, 2021

elasticmachine commented Aug 11, 2021

elasticmachine commented Aug 11, 2021

banderror Aug 11, 2021

banderror Aug 11, 2021

weltenwort Aug 11, 2021

banderror Aug 11, 2021

marshallmain Aug 12, 2021

banderror Aug 12, 2021

banderror Sep 3, 2021

marshallmain left a comment

kibanamachine commented Aug 15, 2021

kibanamachine commented Aug 16, 2021

API count

API count missing comments

Non-exported public API item count

[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation #108115

[RAC][Rule Registry] Improve RuleDataService API and index bootstrapping implementation #108115

Conversation

banderror commented Aug 10, 2021 • edited Loading

Summary

Checklist

elasticmachine commented Aug 11, 2021

elasticmachine commented Aug 11, 2021

elasticmachine commented Aug 11, 2021

elasticmachine commented Aug 11, 2021

banderror Aug 11, 2021

Choose a reason for hiding this comment

banderror Aug 11, 2021

Choose a reason for hiding this comment

weltenwort Aug 11, 2021

Choose a reason for hiding this comment

banderror Aug 11, 2021

Choose a reason for hiding this comment

marshallmain Aug 12, 2021

Choose a reason for hiding this comment

banderror Aug 12, 2021

Choose a reason for hiding this comment

banderror Sep 3, 2021

Choose a reason for hiding this comment

marshallmain left a comment

Choose a reason for hiding this comment

kibanamachine commented Aug 15, 2021

💚 Backport successful

kibanamachine commented Aug 16, 2021

💚 Build Succeeded

Metrics [docs]

API count

API count missing comments

Non-exported public API item count

History

banderror commented Aug 10, 2021 •

edited

Loading