Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor IndexManagement to support custom actions #288

Merged

Conversation

downsrob
Copy link
Contributor

@downsrob downsrob commented Mar 8, 2022

Index State Management Refactor

This PR makes Index Management extensible and refactors the existing IndexStateManagement (ISM) actions to be built using the new extensible interface. This interface enables you to create extension plugins which can be loaded by index management. Your extension plugin can provide two things:

  1. New ISM actions
  2. New index types with custom index metadata services. For example, if you offload your hot data to a persistent storage system, the "cold" index's IndexMetadata is not in cluster state, where hot index metadata is, so you need to provide a custom metadata service for ISM to manage it.

Goals

  • Define a new interface in IndexManagement SPI (SPI interface is provided by OpenSearch framework to extend a plugin) to allow the creation of custom actions in new plugins which extend the Index Management SPI.
  • Refactor all ISM actions in IndexManagement based on the new SPI
  • Refactor the ManagedIndexCoordinator, ManagedIndexRunner and ISM Rest APIs to use the new interface methods to process requests for different index types
  • Enable the creation of custom index types and metadata services, decoupling metadata from the cluster state and allowing an extension to manage off cluster indices.

Key Interfaces

  • IndexManagementExtension - The interface for plugins extending index management. Through this interface, you define the extension name, and the custom action parsers. Optionally, you may provide a map of custom index type to metadata service, enabling an extension to provide a custom method of getting ISMIndexMetadata, which contains index uuid and index creation date. This uuid is used to uniquely identify an index, so that a policy may be attached to it. The index creation date is used for transitions. By decoupling uuid and index creation date from the cluster state, an extension can enable ISM to manage off cluster indices. This interface also allows for a custom StatusChecker, used to denote whether the extension is enabled or disabled.
  • Action - The interface for custom actions. By extending this interface, you may create a custom action with new fields, and add the implementation to correctly serialize the internals of the action when stored internally or returned as response. Actions are further split into Steps where each Action can be executed in multiple steps.
  • ActionParser - The interface for the parsing of custom actions. A custom parser is needed for each custom action, and it provides implementation for the deserialization of the internals of actions. It is also important to note the action type in this interface. This action type must be unique, it typically describes what the action does, such as "rollover" for the rollover action.
  • Step - Provides the implementation of the action execution, as well as fields such as if the step may be safely disabled on, if it is idempotent, and the step name.

Example

Though we do not yet provide an example plugin, a sample parser, action and step are implemented here. This custom action is used for unit testing, and does not actually execute anything meaningful. As Index management internal actions also follow these interfaces, you can refer to the internal actions for more examples. Example step with meaningful execution: attempt_close

Additional Information

This commit squashes all refactoring changes, for a more granular view of the changes you can check out the commit history of the development branch.

CheckList:

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

thalurur and others added 2 commits March 8, 2022 08:59
Signed-off-by: Ravi Thaluru <[email protected]>

Base commit to clean up old action interfaces and disabling all ISM related tests (opensearch-project#218)

Signed-off-by: Ravi Thaluru <[email protected]>

Implement DeleteAcion using new interface (opensearch-project#221)

Signed-off-by: Ravi Thaluru <[email protected]>

Adding base logic to transition step to enable policy execution (opensearch-project#223)

Signed-off-by: Ravi Thaluru <[email protected]>

Support close action using new interface (opensearch-project#224)

* Implement close action

Signed-off-by: Annie Lee <[email protected]>

* Update functions

Signed-off-by: Annie Lee <[email protected]>

* Update AttemptCloseStepTests.kt

Signed-off-by: Annie Lee <[email protected]>

* Mark a test as private for now

Since TransitionAction is not yet implemented. Marking a test as private to avoid integ test failure

Signed-off-by: Annie Lee <[email protected]>

* Update CloseActionIT.kt

Signed-off-by: Annie Lee <[email protected]>

Implement ReadOnlyAction using new interface (opensearch-project#227)

* Refactors ReadOnlyAction

Signed-off-by: Robert Downs <[email protected]>

Implement ReadWriteAction using new interface (opensearch-project#228)

Signed-off-by: Clay Downs <[email protected]>

Implement OpenAction using new interface (opensearch-project#230)

* Support open action

Signed-off-by: Annie Lee <[email protected]>

* Update AttemptOpenStep.kt

Signed-off-by: Annie Lee <[email protected]>

* Add close action test

Signed-off-by: Annie Lee <[email protected]>

* Add open action related tests

Signed-off-by: Annie Lee <[email protected]>

* Add open action test round trip

Signed-off-by: Annie Lee <[email protected]>

* Fix open action xcontent test

Signed-off-by: Annie Lee <[email protected]>

* Modify XContentTests for better comparison

Signed-off-by: Annie Lee <[email protected]>

* Update XContentTests.kt

Signed-off-by: Annie Lee <[email protected]>

Implements RolloverAction with new interface, fixes default action retry commit (opensearch-project#231)

* Implements rollover action with new interface, fixes default action retry

Signed-off-by: Clay Downs <[email protected]>

Support ReplicaCountAction using new interface (opensearch-project#233)

Signed-off-by: Annie Lee <[email protected]>

Refactors rollup action and enables multi step actions (opensearch-project#235)

Signed-off-by: Robert Downs <[email protected]>

Refactors Notification action with new interface (opensearch-project#238)

* Refactors notification action with new interface

Signed-off-by: Robert Downs <[email protected]>

Upgrades detekt version, fixes flaky tests (opensearch-project#254)

* Upgrades detekt version to 1.17.1 (opensearch-project#252)

* Adds detekt ignores to not-yet-refactored files

* Fixes flaky rollup/transform explain IT (opensearch-project#247)

Signed-off-by: Robert Downs <[email protected]>

Support force merge action using new interface (opensearch-project#256)

* Support force merge action

Signed-off-by: Annie Lee <[email protected]>

Refactors Snapshot action to use new interface (opensearch-project#253)

* Refactors snapshot action to use new interface

Signed-off-by: Clay Downs <[email protected]>

Support index priority action using new interface (opensearch-project#257)

Signed-off-by: Annie Lee <[email protected]>

Support Allocation action using new interface (opensearch-project#246)

* Support Allocation action using new interface

Signed-off-by: Annie Lee <[email protected]>

* Pass in required parameter

Signed-off-by: Annie Lee <[email protected]>

* Update AttemptAllocationStep.kt

Adding correct const

Signed-off-by: Annie Lee <[email protected]>

* Update AttemptAllocationStep.kt

Typo in message

Signed-off-by: Annie Lee <[email protected]>

* Debug tests

Signed-off-by: Annie Lee <[email protected]>

* Update XContentTests.kt

Signed-off-by: Annie Lee <[email protected]>

* Revert "Debug tests"

This reverts commit d7123bd.

Signed-off-by: Annie Lee <[email protected]>

* Update IndexPolicyRequestTests.kt

Signed-off-by: Annie Lee <[email protected]>

* Support force merge action using new interface (opensearch-project#256)

* Support force merge action

Signed-off-by: Annie Lee <[email protected]>

* Update AllocationActionIT.kt

Signed-off-by: Annie Lee <[email protected]>

* Revert "Update IndexPolicyRequestTests.kt"

This reverts commit bd34e2e.

Signed-off-by: Annie Lee <[email protected]>

* Add debug log

Signed-off-by: Annie Lee <[email protected]>

* Remove debug log and change order of parameters

Signed-off-by: Annie Lee <[email protected]>

* Update after comments

Fix order of include exclude parameter in parser
Add check to tests
Remove "Config" in message

Signed-off-by: Annie Lee <[email protected]>

Merging changes from main branch (opensearch-project#259)

Signed-off-by: Ravi Thaluru <[email protected]>

merge with main (opensearch-project#270)

Signed-off-by: Ravi Thaluru <[email protected]>

Adds custom action parsing (opensearch-project#273)

* Adds custom action parsing logic

Signed-off-by: Robert Downs <[email protected]>

Refactor coordinator to support create and delete index events when different index types exist on cluster (opensearch-project#272)

* Refactor coordinator to support different index type create and delete events and sweep

Signed-off-by: Ravi Thaluru <[email protected]>

introducing custom uuid setting in SPI (opensearch-project#278)

* introducing custom uuid setting in SPI

Signed-off-by: Ravi Thaluru <[email protected]>

Refactors the managed index runner to work with extensions (opensearch-project#262)

* Adds index creation date

Signed-off-by: Robert Downs <[email protected]>

Adds NewClusterEventListeners and refactors Transition to work with custom actions

Signed-off-by: Robert Downs <[email protected]>

Marks blocked actions list as deprecated

Signed-off-by: Clay Downs <[email protected]>

Asserts the deprecation warning after adding to allow list in test

Signed-off-by: Robert Downs <[email protected]>

Removes allow_list test

Signed-off-by: Robert Downs <[email protected]>

* Fix additional rebase issues

Signed-off-by: Robert Downs <[email protected]>

* Fixes failing tests

Signed-off-by: Robert Downs <[email protected]>

* changes based on comments

Signed-off-by: Robert Downs <[email protected]>

* Adds additional comments

Signed-off-by: Clay Downs <[email protected]>

Makes rest APIs use new metadata service (opensearch-project#245)

* Makes rest API use new metadata service

Signed-off-by: Robert Downs <[email protected]>

Adds missed flags (opensearch-project#286)

Signed-off-by: Clay Downs <[email protected]>

Ports over show-applied-policies logic (opensearch-project#287)

Signed-off-by: Clay Downs <[email protected]>
@downsrob downsrob marked this pull request as ready for review March 8, 2022 19:02
@downsrob downsrob requested review from a team, bowenlan-amzn and dbbaughe and removed request for a team March 8, 2022 19:02
@downsrob downsrob merged commit 2d3fd97 into opensearch-project:main Mar 9, 2022
downsrob added a commit to downsrob/index-management that referenced this pull request Mar 9, 2022
downsrob added a commit to downsrob/index-management that referenced this pull request Mar 9, 2022
…t#288)

* Adding IM SPI (opensearch-project#216)

Signed-off-by: Robert Downs <[email protected]>

Co-authored-by: Ravi <[email protected]>
Signed-off-by: Robert Downs <[email protected]>
downsrob added a commit that referenced this pull request Mar 9, 2022
* Adding IM SPI (#216)

Signed-off-by: Robert Downs <[email protected]>

Co-authored-by: Ravi <[email protected]>
Signed-off-by: Robert Downs <[email protected]>
wuychn pushed a commit to ochprince/index-management that referenced this pull request Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants