Add create component extension point support for AD

[saratvemulapalli]

With AD detectors, lets add support for create components and integrate with the work already done with #44

[joshpalis]

createComponents allows plugins to return a collection of objects to OpenSearch. PluginService iterates through all plugins and passes several services and registries, ie clusterService, environment, client, NamedXContentRegistry to help these plugins to configure and generate their components. These objects are then consolidated into a list, which is then bound to a Guice module during node initialization and then further used for dependency injection.

In the case of extensibility, since these extensions are running on independent processes, we must add support to transport the necessary information from the services required to successfully create these components.

For Anomaly Detection, in order to support creating a detector, the required components are AnomalyDetectionIndices, SearchFeatureDao, and ADTaskManager.

These components require several services from OpenSearch, and currently there is added support for the following :

• Client : replaced by SDKClient

Some of the required services from OpenSearch are not necessary to add support for as they are not required for creating a detector :

• NamedXContentRegistry : replaced with an empty collection

These objects require transport APIs to be created in order to access :

• ClusterService
• Settings

ClusterService is used by Anomaly Detection to access the ClusterSettings in order to add custom settings update consumers. The purpose of these setting consumers is to perform actions defined by the extension whenever a particular setting is updated, usually by updating the field corresponding to the setting with the data passed by the consumer. These custom consumers are added within the object constructors of each component, so it is necessary to enable extensions to perform these actions as part of component creation.

For example :

• For ADTaskManager : https://github.com/joshpalis/anomaly-detection/blob/572fd8d4f08691326d92535f88f7d5f06ea4ea63/src/main/java/org/opensearch/ad/task/ADTaskManager.java#L233
• For AnomalyDetectionIndices : https://github.com/joshpalis/anomaly-detection/blob/572fd8d4f08691326d92535f88f7d5f06ea4ea63/src/main/java/org/opensearch/ad/indices/AnomalyDetectionIndices.java#L201
• For SearchFeatureDao : https://github.com/joshpalis/anomaly-detection/blob/572fd8d4f08691326d92535f88f7d5f06ea4ea63/src/main/java/org/opensearch/ad/feature/SearchFeatureDao.java#L117

To add support for extensions to add custom settings update consumers during create components :

• Create a component settings consumer map within extension runner of Setting objects and their setting update consumers, which components will add to as part of their constructors
• Create a Transport Request from extensions to OpenSearch to add custom settings update consumers. The request parameters will include a List of settings for each component, and the DiscoveryExtension to which the consumer callback method shall point to
• Use ExtensionsOrchestrator's clusterService, which is initialized here and invoke getClusterSettings.addSettingsUpdateConsumer for each transported Setting, and set each corresponding Consumer to send a UpdateSettingsRequest back to the DiscoveryExtension node along with the Setting object, Setting type, and data.
• Send an ExtensionBooleanResponse as an acknowledgement to the extension upon successfully registering the setting update consumer within ClusterService's ClusterSettings
• When a setting is updated, the setting updater that is registered within ClusterSettings validates the setting change and invokes the consumer. This consumer transports an UpdateSettingsRequest to the extension, which receives the request and cross-references the request's Setting with the corresponding Setting within the component setting consumer map
• Once the corresponding consumer is identified, determine the setting type, cast the request data and corresponding consumer to the setting type and invoke the consumer via consumer.accept(data) to update the setting
• Upon successfully updating the setting, the extension sends an ExtensionBooleanResponse back to Opensearch as an acknowledgement

Settings come from the Environment, which is created here . Extensions use environment settings to retrieve values (or default values if the setting is not present in the provided Settings object) in order to set fields during component initialization. To add support for extensions :

Previous Design for Environment Settings

• Pass in environment settings from Node.java to ExtensionsOrchestrator during node initialization
• Create a Transport Request from extensions to OpenSearch for environment settings

New Design for Environment Settings

Upon further discussion, support for environment settings will change. Rather than transporting all the OpenSearch environment settings to extensions, extensions will send OpenSearch a list of Setting objects that it would want to retrieve values from. The following workflow will achieve this support :

• Pass in environment settings from Node.java to ExtensionsOrchestrator during node initialization
• During createComponents, for each component, the Extension will send a request to OpenSearch containing a List of Setting objects
• OpenSearch will receive this list and invoke Setting<T>.get(environmentSettings) to retrieve the corresponding values (or default value if the setting object is not yet registered) and map them to the setting
• OpenSearch will respond with the Map<Setting, value> of which the SDK will maintain (lets name this map ComponentSettingValues)
• The extension will then invoke ComponentSettingsValue.get(Setting) to retrieve the value of the setting from the OpenSearch environment

Once these transport APIs are created, integrating the opensearch-sdk with Anomaly Detection would allow us call these transport APIs to retrieve the necessary information to generate the components needed for creating a detector.

[joshpalis]

For future reference, the create-component workflow for extensions will utilize these new APIs as follows :

1. Previously, components initialize fields with Setting<T>.get(settings). Instead, each component will first compile these settings into a List<Setting<T>> and invoke sendEnvironmentSettingsRequest(transportService, List<Setting<T>>) , which will respond with a Map<Setting<T>, Object>. The fields should then be initialized with the corresponding object.
2. Previously, components then added settings consumers via clusterService.getClusterSettings().addSettingUpdateConsumer(Setting<T>, Consumer<T>) . Instead of this, components will consolidate these Setting<?> and Consumer<?> objects into a map, then invoke sendAddSettingsUpdateConsumerRequest(transportService, Map<Setting<T>, Consumer<T>>) . This does two things. It registers these settings and consumers within the UpdateSettingsRequestHandler and then extracts the keyset of the map and transports this list to the AddSettingsUpdateConsumerRequest to be registered in OpenSearch with a callback method to the extension. This ensures that requests to add setting update consumers within OpenSearch will be sent for each component, but the full map of component settings and update consumers will be available within theUpdateSettingsRequestHandlerwhenever an UpdateSettingsRequest is sent from OpenSearch