[META] Build OpenSearch SDK for extensions

[owaiskazi19]

Is your feature request related to a problem? Please describe.
Currently plugins are tightly coupled with OpenSearch making it complex to independently ship enhancements. Running plugin as a separate process will uncover advantages like dynamic loading of plugins and OpenSearch cluster not getting affected even if plugin fails.

1. Startup script to run TransportService. Startup script to initialize TransportService #1646
2. Add support of TransportService for extension . Add transport support to connect with opensearch service for SDK #2020
3. Integrate Transport Service with an external plugin. Integrate Transport Service with an external plugin #2174
4. Send a TransportRequest to the SDK and make sure communication between SDK and OpenSearch is established - SDK: Add support for communication between SDK and OpenSearch  #2351
5. Invoke the extension points: Invoke the extension point based on the request from OpenSearch to unblock Latency Testing #2691
6. Move the dependencies from SDK to OpenSearch - Move the dependencies from SDK to OpenSearch and add CI, Spotless #2297
7. Implement TransportRequest for the extension points on SDK - [SDK] Add support for createComponent extension points #2981
8. Run AD plugin as an Extension to create a detector - [Feature/extensions] Run AD backend plugin as an Extension to create a detector #3635
9. Implement APIs to invoke extension points in SDK - [Extensions]Implement APIs to invoke extension points in SDK #3078
10. Create a workflow to invoke the listeners of the extension points - [Extensions]Create a workflow to invoke the listeners of the extension points #3079
11. Create extension.yml - Create an extension.yml file to provide the plugin description #2693
    11. Call Overridden functions like getRepositories() -
    https://github.com/opensearch-project/OpenSearch/blob/main/plugins/repositorys3/src/main/java/org/opensearch/repositories/s3/S3RepositoryPlugin.java#L103
    12. Integrate Extension with SDK and Remove pluginsService dependencies from repository-s3 plugin. POC: Integrate SQL plugin with SDK #1695
12. Integration and Unit Tests for SDK #2417

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[Bukhtawar]

Would be great to see the evaluation of advantages and downsides of running plugins as independent processes. Would this mean memory commitments in general? Plugins also interact with the core and has listeners/interceptors configured which is cheaper in-process. Is the plan for specific plugins or in-general all plugins

[saratvemulapalli]

Thanks @Bukhtawar for your thoughts.
Yes we will be looking pro's and con's for having an independent plugin.

Please take a look at #1422. There is a high level proposal listed on the issue. We are trying to achieve plugin sandboxing but started with a POC with independent plugins. If you have any thoughts/suggestions let us know.

[Bukhtawar]

Looks like we are doing a POC at this point, would be great to see few comparisons

1. Overall memory overhead
2. Deployment overhead
3. Feature sandboxing/isolation
4. IPC or remote node communication

[saratvemulapalli]

Looks like we are doing a POC at this point, would be great to see few comparisons

1. Overall memory overhead
2. Deployment overhead
3. Feature sandboxing/isolation
4. IPC or remote node communication

Absolutely. We are still working on the POC and in this order:

1. IPC or remote node communication
2. Feature sandboxing/isolation
3. Perfomance analysis
4. Overall memory overhead
5. Deployment overhead

But thanks for the suggestions.