Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Close Opensearch RestHighLevelClient in OpenSearchClientRefresher on shutdown and initialization failure #4770

Closed
dinujoh opened this issue Jul 29, 2024 · 2 comments
Labels
bug Something isn't working
Milestone

Comments

@dinujoh
Copy link
Member

dinujoh commented Jul 29, 2024

Describe the bug
When the data prepper pipeline Opensearch Sink initialization fails, data prepper continuously retries to recreate Opensearch client and OpenSearchClientRefresher (refresh credentials). Looks like on initialization failure, we only close Opensearch client and not OpenSearchClientRefresher (this has another reference to Opensearch client). These unclosed Opensearch clients are consuming threads and are resulting in OOM.

To Reproduce
If the Opensearch Sink credentials or data/network access policy is not configured correctly, data prepper continuously retries to recreate Opensearch client. Below OOM exception is seen in the logs.

 ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [cdc-pipeline] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]

Expected behavior
The Opensearch Sink/Source should gracefully close the opened connection.

Code Reference
Should add close method to OpenSearchClientRefresher to close OpenSearchClient and call it here.

@dinujoh dinujoh added bug Something isn't working untriaged labels Jul 29, 2024
@dinujoh
Copy link
Member Author

dinujoh commented Jul 30, 2024

#4758

@dlvenable dlvenable added this to the v2.9 milestone Jul 30, 2024
@dlvenable
Copy link
Member

@dinujoh , Do you think that #4758 will resolve this? Is there anything missing that you see?

@dlvenable dlvenable modified the milestones: v2.9, v2.10 Aug 26, 2024
@dinujoh dinujoh closed this as completed Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants