You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When the data prepper pipeline Opensearch Sink initialization fails, data prepper continuously retries to recreate Opensearch client and OpenSearchClientRefresher (refresh credentials). Looks like on initialization failure, we only close Opensearch client and not OpenSearchClientRefresher (this has another reference to Opensearch client). These unclosed Opensearch clients are consuming threads and are resulting in OOM.
To Reproduce
If the Opensearch Sink credentials or data/network access policy is not configured correctly, data prepper continuously retries to recreate Opensearch client. Below OOM exception is seen in the logs.
ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [cdc-pipeline] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
Expected behavior
The Opensearch Sink/Source should gracefully close the opened connection.
Describe the bug
When the data prepper pipeline Opensearch Sink initialization fails, data prepper continuously retries to recreate Opensearch client and OpenSearchClientRefresher (refresh credentials). Looks like on initialization failure, we only close Opensearch client and not OpenSearchClientRefresher (this has another reference to Opensearch client). These unclosed Opensearch clients are consuming threads and are resulting in OOM.
To Reproduce
If the Opensearch Sink credentials or data/network access policy is not configured correctly, data prepper continuously retries to recreate Opensearch client. Below OOM exception is seen in the logs.
Expected behavior
The Opensearch Sink/Source should gracefully close the opened connection.
Code Reference
Should add close method to OpenSearchClientRefresher to close OpenSearchClient and call it here.
The text was updated successfully, but these errors were encountered: