Getting "possibly out of memory or process/resource limits reached" followed by "BoundedExecutor is in a failed state" after upgrading to version 446

[sravyanu-kore]

Hi Team,

We are getting "possibly out of memory or process/resource limits reached" followed by "BoundedExecutor is in a failed state" after upgrading to version 446. Our previous version was 430. Downgrading to 430 version is solving the issue. However, we are getting these errors once we upgrade to 446 in our higher environments. Any help is appreciated.

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:1553)
at java.base/java.lang.System$2.start(System.java:2577)
at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1375)
at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:640)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:73)
at io.trino.execution.executor.scheduler.FairScheduler.submit(FairScheduler.java:155)
at io.trino.execution.executor.dedicated.TaskEntry.runSplit(TaskEntry.java:136)
at io.trino.execution.executor.dedicated.ThreadPerDriverTaskExecutor.enqueueSplits(ThreadPerDriverTaskExecutor.java:158)
at io.trino.execution.SqlTaskExecution.enqueueDriverSplitRunner(SqlTaskExecution.java:402)
at io.trino.execution.SqlTaskExecution.scheduleDriversForTaskLifeCycle(SqlTaskExecution.java:391)
at io.trino.execution.SqlTaskExecution.start(SqlTaskExecution.java:207)
at io.trino.execution.SqlTask.tryCreateSqlTaskExecution(SqlTask.java:572)
at io.trino.execution.SqlTask.updateTask(SqlTask.java:514)
at io.trino.execution.SqlTaskManager.doUpdateTask(SqlTaskManager.java:558)
at io.trino.execution.SqlTaskManager.lambda$updateTask$9(SqlTaskManager.java:490)
at io.trino.$gen.Trino_446____20240921_071353_2.call(Unknown Source)

ERROR http-worker-713220 com.google.common.util.concurrent.AbstractFuture RuntimeException while executing runnable CallbackListener{io.airlift.jaxrs.AsyncResponseHandler$1@57de90b0} with executor io.airlift.concurrent.BoundedExecutor@3311db54
java.util.concurrent.RejectedExecutionException: BoundedExecutor is in a failed state
at io.airlift.concurrent.BoundedExecutor.execute(BoundedExecutor.java:55)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:762)
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.addListener(FluentFuture.java:114)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1103)
at io.airlift.jaxrs.AsyncResponseHandler.bindAsyncResponse(AsyncResponseHandler.java:49)
at io.trino.server.TaskResource.getResults(TaskResource.java:351)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:263)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)

[sriharshaj]

It can be related to this:
https://trinodb.slack.com/archives/CGB0QHWSW/p1721671769818799

You may want to try 454, and see if it fixes the issue.

[hashhar]

unable to create native thread: possibly out of memory or process/resource limits reached

This suggests you have reached nproc limits possibly. What does ulimits -a return for the user as which Trino process is running?