[CI] ManyShardsIT testConcurrentQueries failing

[kingherc]

Build scan:
https://gradle-enterprise.elastic.co/s/mshv632yq64jg/tests/:x-pack:plugin:esql:internalClusterTest/org.elasticsearch.xpack.esql.action.ManyShardsIT/testConcurrentQueries

Reproduction line:

./gradlew ':x-pack:plugin:esql:internalClusterTest' --tests "org.elasticsearch.xpack.esql.action.ManyShardsIT.testConcurrentQueries" -Dtests.seed=6315C2A6759430E2 -Dtests.locale=et -Dtests.timezone=MST7MDT -Druntime.java=18

Applicable branches:
8.12

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.xpack.esql.action.ManyShardsIT#testConcurrentQueries

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=84, name=Thread-16, state=RUNNABLE, group=TGRP-ManyShardsIT]

  at __randomizedtesting.SeedInfo.seed([6315C2A6759430E2:218E2183F5F345C3]:0)

  Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of TimedRunnable{original=processing of [1302][indices:data/read/esql/compute]: org.elasticsearch.compute.operator.DriverTaskRunner$DriverRequest/zaPy3ZDgTMa4KptMG4wbsA:576, creationTimeNanos=850520350585, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = node_s1/esql, queue capacity = 1000, task execution EWMA = 74.1micros, total task execution time = 1.1s, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@6b99eb0b[Running, pool size = 1, active threads = 1, queued tasks = 1000, completed tasks = 162]]

    at __randomizedtesting.SeedInfo.seed([6315C2A6759430E2]:0)
    at org.elasticsearch.common.util.concurrent.EsRejectedExecutionHandler.newRejectedException(EsRejectedExecutionHandler.java:40)
    at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:34)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
    at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:72)
    at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:1025)
    at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:143)
    at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:952)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:847)
    at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:911)
    at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:888)
    at org.elasticsearch.compute.operator.DriverTaskRunner$1.start(DriverTaskRunner.java:49)
    at org.elasticsearch.compute.operator.DriverRunner.runToCompletion(DriverRunner.java:90)
    at org.elasticsearch.compute.operator.DriverTaskRunner.executeDrivers(DriverTaskRunner.java:65)
    at org.elasticsearch.xpack.esql.plugin.ComputeService.runCompute(ComputeService.java:321)
    at org.elasticsearch.xpack.esql.plugin.ComputeService$DataNodeRequestHandler.lambda$messageReceived$5(ComputeService.java:509)
    at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178)
    at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:306)
    at org.elasticsearch.xpack.esql.plugin.ComputeService.lambda$acquireSearchContexts$18(ComputeService.java:349)
    at org.elasticsearch.index.shard.IndexShard.ensureShardSearchActive(IndexShard.java:3967)
    at org.elasticsearch.xpack.esql.plugin.ComputeService.acquireSearchContexts(ComputeService.java:347)
    at org.elasticsearch.xpack.esql.plugin.ComputeService$DataNodeRequestHandler.messageReceived(ComputeService.java:507)
    at org.elasticsearch.xpack.esql.plugin.ComputeService$DataNodeRequestHandler.messageReceived(ComputeService.java:498)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:288)
    at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:301)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(Thread.java:833)

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[mark-vieira]

Got another one. This happens occassionally across main, 8.12 and 8.11 branches.

https://gradle-enterprise.elastic.co/s/addygp5ihbif6/tests/:x-pack:plugin:esql:internalClusterTest/org.elasticsearch.xpack.esql.action.ManyShardsIT/testConcurrentQueries

[dnhatn]

This test no longer fails since we introduced batching in #104832.