@Blocking on a standard grpc service run on a single thread

[nahguam]

Describe the bug

If I implement a standard grpc service (not mutiny) and annotate it with @Blocking then requests are correctly dispatched to the core thread pool instead of the event loop. However, requests are always dispatched to the same core thread. This results in effectively only being able to process 1 request at a time.

I've tried increasing quarkus.thread-pool.core-threads but it is having no effect.

Expected behavior

Requests are dispatched to a new/free threads.

Actual behavior

Requests are dispatched to the same thread and queued.

How to Reproduce?

Create grpc starter app

Add proto:

syntax = "proto3";

option java_multiple_files = true;
option java_package = "org.acme";

package hello;

service StandardBlockingGrpcService {
  rpc Invoke (Request) returns (Response) {}
}

message Request {
  int32 id = 1;
}

message Response {
  int32 id = 1;
}

Add service implementation:

class GrpcServices {
  private static final Logger logger = LoggerFactory.getLogger(GrpcServices.class);

  @GrpcService
  @Blocking
  static class StandardBlocking extends StandardBlockingGrpcServiceImplBase {
    @Override
    public void invoke(Request request, StreamObserver<Response> responseObserver) {
      logger.info("start: {}", request.getId());
      try {
        Thread.sleep(Duration.ofSeconds(2).toMillis());
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        throw new RuntimeException(e);
      }
      responseObserver.onNext(Response.newBuilder().setId(request.getId()).build());
      responseObserver.onCompleted();
      logger.info("end: {}", request.getId());
    }
  }
}

Add test:

@QuarkusTest
class GrpcServicesTest {
  private static final Logger logger = LoggerFactory.getLogger(GrpcServicesTest.class);

  @GrpcClient
  StandardBlockingGrpcServiceBlockingStub standardBlocking;

  ExecutorService executor = Executors.newCachedThreadPool();

  @Test
  void standardBlockingTest() {
    var futures = IntStream.range(0, 5)
        .mapToObj(i -> CompletableFuture.runAsync(() -> standardBlocking(i), executor))
        .toArray(CompletableFuture[]::new);
    CompletableFuture.allOf(futures).join();
  }

  void standardBlocking(int id) {
    logger.info("start: {}", id);
    standardBlocking.invoke(Request.newBuilder().setId(id).build());
    logger.info("end: {}", id);
  }
}

Log output:

2024-04-19 12:53:53,209 WARN  [io.qua.grp.run.GrpcServerRecorder] (main) Using legacy gRPC support, with separate new HTTP server instance. Switch to single HTTP server instance usage with quarkus.grpc.server.use-separate-server=false property
2024-04-19 12:53:53,236 WARN  [io.qua.virtual-threads] (vert.x-eventloop-thread-0) You weren't able to create an executor that spawns virtual threads, the default blocking executor will be used, please check that your JDK is compatible with virtual threads
2024-04-19 12:53:53,273 INFO  [io.qua.grp.run.GrpcServerRecorder] (vert.x-eventloop-thread-1) Started gRPC server on 0.0.0.0:9001 [TLS enabled: false]
2024-04-19 12:53:53,294 INFO  [io.quarkus] (main) code-with-quarkus 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.6.9) started in 0.826s. Listening on: http://localhost:8081
2024-04-19 12:53:53,295 INFO  [io.quarkus] (main) Profile test activated. 
2024-04-19 12:53:53,295 INFO  [io.quarkus] (main) Installed features: [cdi, grpc-client, grpc-server, smallrye-context-propagation, vertx]
2024-04-19 12:53:53,300 INFO  [io.qua.grp.run.sup.Channels] (main) gRPC client standardBlocking created without configuration. We are assuming that it's created to test your gRPC services.
2024-04-19 12:53:53,304 INFO  [io.qua.grp.run.sup.Channels] (main) Creating Netty gRPC channel ...
2024-04-19 12:53:53,319 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-5) start: 4
2024-04-19 12:53:53,319 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-2) start: 1
2024-04-19 12:53:53,319 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-3) start: 2
2024-04-19 12:53:53,319 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-4) start: 3
2024-04-19 12:53:53,319 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-1) start: 0
2024-04-19 12:53:53,389 INFO  [org.acm.GrpcServices] (executor-thread-4) start: 4
2024-04-19 12:53:55,396 INFO  [org.acm.GrpcServices] (executor-thread-4) end: 4
2024-04-19 12:53:55,397 INFO  [org.acm.GrpcServices] (executor-thread-4) start: 1
2024-04-19 12:53:55,400 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-5) end: 4
2024-04-19 12:53:57,400 INFO  [org.acm.GrpcServices] (executor-thread-4) end: 1
2024-04-19 12:53:57,400 INFO  [org.acm.GrpcServices] (executor-thread-4) start: 3
2024-04-19 12:53:57,403 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-2) end: 1
2024-04-19 12:53:59,404 INFO  [org.acm.GrpcServices] (executor-thread-4) end: 3
2024-04-19 12:53:59,405 INFO  [org.acm.GrpcServices] (executor-thread-4) start: 0
2024-04-19 12:53:59,407 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-4) end: 3
2024-04-19 12:54:01,409 INFO  [org.acm.GrpcServices] (executor-thread-4) end: 0
2024-04-19 12:54:01,409 INFO  [org.acm.GrpcServices] (executor-thread-4) start: 2
2024-04-19 12:54:01,410 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-1) end: 0
2024-04-19 12:54:03,413 INFO  [org.acm.GrpcServices] (executor-thread-4) end: 2
2024-04-19 12:54:03,415 INFO  [org.acm.GrpcServicesTest] (pool-8-thread-3) end: 2
2024-04-19 12:54:03,438 INFO  [io.qua.grp.run.sup.Channels] (main) Shutting down gRPC channel ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=4, target=dns:///0.0.0.0:9001}}
2024-04-19 12:54:03,446 INFO  [io.quarkus] (main) code-with-quarkus stopped in 0.019s

Output of uname -a or ver

Darwin -MacBook-Pro-16-inch-2023- 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6020 arm64

Output of java -version

openjdk version "11.0.19" 2023-04-18 OpenJDK Runtime Environment Temurin-11.0.19+7 (build 11.0.19+7) OpenJDK 64-Bit Server VM Temurin-11.0.19+7 (build 11.0.19+7, mixed mode)

Quarkus version or git rev

3.6.9

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae) Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec Java version: 11.0.19, vendor: Eclipse Adoptium, runtime: /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home Default locale: en_GB, platform encoding: UTF-8 OS name: "mac os x", version: "13.6", arch: "aarch64", family: "mac"

Additional information

No response

[quarkus-bot]

/cc @alesj (grpc), @cescoffier (grpc)

[cescoffier]

Thanks for reporting. I've implemented a fix in #40173

[nahguam]

Thanks, will this be backported to 3.6? (We're stuck on Java 11)

[jbonofre]

I think it should be backported to 3.6 as it's impactful issue. @cescoffier thoughts ?

[cescoffier]

A bit of hardening time is needed before being considered for a backport.

There are workarounds (using a reactive messaging and dealing with the blocking part yourself)

[jbonofre]

@cescoffier do you want us to do a new pass on this ?

[nahguam]

Thanka @cescoffier . Yes, this what we have done for now.

[gsmet]

I think it should be backported to 3.6 as it's impactful issue. @cescoffier thoughts ?

3.6 is not a supported version. We won't publish any new 3.6 (we announced it quite some time ago already).

[gsmet]

And while this particular issue might affect you, I would be more worried about the security fixes: 3.6 doesn't have any security fix applied.

You can either use 3.2 LTS, 3.8 LTS or latest.
I'm not entirely sure we would take the risk to backport this patch on 3.2 though. Definitely not without sufficient bake time as mentioned by @cescoffier .

[jbonofre]

@gsmet yeah, that's ok, thanks. I'm just wondering which LTS supports Java11 (that's why we are using Quarkus 3.6.x). We can't upgrade to 3.8.x as we are not Java17 ready yet.

[cescoffier]

It's 3.2, and no, I don't believe we will backport this to 3.2. Lots of things have changed internally, I'm not sure it can be backported as it is.

[jbonofre]

Yeah, 3.2 is not an option for us as we need others fixes and improvements.