Quarkus v3 gRPC server error with big requests and Quarkus v2 client

[jdussouillez]

Describe the bug

I'm trying to upgrade my projects to Quarkus v3+ but I'm facing an issue when calling a Quarkus v3 gRPC server using a Quarkus v2 gRPC client.
The calls with small data work fine, but if I send more than ~64KB (in one big message or multiple small ones) then the connection is closed.

This works fine when both projects are using Quarkus v3 or v2, but not when the server is in v3 and client in v2. Which can be a problem because we are migrating our projects step by step.

Expected behavior

No connection lost when calling a v3 gRPC server with a v2 client.

Actual behavior

• If I send one big message in a gRPC stream that is bigger than ~64KB, it fails.
• If I send a lot of small messages in a gRPC stream but the sum of all messages are greater or equals than ~64KB (let's say 20 messages of 5KB each), it fails.

With a single message of 65526 bytes (not 65536, 65526) ✔️

Sending gRPC request of size 65526
gRPC server result: [content: "I\'m your PDF content"
]

With a single message of 65527 bytes 🔴

Sending gRPC request of size 65527
2023-05-17 13:45:37,597 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): io.vertx.core.http.HttpClosedException: Connection was closed

See the logs in the project to reproduce the bug: https://github.com/jdussouillez/quarkus-grpc-req-memory-bug#client

How to Reproduce?

Project to reproduce: https://github.com/jdussouillez/quarkus-grpc-req-memory-bug

Output of uname -a or ver

Linux frpc245 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "17.0.7" 2023-04-18 OpenJDK Runtime Environment (build 17.0.7+7-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 17.0.7+7-Ubuntu-0ubuntu120.04, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

• server: 3.0.3.Final
• client: 2.16.7.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.6

Additional information

No response

[quarkus-bot]

/cc @alesj (grpc), @cescoffier (grpc)

[alesj]

Are you sure it works with the same versions?
Since we haven't changed that much (if at all) between the 2 version you listed.
Afair, only small changes / fixes and that Jakarta overhaul.

[jdussouillez]

@alesj Yes it seems to be working when both server and client have the same Quarkus versions (v2 or v3).

$ java -jar target/quarkus-app/quarkus-run.jar "ko-1r"
# OK
$ java -jar target/quarkus-app/quarkus-run.jar "ko-xr"
# OK

I created branches on the test project :

• client-v3 - Client in v3 (so both are in v3)
• server-v2 - Server in v2 (so both are in v2)
• master is server v3 and client v2

[alesj]

Uh, so you're saying that it works up till 65526 (or is it 65536 = 256^2 = 2^16 ?) -- the mixed version?
And then it fails ...

[jdussouillez]

Yes I know it sounds weird, I went insane for a few days trying to debug and understand this.

The limit is not the same when I send a single message (65526 bytes) or multiple messages. My first guess is the limit is 65536 but there are some headers or meta data that takes some space, so the message body content limit is a bit less than 65536.

[jdussouillez]

With multiple messages on the same stream, the values are different but I have the same kind of error:

If I use Quarkus v3 in the client, it works fine:

[alesj]

Yes I know it sounds weird, I went insane for a few days trying to debug and understand this.

The limit is not the same when I send a single message (65526 bytes) or multiple messages. My first guess is the limit is 65536 but there are some headers or meta data that takes some space, so the message body content limit is a bit less than 65536.

So this one first works and then on the 2nd try it fails?

[jdussouillez]

So this one first works and then on the 2nd try it fails?

It's not a 2nd try, it's a different call with a bigger file (option ok-1r means OK, 1 request, it's a file of 65526 bytes, ko-1r means KO/failure, 1 request, it's a file of 65527 bytes).

I start the client with ko-1r (so using a file with one 65527 bytes), it fails with Connection was closed.

The server is not shutdown after the first call. I've made hundreds of tests, I can call the server with my clients multiple times in a row with a small file (<= 65526). When I send > 65526 bytes, it fails. Then I can start again to send small requests and it works. Without restarting the server.

[jdussouillez]

I migrated all my projects to Quarkus v3 and no similar issues.

If nobody else has the problem, let's sweep it under the carpet. Maybe just add it in the documentation and let's move on.

[cescoffier]

Closing. See comment.