Quarkus version change from 2.10.1.Final to 2.11.1.Final has breaking changes with Percona SqlProxy #27386

harpermj66 · 2022-08-19T15:29:04Z

Describe the bug

Quarkus with hibernate reactive and mysql database configured with a version to 2.11.1 version of quarkus allowed data retrieval from Percona SqlProxy.
It worked very well in fact.
As of 2.11.1 I'm not able to retrieve data.

Expected behavior

Executing queries writen using hibernate reactive with panache should continue to function as they did in version 2.10.1

Actual behavior

a) on the client Failed to execute statement [select terminalen0_.TERMINAL_ID as terminal1_22_, terminalen0_.COMPANY_ID as company_2_22_, terminalen0_.NAME as name3_22_, terminalen0_.UUID as uuid4_22_ from TERMINAL terminalen0_ where TERMINAL_ID=?]: could not execute query: java.util.concurrent.CompletionException: io.vertx.core.impl.NoStackTraceThrowable: Pending requests failed to be sent due to connection has been closed.

b) on SqlProxy itself

2022-08-19 15:20:29 MySQL_Session.cpp:3364:handler___status_NONE_or_default(): [ERROR] Unexpected packet from client 127.0.0.1 . Session_status: 21 , client_status: 0 Disconnecting it
2022-08-19 15:20:29 MySQL_Session.cpp:149:kill_query_thread(): [WARNING] KILL CONNECTION 1254813 on 10.2.5.21:3306

How to Reproduce?

configure a mysql cluster with a Percona SqlProxy
configure a quarkus test to retrieve data via reactive hibernate panache
use verstion 2.10.1 of quarkus to verify the data is returned.
upgrade to version 2.11.1 and observer the test will error as above.

Output of `uname -a` or `ver`

Linux locus-dev01 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Output of `java -version`

openjdk version "11.0.16" 2022-07-19 OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.11.1

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

Only breaks when going through the percona SqlProxy

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2022-08-19T21:37:09Z

/cc @DavideD, @Sanne, @gavinking

gastaldi · 2022-08-22T15:49:24Z

Can you provide a test project reproducing the error? Also, as I have never used Percona SqlProxy, maybe a Docker compose file can help in setting up the environment to reproduce this issue

harpermj66 · 2022-08-22T15:56:54Z

I would be extremely difficult for me to provide a test project that would have 3 mysql servers in a cluster and another 3 percona sql proxies. The setup of a proxy itself is no walk in the park.

What about taking the what did we change between versions route? I have feeling it may just be a configuration default that we may be able to test reverting.

Sanne · 2022-08-22T16:08:07Z

We use the Reactive Client drivers from Vert.x - the most likely "change" was a version upgrade in these drivers.

Do you think you could test by overriding the vert.x version for client? Unfortunately the coupling Quarkus/Vert.x is quite high so it might not be possible to get a functioning application with a different driver, but provided you're only changing some micro versions it might work well enough to narrow this problem down.

cc/ @tsegismont

harpermj66 · 2022-08-22T16:12:18Z

On 22 Aug 2022, at 17:08, Sanne Grinovero ***@***.***> wrote: We use the Reactive Client drivers from Vert.x - the most likely "change" was a version upgrade in these drivers. Do you think you could test by overriding the vert.x version for client? Unfortunately the coupling Quarkus/Vert.x is quite high so it might not be possible to get a functioning application with a different driver, but provided you're only changing some micro versions it might work well enough to narrow this problem down. cc/ @tsegismont <https://github.com/tsegismont> — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKVBAOWGWXMDJIMNMSMTEFDV2OQXDANCNFSM57BD4YQA>. You are receiving this because you authored the thread.

I’ll see if I can do that. What’s the easiest way to do an override the vert.x version? Just in the build file itself?

Sanne · 2022-08-22T16:19:23Z

In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence:

  
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.vertx</groupId>
                <artifactId>vertx-mysql-client</artifactId>
                <version>something-something</version>
            </dependency>
            <dependency>
            <groupId>io.quarkus</groupId>
                <artifactId>quarkus-bom</artifactId>
                <version>${quarkus.platform.version}</version>
                <scope>import</scope>
                <type>pom</type>
            </dependency>

harpermj66 · 2022-08-22T16:59:06Z

Sorry I took me some time because we are on Gradle. implementation("io.vertx:vertx-mysql-client:4.3.0"){ isForce = true } Seems to be the culprit

…

On 22 Aug 2022, at 17:19, Sanne Grinovero ***@***.***> wrote: In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence: <dependencyManagement> <dependencies> <dependency> <groupId>io.vertx</groupId> <artifactId>vertx-mysql-client</artifactId> <version>something-something</version> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-bom</artifactId> <version>${quarkus.platform.version}</version> <scope>import</scope> <type>pom</type> </dependency> — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKVBAOQEXM3GKCGHPVWN5PDV2OSBNANCNFSM57BD4YQA>. You are receiving this because you authored the thread.

tsegismont · 2022-08-23T12:54:38Z

This might be related to pipelining (eclipse-vertx/vertx-sql-client#1168).

I am trying to reproduce atm

tsegismont · 2022-08-23T16:35:26Z

I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved:

with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received
with Quarkus 2.10.2, the client waits for response ok packet before sending close packet

harpermj66 · 2022-08-23T18:01:36Z

Great that you found a reason, haven’t got clue obviously how it can be changed/fixed if it’s broken or sql proxy just can’t cope - Lol

…

On 23 Aug 2022, at 17:35, Thomas Segismont ***@***.***> wrote: I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved: with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received with Quarkus 2.10.2, the client waits for response ok packet before sending close packet — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA>. You are receiving this because you authored the thread.

tsegismont · 2022-08-24T08:32:22Z

I'm trying to figure out a workaround for now. I'll get in touch with MySQL proxy community. Le mar. 23 août 2022 à 20:01, harpermj66 ***@***.***> a écrit :

…

Great that you found a reason, haven’t got clue obviously how it can be changed/fixed if it’s broken or sql proxy just can’t cope - Lol > On 23 Aug 2022, at 17:35, Thomas Segismont ***@***.***> wrote: > > > I was able to reproduce. I captured network traffic and, indeed, pipelining seems involved: > > with Quarkus 2.12.2, the client sends execute and close packets immediately after prepare response packet is received > with Quarkus 2.10.2, the client waits for response ok packet before sending close packet > — > Reply to this email directly, view it on GitHub < #27386 (comment)>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA >. > You are receiving this because you authored the thread. > — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALOLNWHBKGBKVXBX2OKZGLV2UGYXANCNFSM57BD4YQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

harpermj66 · 2022-08-24T08:39:20Z

I reached out to Percona because we have support with them initially. They did say we were their first client to mention reactive hibernate so they were keen to set up a meeting. I can ask my contact at percona to link up with you if necessary.

…

On 24 Aug 2022, at 09:32, Thomas Segismont ***@***.***> wrote: I'm trying to figure out a workaround for now. I'll get in touch with MySQL proxy community. Le mar. 23 août 2022 à 20:01, harpermj66 ***@***.***> a écrit : > Great that you found a reason, haven’t got clue obviously how it can be > changed/fixed if it’s broken or sql proxy just can’t cope - Lol > > > On 23 Aug 2022, at 17:35, Thomas Segismont ***@***.***> wrote: > > > > > > I was able to reproduce. I captured network traffic and, indeed, > pipelining seems involved: > > > > with Quarkus 2.12.2, the client sends execute and close packets > immediately after prepare response packet is received > > with Quarkus 2.10.2, the client waits for response ok packet before > sending close packet > > — > > Reply to this email directly, view it on GitHub < > #27386 (comment)>, > or unsubscribe < > https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA > >. > > You are receiving this because you authored the thread. > > > > — > Reply to this email directly, view it on GitHub > <#27386 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AALOLNWHBKGBKVXBX2OKZGLV2UGYXANCNFSM57BD4YQA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKVBAOUGDCSQFACH4AV34K3V2XM2FANCNFSM57BD4YQA>. You are receiving this because you authored the thread.

tsegismont · 2022-08-24T08:52:19Z

Sure. My address is [github id] at [redhat] dot com. Le mer. 24 août 2022 à 10:39, harpermj66 ***@***.***> a écrit :

…

I reached out to Percona because we have support with them initially. They did say we were their first client to mention reactive hibernate so they were keen to set up a meeting. I can ask my contact at percona to link up with you if necessary. > On 24 Aug 2022, at 09:32, Thomas Segismont ***@***.***> wrote: > > > I'm trying to figure out a workaround for now. I'll get in touch with MySQL > proxy community. > > > Le mar. 23 août 2022 à 20:01, harpermj66 ***@***.***> a > écrit : > > > Great that you found a reason, haven’t got clue obviously how it can be > > changed/fixed if it’s broken or sql proxy just can’t cope - Lol > > > > > On 23 Aug 2022, at 17:35, Thomas Segismont ***@***.***> wrote: > > > > > > > > > I was able to reproduce. I captured network traffic and, indeed, > > pipelining seems involved: > > > > > > with Quarkus 2.12.2, the client sends execute and close packets > > immediately after prepare response packet is received > > > with Quarkus 2.10.2, the client waits for response ok packet before > > sending close packet > > > — > > > Reply to this email directly, view it on GitHub < > > #27386 (comment) >, > > or unsubscribe < > > https://github.com/notifications/unsubscribe-auth/AKVBAOTNZMWMFRSTKFAIQWTV2T4VTANCNFSM57BD4YQA > > >. > > > You are receiving this because you authored the thread. > > > > > > > — > > Reply to this email directly, view it on GitHub > > < #27386 (comment) >, > > or unsubscribe > > < https://github.com/notifications/unsubscribe-auth/AALOLNWHBKGBKVXBX2OKZGLV2UGYXANCNFSM57BD4YQA > > > . > > You are receiving this because you were mentioned.Message ID: > > ***@***.***> > > > — > Reply to this email directly, view it on GitHub < #27386 (comment)>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AKVBAOUGDCSQFACH4AV34K3V2XM2FANCNFSM57BD4YQA >. > You are receiving this because you authored the thread. > — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALOLNTU2ZCCDONFBUHZLULV2XNUHANCNFSM57BD4YQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

geoand · 2022-09-01T07:13:00Z

Should this be closed as the actual issue is tracked in Vert.x?

tsegismont · 2022-09-01T08:06:27Z

I believe we can keep it open until the fixed version is integrated in Quarkus.

geoand · 2022-09-01T08:13:22Z

Okay, no problem

rasvi24 · 2022-09-03T10:50:08Z

Great, this saved me. Thank you. This error gave me no stacktrace even in TRACE level. After trying hours with onFailure just after the query execution, found this issue here just before creating a new one. For anybody trying to use proxysql with the latest 2.12.final version Just downgrade the vertx mysql client as mentioned above. Below version is working.

        <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-mysql-client</artifactId>
            <version>4.2.7</version>
        </dependency>

tsegismont · 2022-09-27T21:14:44Z

Fixed upstream in eclipse-vertx/vertx-sql-client#1236 (Vert.x 4.3.4)

geoand · 2022-09-28T05:05:28Z

Thanks for the update @tsegismont.

Just out of curiosity, when is Vert.x 4.3.4 planned?

harpermj66 · 2022-09-28T07:44:24Z

Great, thank you!

…

On 28 Sep 2022, at 06:05, Georgios Andrianakis ***@***.***> wrote: Thanks for the update @tsegismont <https://github.com/tsegismont>. Just out of curiosity, when is Vert.x 4.3.4 planned? — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKVBAOQM3VBYAHQGBKN4UVTWAPG2FANCNFSM57BD4YQA>. You are receiving this because you authored the thread.

tsegismont · 2022-09-28T08:23:37Z

@geoand very soon, beginning of next week maybe, right @vietj?

@harpermj66 you're welcome. Any news from your contacts at Percona? Are they still willing to talk about this?

harpermj66 · 2022-10-06T13:40:57Z

Embarressingly not willing to engage for reasons which I don't really go along with.
Is there something they should be doing for future proofing?

tsegismont · 2022-10-06T14:09:50Z

No worries, it is fine to not support pipelining. I can think of reasons why ProxySQL doesn't (shared connections).

harpermj66 · 2022-10-11T07:08:57Z

4.2.7 is a version that works with Quarks 2.11.1 and is the one bundled in 2.10.1 The next one I could find was 4.3.0 which didn’t work with 2.11.1

…

On 22 Aug 2022, at 17:19, Sanne Grinovero ***@***.***> wrote: In your dependencyManagement section, add the vert.x client version above the import for the Quarkus BOM, so that it takes precedence: <dependencyManagement> <dependencies> <dependency> <groupId>io.vertx</groupId> <artifactId>vertx-mysql-client</artifactId> <version>something-something</version> </dependency> <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-bom</artifactId> <version>${quarkus.platform.version}</version> <scope>import</scope> <type>pom</type> </dependency> — Reply to this email directly, view it on GitHub <#27386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKVBAOQEXM3GKCGHPVWN5PDV2OSBNANCNFSM57BD4YQA>. You are receiving this because you authored the thread.

tsegismont · 2022-10-11T09:21:29Z

This should be fixed in 2.13.1.Final. Can you have a try? Thank you

harpermj66 · 2022-10-11T11:27:20Z

I don't know where that comment came from 4 hours ago :-)
I've already been using 2.13.1.Final and had no issues but we'll do a bit more testing.

diegoherrojo · 2022-10-20T13:50:16Z

Hi everyone!

We are developing a Quarkus based app (with reactive Panache stack), and we cannot make it work with version 2.13.3.Final (or with any version equals or greater than 2.11.0) with a database that sits behind a SQL proxy. It works perfectly when we connect with a DB without a proxy in the middle, or when we use up to 2.10.4 version.
We can override vertx mysql driver to version 4.2.7, but from 2.13.0 that override produces runtime errors.

The error we get when we execute transactions against a DB behind a SQL proxy is:

(vert.x-eventloop-thread-0) Unexpected error: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
	[Exception 0] io.vertx.core.impl.NoStackTraceThrowable: Fail to read any response from the server, the underlying connection might get lost unexpectedly.
	[Exception 1] io.vertx.core.impl.NoStackTraceThrowable: Connection is not active now, current status: CLOSED

Is there any workaround to this problem?

Thanks!

tsegismont · 2022-10-24T12:25:43Z

@diegoherrojo thanks for reporting.

Can you share a simple reproducer? We have a test upstream which verifies connectivity through ProxySQL, so I'm confused.

Can you verify with mvn dependency:tree if your project uses vertx-mysql-client 4.3.4 ?

diegoherrojo · 2022-10-24T18:08:42Z

Thanks for your quick response @tsegismont!

Yes, I verified that we are using vertx-mysql-client 4.3.4.

Let me create a simple test to share here.

diegoherrojo · 2022-10-26T20:27:01Z

Hi everyone!

@tsegismont I created a small app reproducing the error with Quarkus 2.13.2.Final.

In order to run it, please unzip test.zip and then execute run.sh.
Docker compose, maven and jdk 17 are needed for the test.

The test spins up a MySQL and a ProxySQL and then executes 2 times a POST to the app API (which creates some entities in DB):

Once connecting directly to MySQL (successful).
Second time connecting through ProxySQL (fails).

Thanks!

tsegismont · 2022-10-27T09:06:17Z

Thank you, I'll try this out asap and come back to you @diegoherrojo

diegoherrojo · 2022-10-27T11:16:41Z

Thank you @tsegismont !

tsegismont · 2022-11-08T17:02:15Z

@diegoherrojo I was able to reproduce today and will come back to you shortly

diegoherrojo · 2022-11-08T17:11:53Z

Thanks @tsegismont !

tsegismont · 2022-11-09T17:00:58Z

This is another with pipelining which I had not identified in the previous fix.

I've filed eclipse-vertx/vertx-sql-client#1254

diegoherrojo · 2022-11-09T17:16:28Z

Oh great @tsegismont! Thanks for you help!

tsegismont · 2022-11-16T10:48:18Z

@diegoherrojo the fix has been merged in eclipse-vertx/vertx-sql-client#1257

I tried a snapshot version with your reproducer and the problem goes away.

Vert.x 4.3.5 will be released very soon. When Quarkus upgrades to this version we can close this issue.

diegoherrojo · 2022-11-16T12:16:15Z

Thanks @tsegismont for all your effort and support! 👌

tsegismont · 2022-11-24T19:34:31Z

Closed by #29246

tsegismont · 2022-11-24T19:35:40Z

@diegoherrojo if you still experience issues with a snasphot version or 2.15 (when it is released) please reopen.

diegoherrojo · 2022-11-25T13:34:02Z

Thank you @tsegismont !

harpermj66 added the kind/bug Something isn't working label Aug 19, 2022

quarkus-bot bot added the triage/needs-triage label Aug 19, 2022

gastaldi added area/hibernate-reactive Hibernate Reactive and removed triage/needs-triage labels Aug 19, 2022

This was referenced Sep 16, 2022

MySQL Client sends close statement commend before all rows are decoded eclipse-vertx/vertx-sql-client#1234

Closed

Fixes to MySQL pipelining eclipse-vertx/vertx-sql-client#1236

Merged

tsegismont self-assigned this Sep 27, 2022

tsegismont added this to the 2.15 - main milestone Nov 24, 2022

tsegismont closed this as completed Nov 24, 2022

Quarkus version change from 2.10.1.Final to 2.11.1.Final has breaking changes with Percona SqlProxy #27386

Quarkus version change from 2.10.1.Final to 2.11.1.Final has breaking changes with Percona SqlProxy #27386

Comments

harpermj66 commented Aug 19, 2022 • edited Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

quarkus-bot bot commented Aug 19, 2022

gastaldi commented Aug 22, 2022

harpermj66 commented Aug 22, 2022 • edited Loading

Sanne commented Aug 22, 2022

harpermj66 commented Aug 22, 2022 via email

Sanne commented Aug 22, 2022

harpermj66 commented Aug 22, 2022 via email

tsegismont commented Aug 23, 2022

tsegismont commented Aug 23, 2022

harpermj66 commented Aug 23, 2022 via email

tsegismont commented Aug 24, 2022 via email

harpermj66 commented Aug 24, 2022 via email

tsegismont commented Aug 24, 2022 via email

geoand commented Sep 1, 2022

tsegismont commented Sep 1, 2022

geoand commented Sep 1, 2022

rasvi24 commented Sep 3, 2022 • edited Loading

tsegismont commented Sep 27, 2022

geoand commented Sep 28, 2022

harpermj66 commented Sep 28, 2022 via email

tsegismont commented Sep 28, 2022

harpermj66 commented Oct 6, 2022

tsegismont commented Oct 6, 2022

harpermj66 commented Oct 11, 2022 via email

tsegismont commented Oct 11, 2022

harpermj66 commented Oct 11, 2022

diegoherrojo commented Oct 20, 2022 • edited Loading

tsegismont commented Oct 24, 2022

diegoherrojo commented Oct 24, 2022

diegoherrojo commented Oct 26, 2022

tsegismont commented Oct 27, 2022

diegoherrojo commented Oct 27, 2022

tsegismont commented Nov 8, 2022

diegoherrojo commented Nov 8, 2022

tsegismont commented Nov 9, 2022

diegoherrojo commented Nov 9, 2022 • edited Loading

tsegismont commented Nov 16, 2022

diegoherrojo commented Nov 16, 2022

tsegismont commented Nov 24, 2022

tsegismont commented Nov 24, 2022

diegoherrojo commented Nov 25, 2022

harpermj66 commented Aug 19, 2022 •

edited

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

harpermj66 commented Aug 22, 2022 •

edited

Loading

rasvi24 commented Sep 3, 2022 •

edited

Loading

diegoherrojo commented Oct 20, 2022 •

edited

Loading

diegoherrojo commented Nov 9, 2022 •

edited

Loading