Quarkus keeps dead database connections in its connection pool

[TobiWeier]

Describe the bug
Dead database connections are not removed from the pool. Since these are unusable, Quarkus has to make further connections. If the pool is filled completely with dead connections, it becomes unusable. The status of the pool remains permanently on DOWN, even when the database is available again.

Expected behavior
Agroal connection pool should be able to remove dead database connections from pool.

Actual behavior
If the connected database crashes or the connected database cluster performs a leader switch, agroal connection pool helds dead connections as available connections in pool. Every crash/leader-switch increases the number of supposedly "available" but in fact dead connections in pool. If the maximum number of connections is reached, the pool status remains permanently on DOWN, even when the database is available again.

To Reproduce
Link to a small reproducer (preferably a Maven project if the issue is not Gradle-specific).
https://github.com/TobiWeier/quarkus-dead-connections-issue

Steps to reproduce the behavior (with creating project from scratch):

1. Create quarkus project

    mvn io.quarkus:quarkus-maven-plugin:1.11.3.Final:create -DprojectGroupId=org.acme -DprojectArtifactId=dead-conns-issue -DclassName="org.acme.getting.started.GreetingResource" -Dpath="/hello"
   

2. add hibernate-orm dependencies

    <!-- Hibernate ORM specific dependencies -->
    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-hibernate-orm</artifactId>
    </dependency>
    <!-- JDBC driver dependencies -->
    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-jdbc-postgresql</artifactId>
    </dependency>
   

3. add quarkus-smallrye-metrics dependency

    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-smallrye-metrics</artifactId>
    </dependency>
   

4. add quarkus-smallrye-health dependency

    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-smallrye-health</artifactId>
    </dependency>
   

5. add properties to application.properties (as described below)

6. start database

    docker run -d -e POSTGRES_USER=hibernate -e POSTGRES_PASSWORD=hibernate -e POSTGRES_DB=hibernate_db -e POSTGRES_PORT=5432 --name mydb -p 5432:5432 postgres
   

7. start quarkus

    mvn clean quarkus:dev
   

8. use command for watching metrics from connection pool

    watch -n 2 "curl -s http://localhost:8080/q/metrics | grep -E '^vendor_agroal'"
   

9. use command for watching health status

    watch -n 2 "curl -is http://localhost:8080/q/health"
   

10. kill or stop database

     docker kill mydb   
    
     or   
     
     docker stop mydb
    

11. restart database

     docker start mydb
    

After db restart the number of available connections in pool has grown from 3 to 4. Now repeat steps 10 and 11 two times and observe the metrics values "vendor_agroal_available_count" and the health status. If the maximum number of connections is reached, the pool status remains permanently on DOWN, even when the database is available again.

Configuration

quarkus.datasource.db-kind = postgresql
quarkus.datasource.username = hibernate
quarkus.datasource.password = hibernate
quarkus.datasource.jdbc.url = jdbc:postgresql://localhost:5432/hibernate_db
quarkus.hibernate-orm.database.generation=drop-and-create
quarkus.hibernate-orm.dialect=org.hibernate.dialect.PostgreSQLDialect
quarkus.hibernate-orm.log.jdbc-warnings=true
quarkus.datasource.health.enabled=true
quarkus.datasource.metrics.enabled=true
quarkus.datasource.jdbc.enable-metrics=true
quarkus.datasource.jdbc.max-size=5
quarkus.datasource.jdbc.min-size=3

Screenshots
(If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

• Output of uname -a or ver:
  Linux nb-8y1r 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Output of java -version:

   openjdk version "14.0.2" 2020-07-14
   OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04)
   OpenJDK 64-Bit Server VM (build 14.0.2+12-Ubuntu-120.04, mixed mode, sharing)
  

• GraalVM version (if different from Java):

• Quarkus version or git rev:
  1.11.3.Final

• Build tool (ie. output of mvnw --version or gradlew --version):
  Apache Maven 3.6.3

Additional context
(Add any other context about the problem here.)

[famod]

/cc @barreiro @gsmet @Sanne @yrodiere

[gsmet]

@yrodiere maybe you could have a look to this one given you played a bit with connections lately?

[barreiro]

It's a leak in Agroal. I've been trying to figure out how it could happen for a few days now, and with the reproducer provided by @TobiWeier I was finally able to crack it down.

[arvid-villen-adtoox]

We might be affected by this as well.
Does anybody know which versions of Quarkus this affects?

[TobiWeier]

In which version of Quarkus we can expect a solution to the problem?

[stuartwdouglas]

I think agroal/agroal@1929c97 is the relevant commit, it will depend on when @barreiro does a release of Agroal.