CXF clients based on java.net.http.HttpClient leak threads

[ppalaga]

This is a known CXF issue reported as https://issues.apache.org/jira/browse/CXF-8885

It is reproducible also with Quarkus CXF 2.3.0 and 2.2.2, when the following conditions are met:

• java.net.http.HttpClient is used under the hood
• The CXF client is created per request

Steps to reproduce:

cd quarkus-cxf
git remote add ppalaga [email protected]:ppalaga/quarkus-cxf.git
git fetch ppalaga CXF-8885-reproducer
git tag CXF-8885-reproducer FETCH_HEAD
git checkout CXF-8885-reproducer
mvnd clean install -DskipTests -Dquarkus.build.skip
cd integration-tests/mtom
export QUARKUS_CXF_MTOM_SOAK_ITERATIONS=100000
mvn clean test -Dtest=MtomTest#soak
...
2023-08-31 18:55:10,337 INFO  [io.qua.cxf.it.ws.mto.ser.MtomServiceImpl] (executor-thread-1) Received 64647 bytes of content type application/octet-stream
2023-08-31 18:55:10,339 INFO  [io.qua.cxf.it.ws.mto.ser.MtomTest] (main) Soaking with 64647 bytes, round 11318
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f9060c00000, 16384, 0) failed; error='Not enough space' (errno=12)
2023-08-31 18:55:10,343 INFO  [io.qua.cxf.it.ws.mto.ser.MtomServiceImpl] (executor-thread-1) Received 64647 bytes of content type application/octet-stream
2023-08-31 18:55:10,345 INFO  [io.qua.cxf.it.ws.mto.ser.MtomTest] (main) Soaking with 64647 bytes, round 11319
2023-08-31 18:55:10,349 INFO  [io.qua.cxf.it.ws.mto.ser.MtomServiceImpl] (executor-thread-1) Received 64647 bytes of content type application/octet-stream
2023-08-31 18:55:10,351 INFO  [io.qua.cxf.it.ws.mto.ser.MtomTest] (main) Soaking with 64647 bytes, round 11320
2023-08-31 18:55:10,354 INFO  [io.qua.cxf.it.ws.mto.ser.MtomServiceImpl] (executor-thread-1) Received 64647 bytes of content type application/octet-stream
2023-08-31 18:55:10,356 INFO  [io.qua.cxf.it.ws.mto.ser.MtomTest] (main) Soaking with 64647 bytes, round 11321
2023-08-31 18:55:10,360 INFO  [io.qua.cxf.it.ws.mto.ser.MtomServiceImpl] (executor-thread-1) Received 64647 bytes of content type application/octet-stream
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[WARNING] Corrupted channel by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /home/ppalaga/orgs/cxf/quarkiverse-cxf/integration-tests/mtom/target/surefire-reports/2023-08-31T18-53-59_706-jvmRun1.dumpstream

When inpecting the JVM during the test, one sees that the number of thread is rising continuously. Those are mostly HttpClient related threads like HttpClient-*-SelectorManager.

Possible workarounds:

• Avoid creating CXF clients too often, such as per request
• Use a different HTTP client implementation, such as
  • HC5 (see https://docs.quarkiverse.io/quarkus-cxf/dev/reference/extensions/quarkus-cxf-rt-transports-http-hc5.html )
  • Configure CXF to use org.apache.cxf.transport.http.URLConnectionHTTPConduit via setting HTTPConduitFactory.class on CXF Bus

A proper fix needs to be done in CXF via https://issues.apache.org/jira/browse/CXF-8885

[famod]

Now that CXF 4.0.3 is released (which apparently fixed https://issues.apache.org/jira/browse/CXF-8885) and updated in quarkus-cxf, is there anything to do here?

[ppalaga]

is there anything to do here?

I need to verify that 4.0.3 behaves as expected. If so, we probably need to document that java.net.http.HttpClient based clients have to be closed to prevent the leak. I need to figure out whether there is a way to do this automatically via means offered by CDI.

Once all the above is done, we can consider changing the default conduit-factory in the next minor/major release

[famod]

Sounds good, thanks for clarifying!