Queries may be blocked forever when poco session pool is enabled under disagg arch

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

In a previous commit, I introduce storage.s3.enable_http_pool and enable it by default. It help us reduce some error when testing with MinIO at that time. But I find it make queries hang in the latest version.

1. Deploy a tidb cluster with 1 wn and 1 rn (disagg arch)
2. Load chbenchmark with 1 tiflash replica
3. Execute set tidb_isolation_read_engines = 'tiflash'; select count(*) from order_line; and select * from order_line limit 1;

2. What did you expect to see? (Required)

All queries finish normally

3. What did you see instead (Required)

The last query get blocks, and we can see following message in the logging

[2023/06/06 16:37:37.906 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:38.906 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:39.906 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:40.907 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:41.907 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:42.907 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:43.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:44.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:45.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:46.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:47.909 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:48.909 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:49.909 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:50.910 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:51.910 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]

4. What is your TiFlash version? (Required)

edf82d2