ConnectionError with redis.asyncio.ConnectionPool in FastAPI/uvicorn

[andreabravetti]

Version:

redis-py>=5.0.0 (not sure previous versions)
redis==7.2.4

Platform:

Python 3.12 on Fedora 40 and Debian 12

Description:

I created a dedicated repo here with examples and explanation, so you can easily reproduce the problem.

I will sum it up here:

I have a very small and very fast asyncio FastAPI service that need to read from Redis.

I created a ConnectionPool in the lifespan:

@asynccontextmanager
async def lifespan(application: FastAPI):
    application.state.async_pool = redis.asyncio.ConnectionPool(
        host=REDIS_HOST, port=REDIS_PORT)
    yield
    await application.state.async_pool.aclose()

app = FastAPI(lifespan=lifespan)

I get a Client with a Connection from the ConnectionPool:

@app.get("/v1/async/pool")
async def get_from_redis():
    async with redis.asyncio.Redis.from_pool(app.state.async_pool) as client:
        await client.get("foo")
        ...

It seems to work but under heavy load it gives some random error like:

Traceback 1:

  ...
  File "uvloop/handles/stream.pyx", line 687, in uvloop.loop.UVStream.writelines
  File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x560d4f4ba590>; the handler is closed

Traceback 2:

  ...
  File "/home/andrea/Devel/redis-async-test/venv/lib64/python3.12/site-packages/redis/asyncio/connection.py", line 497, in send_command
    await self.send_packed_command(
  File "/home/andrea/Devel/redis-async-test/venv/lib64/python3.12/site-packages/redis/asyncio/connection.py", line 484, in send_packed_command
    raise ConnectionError(
redis.exceptions.ConnectionError: Error UNKNOWN while writing to socket. Connection lost.

Traceback 3 (only with hiredis):

  ...
  File "/home/andrea/Devel/redis-async-test/venv/lib64/python3.12/site-packages/redis/_parsers/hiredis.py", line 206, in read_response
    await self.read_from_socket()
  File "/home/andrea/Devel/redis-async-test/venv/lib64/python3.12/site-packages/redis/_parsers/hiredis.py", line 186, in read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR) from None
redis.exceptions.ConnectionError: Connection closed by server.

Traceback 4 (only without hiredis):

  ...
  File "/home/andrea/Devel/redis-async-test/venv/lib64/python3.12/site-packages/redis/_parsers/resp2.py", line 90, in _read_response
    raw = await self._readline()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/andrea/Devel/redis-async-test/venv/lib64/python3.12/site-packages/redis/_parsers/base.py", line 221, in _readline
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

I suspect there is an error, some type of race condition, in the way redis.asyncio.ConnectionPool handles the connections to the server, or I don't understand how it's supposed to work.

If I add a little sleep (await asyncio.sleep(0.1) or even less) inside the FastAPI function it works and I have no errors.

It may be related to #2065, #2491 and #3066, don't know at this time.

[andreabravetti]

I'm sorry, it was my mistake, it turns out I was wrong in understanding Clients and ConnectionPools.

If anyone ends up reading here this works fine:

@asynccontextmanager
async def lifespan(application: FastAPI):
    application.state.async_pool = redis.asyncio.ConnectionPool(
        host=REDIS_HOST, port=REDIS_PORT)
    application.state.async_client = redis.asyncio.Redis.from_pool(
        application.state.async_pool)
    yield
    application.state.async_client.close()

app = FastAPI(lifespan=lifespan)

And:

@app.get("/v1/async/pool")
async def get_from_redis():
    await app.state.async_client.get("foo")
    ...