Server cannot shutdown on python3.12

[masenf]

Not sure if uvicorn support for python 3.12.0 is expected yet, but my testing did not get far.

Using python.org 3.12 compiled for apple silicon, I have not tested other platforms yet.

masenf@minicone uvicorn-hang % python3.12 -V
Python 3.12.0
masenf@minicone uvicorn-hang % python3.12 -m venv VENV-312
masenf@minicone uvicorn-hang % ./VENV-312/bin/pip install uvicorn==0.23.2
Collecting uvicorn==0.23.2
  Obtaining dependency information for uvicorn==0.23.2 from https://files.pythonhosted.org/packages/79/96/b0882a1c3f7ef3dd86879e041212ae5b62b4bd352320889231cc735a8e8f/uvicorn-0.23.2-py3-none-any.whl.metadata
  Using cached uvicorn-0.23.2-py3-none-any.whl.metadata (6.2 kB)
Collecting click>=7.0 (from uvicorn==0.23.2)
  Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting h11>=0.8 (from uvicorn==0.23.2)
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Using cached uvicorn-0.23.2-py3-none-any.whl (59 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Installing collected packages: h11, click, uvicorn
Successfully installed click-8.1.7 h11-0.14.0 uvicorn-0.23.2

app.py

async def app(scope, receive, send):
    assert scope["type"] == "http"
    await send({"type": "http.response.start", "status": 204, "headers": []})
    await send({"type": "http.response.body", "body": b"", "more_body": False})

masenf@minicone uvicorn-hang % ./VENV-312/bin/uvicorn app:app
INFO:     Started server process [84185]
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     127.0.0.1:63725 - "GET / HTTP/1.1" 204 No Content
^CINFO:     Shutting down

☝️ and it just kind of sits there. It exits as expected on python3.11. (If the server does not receive any requests, then it also exits without hanging)

I ran a debugger in the shutdown function and it appears to be hanging at

uvicorn/uvicorn/server.py

Lines 271 to 272 in 40b99b8

	for server in self.servers:
	await server.wait_closed()

server._active_count is non-zero and does not seem to be making progress toward shutting down.

I try with the following patch and it seems to resolve the issue:

From: Masen Furer <[email protected]>
Date: Wed, 4 Oct 2023 13:09:55 -0700
Subject: [PATCH] server: wait_close after closing connections

Avoid hang during shutdown with python3.12
---
 uvicorn/server.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/uvicorn/server.py b/uvicorn/server.py
index 3e0db9d..d4414af 100644
--- a/uvicorn/server.py
+++ b/uvicorn/server.py
@@ -268,14 +268,15 @@ class Server:
             server.close()
         for sock in sockets or []:
             sock.close()
-        for server in self.servers:
-            await server.wait_closed()
 
         # Request shutdown on all existing connections.
         for connection in list(self.server_state.connections):
             connection.shutdown()
         await asyncio.sleep(0.1)
 
+        for server in self.servers:
+            await server.wait_closed()
+
         # When 3.10 is not supported anymore, use `async with asyncio.timeout(...):`.
         try:
             await asyncio.wait_for(

However I'm not sure if this change would have broader implications for uvicorn or libraries that interact with it, but the tests do seem to pass with py3.11 and py3.12 after this change (except test_sigint_abort_req, which also failed before the change). I did also have to qualify the uvloop dep since they do not support 3.12 yet.

This issue might be similar to #1803, asked last year, but i think that was more gunicorn-specific.

[jcheng5]

This behavior appears to be because of this change in Python: python/cpython#98582 It looks to me like @masenf's suggested change is exactly the correct fix, as server.wait_closed's documented behavior has always been to wait for existing connections to go away on their own--it just happened not to do that pre-3.12. So the existing Uvicorn code requests existing connections to shutdown, but not until after it waits for them to leave voluntarily.

This is a big problem for applications and frameworks that build on Uvicorn and use WebSockets--it puts Python in a state where it appears hung and hitting Ctrl+C does nothing.

[jcheng5]

Or perhaps the wait_closed for loop should go inside of the _wait_tasks_to_complete; it seems to be in the spirit of the other logic in there, and would be subject to the same timeout.

[Kludex]

I cannot reproduce it.

[masenf]

I tried the following and was not able to repro on linux/docker. Although in my bigger, more complicated project that is using websockets (where i first noticed the issue), the problem still arises, even on linux.

docker run -it -p8000:8000 python:3.12 bash
cat - <<EOF > app.py
async def app(scope, receive, send):
    assert scope["type"] == "http"
    await send({"type": "http.response.start", "status": 204, "headers": []})
    await send({"type": "http.response.body", "body": b"", "more_body": False})
EOF
pip install uvicorn==0.23.2
uvicorn app:app & sleep 1 && curl http://localhost:8000; fg

(Then CTRL-C to stop uvicorn.)

---

I was able to reproduce on Windows 11 and macOS 13.6 with the steps in my original message. However macOS 14 is not reproducing the issue consistently for me (upgraded from 13.5.4 last week). Not sure if this is relevant, but data points.

[Kludex]

It looks like the pipeline fails on Windows for 3.12: #2129

[jcheng5]

Sorry, I should've clarified. It only reproduces when a connection is left open. So the docker snippet @masenf posted won't do it because curl closes so quickly.

Try this instead:

docker run -it -p8000:8000 python:3.12 bash
cat - <<EOF > app.py
async def app(scope, receive, send):
    assert scope["type"] == "http"
    await send({"type": "http.response.start", "status": 204, "headers": []})
    await send({"type": "http.response.body", "body": b"", "more_body": False})
EOF
pip install uvicorn==0.23.2
apt update && apt install netcat-openbsd
uvicorn app:app & sleep 1 && nc localhost 8000 & fg -

Then Ctrl+C.

Our framework is heavily WebSocket based, so there's essentially always a connection open, especially when developing locally.

[Kludex]

I'll see if I can get a Windows.

[jcheng5]

No Windows machine is needed--this repro is on Docker but it reproduces on any platform. You just need to hold a connection open somehow--a WebSocket will work great but, as in this case, so will nc localhost <port> (or telnet localhost <port>).

[Kludex]

This is a bug on CPython. Discussion can be found here: python/cpython#79033

[Kludex]

Let's wait for the next patch release.

[jcheng5]

From the description of python/cpython#111336 that change is not going to fix uvicorn, and the behavior that's causing uvicorn to hang is not a bug--it's the fact that it wasn't hanging before 3.11 that is considered a bug!

Historical note: In 3.11 and before, this was broken, returning
immediately if the server was already closed, even if there
were still active connections. An attempted fix in 3.12.0 was
still broken, returning immediately if the server was still
open and there were no active connections. Hopefully in 3.12.1
we have it right.

Just to restate the problem: uvicorn is saying "wait for every active connection to close" without first asking the connections to close. In fact, it waits until every active connection has closed of its own volition and then asks every active connection to close. I can't imagine a "fix" for this logic is going to come from CPython.

[jcheng5]

@Kludex would you be open to a PR? I'm MORE than happy to create one.

[Kludex]

sure, I can review it.

[jcheng5]

Thank you @Kludex! #2145

[masenf]

Resolved with the release of uvicorn-0.24.0; Thanks to @jcheng5 @Kludex and @zanieb for making it happen 🙌

[Kludex]

Sorry for the delay. 🙏