-
-
Notifications
You must be signed in to change notification settings - Fork 754
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RuntimeError: unable to perform operation on <TCPTransport #1251
Comments
Hi, in my case i come into same error. I use socketio and deploy with uvicorn.
My first look is to closed handler this seems to be closed before invoking self.close() in server.py at line 232. So i solved it adding :
This seems to be worked for me but i'm not sure this is the defenitive solution. |
fixed with 10.1 release of websockets? https://websockets.readthedocs.io/en/stable/project/changelog.html#id2 |
Hi! Wanted to leave a comment here about the traceback @Lundark posted last month. We had an incident over the weekend where we started encountering random CloudFront HTTP 504s ultimately caused by that exact same We noticed today that Uvicorn had a new release |
Considering that those PRs were merged less than a week, it's probably not tightly related to this issue, even if the error message is the same. If possible, would you mind creating a new issue with additional info? |
I can try to find some time, we're pretty swamped with work currently. It also might have more to do with our deployment setup than Uvicorn itself, so our case might not even be reproducible. Thanks for the reply! |
It seems this issue is related: #1345 |
I only saw the issue over the Internet in the context of WebSockets, but I've actually faced it in a little bit of different setup: I'm keeping an Environment
Custom workerclass CustomWorker(UvicornWorker):
CONFIG_KWARGS = {"lifespan": "off"} # also tried `"http": "h11"` to no avail
|
@caleb15 Nope, unfortunately. Apparently Did you maybe upgrade/downgrade anything else on that date? |
Not that I'm aware of. |
I started having the same traceback as the original issue right after upgrading to websockets==10.2 (latest uvicorn release 0.17.5) It seems something changed here: python-websockets/websockets@10.1...10.2#diff-3851a087134dc94d4edce57e405054b17b85aed1639b8f62f1a79ba582345c3fL1306 related to closing the connection, and the line numbers in my sentry reports agree.
I'm not sure if websockets library or uvicorn is the culprit here ? |
I also faced the same issue the moment I upgraded to uvcorn 0.17.5 release.
|
Happening to us too,
|
I again downgraded the version of uvicorn==0.15 with websockets==10.2 but I saw the same issue happening again. |
We're going to test wsproto and see if the issue still occurs with that. |
facing same issue fastapi==0.75.0 |
I'm having the same issue. Tried running with both Gunicorn and Uvicorn. Same versions as previous comment. |
Downgrading to websockets==10.1 resolved the issue for me. |
This one worked for me too! |
I envy you both, gave
|
@ChronoDK @Muhammed-Rajab May I ask your versions of Actually, in my particular stacktrace there's nothing related to File "httpx/_client.py", line 1506, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "httpx/_client.py", line 1593, in send
response = await self._send_handling_auth(
File "httpx/_client.py", line 1621, in _send_handling_auth
response = await self._send_handling_redirects(
File "httpx/_client.py", line 1658, in _send_handling_redirects
response = await self._send_single_request(request)
File "httpx/_client.py", line 1695, in _send_single_request
response = await transport.handle_async_request(request)
File "httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "httpcore/_async/connection_pool.py", line 220, in handle_async_request
await self._close_expired_connections()
File "httpcore/_async/connection_pool.py", line 188, in _close_expired_connections
await connection.aclose()
File "httpcore/_async/connection.py", line 159, in aclose
await self._connection.aclose()
File "httpcore/_async/http11.py", line 212, in aclose
await self._network_stream.aclose()
File "httpcore/backends/asyncio.py", line 48, in aclose
await self._stream.aclose()
File "anyio/streams/tls.py", line 168, in aclose
await self.transport_stream.aclose()
File "anyio/_backends/_asyncio.py", line 1159, in aclose
self._transport.write_eof()
File "uvloop/handles/stream.pyx", line 696, in uvloop.loop.UVStream.write_eof
self._ensure_alive()
File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
raise RuntimeError( |
Same happens with File "anyio/streams/tls.py", line 168, in aclose
await self.transport_stream.aclose()
File "anyio/_backends/_asyncio.py", line 1163, in aclose
self._transport.close()
File "asyncio/selector_events.py", line 700, in close
self._loop.call_soon(self._call_connection_lost, None)
File "asyncio/base_events.py", line 745, in call_soon
self._check_closed()
File "asyncio/base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed') That's weird 🤔 |
anyio == 3.5.0 |
This is the error I'm getting when using websockets 10.2:
|
Thanks @ChronoDK! Curiouser and curiouser… Your libs are the same as mine, and I'm not even using |
Apparently, this is the right direction. In my case Uvicorn runs different requests in different event loops. Once the connection pool is full, it tries to clear itself – including the connections created in other event loops which are possibly closed. |
lots of info here, a minimal reproducible example would tremendously help, I'm not able to reproduce |
Hi @euri10, Sure, sorry to put the burden on you. This is what reproduces the situation for me: import asyncio
from time import sleep
from httpcore._async import AsyncConnectionPool
pool = AsyncConnectionPool(keepalive_expiry=1.0)
async def main():
response = await pool.request("GET", "https://httpbin.org/get")
assert response.status == 200
# First request creates a new connection within the event loop.
asyncio.run(main())
# Sleep for longer than the `keepalive_expiry` to make the pool close expired connections.
sleep(2.0)
# Sending another request from a different event loop.
# Why different? For me that's the case in an ASGI application (Gunicorn + Django),
# where ingoing requests may be handled in different event loops.
# I'm trying to keep a global connection pool in the app which results in calling
# `pool.request` from different event loops.
asyncio.run(main()) Here's the traceback again: Traceback (most recent call last):
File "/Users/eigenein/GitHub/blitz-lab/src/mre.py", line 16, in <module>
asyncio.run(main())
File "/usr/local/Cellar/[email protected]/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/Cellar/[email protected]/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
return future.result()
File "/Users/eigenein/GitHub/blitz-lab/src/mre.py", line 10, in main
response = await pool.request("GET", "https://httpbin.org/get")
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/interfaces.py", line 41, in request
response = await self.handle_async_request(request)
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 220, in handle_async_request
await self._close_expired_connections()
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 188, in _close_expired_connections
await connection.aclose()
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/connection.py", line 159, in aclose
await self._connection.aclose()
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 212, in aclose
await self._network_stream.aclose()
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 48, in aclose
await self._stream.aclose()
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/anyio/streams/tls.py", line 168, in aclose
await self.transport_stream.aclose()
File "/Users/eigenein/GitHub/blitz-lab/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1163, in aclose
self._transport.close()
File "/usr/local/Cellar/[email protected]/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 700, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/usr/local/Cellar/[email protected]/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 745, in call_soon
self._check_closed()
File "/usr/local/Cellar/[email protected]/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed |
I had the same issue. I downgraded websockets to 10.1 as suggested by comments, and the problem is gone. |
Where is even uvicorn? |
What is needed to debug here:
Is there someone who can help with that data? |
@Kludex That's certainly a good point. I doubted if my example was valid, but decided to publish since |
If someone faces this again, please create a new issue with:
|
Checklist
master
.Describe the bug
We are seeing frequent
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x7f25014d3810>; the handler is closed
To reproduce
Run django with latest uvicorn inside kubernetes.
Expected behavior
No error
Actual behavior
Error
Debugging material
Environment
Additional context
This started happening when we started using uvicorn. In production it happens quite often:
There are some similar issues:
The text was updated successfully, but these errors were encountered: