uvicorn workres stoop working after some time

[hudecof]

Hi there,

I have 3 deployments os my fastapi+sqlalchemy application.
After some time, in days, the api not very often user, still in UA, the application stopped working on all 3 instances.
There is no log in access/error log, just one of these error

$ curl -X 'GET'   'http://127.0.0.1:8004/api/v1/check/ping'   -H 'accept: */*' -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8004 (#0)
> GET /api/v1/check/ping HTTP/1.1
> Host: 127.0.0.1:8004
> User-Agent: curl/7.61.1
> accept: */*
> 
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server
[PROD] app_captive_api@captive-log: ~ $ curl -X 'GET'   'http://127.0.0.1:8004/api/v1/check/ping'   -H 'accept: */*' -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8004 (#0)
> GET /api/v1/check/ping HTTP/1.1
> Host: 127.0.0.1:8004
> User-Agent: curl/7.61.1
> accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

I already creates issue for guvicorn, since I was using the guvicron to run the FastAPI, but with the pure uvicorn is the same
benoitc/gunicorn#2776

The only solution is to restart the application. Any help? Running on Rocky Linux 8.4.
Accorting the TCP dumo the packets get to the application, since there is from time to time empty answer, but there is no DEBUG log, and the debuging is on.

Running as /opt/captive-api/current/env/bin/uvicorn --host 127.0.0.1 --port 8004 --workers 4 --access-log --log-level debug --log-config /opt/captive-api/current/uvicorn.logging.yml main:app

# cat requirements.rocky.txt 
anyio==3.5.0
asgiref==3.4.1
click==8.0.4
contextlib2==21.6.0
contextvars==2.4
dataclasses==0.8
fastapi==0.75.0
greenlet==1.1.2
gunicorn==20.1.0
h11==0.13.0
httptools==0.3.0
idna==3.3
immutables==0.16
importlib-metadata==4.8.3
pydantic==1.9.0
PyMySQL==1.0.2
python-dotenv==0.20.0
PyYAML==6.0
sniffio==1.2.0
SQLAlchemy==1.4.32
starlette==0.17.1
typing_extensions==4.1.1
uvicorn==0.16.0
uvloop==0.14.0
watchgod==0.7
websockets==9.1
zipp==3.6.0

[euri10]

it's going to be hard to help, the only thing I'd try is maybe to avoid running it with --workers, without gunicorn, and see if this happens again.
One reason a device will send a RST is in response to receiving a packet for a closed socket, but there might be other reasons, so hard to say.

[hudecof]

both scenarios got the same problem

• gunicorn with unicorn workers
• pure unicorn

If the replay is Connection reset by peer, there is RST packet.
If the replay is Empty reply from server, tehre is not RST packet.

I'm running on another projest FastAPI+UVICORN for 2 years without problem, The difference is SQL database, in that solution I do not have any.

I could disable the local firewall and chech it, but fir localhost are all connection allowed. The MariaDB is also local.

[hudecof]

seems to be related to the #1199

There is zabbix monitoring checking the TCP port.

# netstat -apn |   grep CLOSE_WAIT | grep 8004 | wc -l
2377

[Kludex]

Should we add to the PR/issues templates "Only create an issue if you're on the latest uvicorn version"? I have the feeling that it would avoid most of the discussions. 🤔

[euri10]

yup good call, I didn't check at first the list OP provided, so my bad for that

[hudecof]

can you upgrade uvicorn please ? you're on 0.16.0 and it seems this has been fixed in 0.17.1

No, the 0.16.0 seems to be the latest version available for python 3.6. I did the upgrade, but the latest was the one in this discussion. So I need to switch to H11 worker to fix the issue.

[euri10]

mm ok, maybe time to upgrade python version, but I suppose you have your own set of constraints.

[hudecof]

for the rocky linux 8 the 3.6 python is the default one. The development is on python 3.7/Debian 11.
The migration to the 3.8, alternative python for RL8, should be seamless, but requires testing, so need to approval from the project.