High CPU usage after workers are idle for a while

[nhumrich]

I am using aiohttp with gunicorn using the aiohttp worker

gunicorn my_app_module:my_web_app -w 2 --bind localhost:8080 --worker-class aiohttp.worker.GunicornWebWorker

I have also tried the aiohttp.worker.GunicornUVLoopWebWorker as well.
Gunicorn works great, and I have even done load testing, and everything is ok, then I leave the application running on a server, and after a few hours of sitting idle, the CPU starts going high. Upon running top I see that one specific worker is taking a lot of CPU, and the other is fine. If I try to kill the specific process, it is unresponsive, and I have to kill -9 it. If I kill the process, a new one starts up and all is well. I can also restart gunicorn and everything is good.
This has happened a couple of times.

Mostly, I just dont know how to investigate further. Is there anything I can do to get more information?
Any ideas what could be causing the issue?

[asvetlov]

Just enabling statistical profiler like https://vmprof.readthedocs.io/en/latest/ or https://pypi.python.org/pypi/statprof/ may point on the problem.

[nhumrich]

Sorry im a noob. But both of these look like they only work directly on python files. Since I startup with gunicorn, how do I profile?

[asvetlov]

You might enable vmprof via API: https://vmprof.readthedocs.io/en/latest/vmprof.html#module-level-functions and save result into file. Perhaps unix signals is the most convenient way for enabling/disabling profiling.
Also you might use dedicated debug endpoints of your web server, but it's tricky for multiprocess system.

The problem might be in your code, it's not necessary gunicorn or aiohttp (or whatever library used by you) -- but maybe one of these libraries have a bug.

[nhumrich]

Got some more info:
http://vmprof.com/#/92a70314999fc3e52397165d421196a4

Looks like its constantly restarting workers? not sure, but almost everything being called is by gunicorn. The only other thing is "expire cookie" from aiohttp.helpers. I am not even using aiohttp cookies, so im not sure why thats getting called.

[asvetlov]

No, it's aio-libs/aiohttp#1061

The issue is not related to gunicorn.
Closing.