ProcessPoolExecutor in background cause freeze after keep-alive timeout #1883

voronin-opensoft · 2017-05-11T22:52:10Z

Long story short

If we run heavy task in ProcessPoolExecutor in background, aiohttp don't handle request after some timeout. It happens after ~75 seconds. It's equal keepalive_timeout value at https://github.com/aio-libs/aiohttp/blob/2.0.7/aiohttp/web_protocol.py#L82
If we change keepalive_timeout value we get freeze after specific time.

Expected behaviour

Handle all requests.

Actual behaviour

Freeze on request after keep-alive timeout.

Steps to reproduce

We set keep-alive timeout to 1 second for faster bug reproducing.

Run server

import asyncio
import logging
from multiprocessing import Manager
from concurrent.futures import ProcessPoolExecutor

from aiohttp.web import Application, Response, run_app


logging.basicConfig(level=logging.DEBUG)


async def index(request):
    body = '''<!DOCTYPE html>
    <html>
    <body>
    <script>
        function request(path) {
            var xhr = new XMLHttpRequest();
            
            xhr.open('GET', path, false);
            xhr.onload = function (e) {
                if (xhr.readyState == 4 && xhr.status == 200) {
                    console.log(xhr.responseText);
                }
            }
            xhr.send(null);
        }
        
        function start() {
            return request('/start');
        }
        
        function stop() {
            return request('/stop');
        }
    </script>
    <button style="width: 200px; height: 100px;" onclick="start()"> Start </button>
    <button style="width: 200px; height: 100px;" onclick="stop()"> Stop </button>
    </body>
    </html>
    '''
    return Response(body=body, content_type='document')


async def start(request):
    await request.app['worker'].queue.put('data')
    return Response(body='Start')


async def stop(request):
    request.app['worker'].stop_event.set()
    return Response(body='Stop')


def hard_work(stop_event):
    while True:
        if stop_event.is_set():
            break


class Worker:

    async def start(self, loop):
        self.queue = asyncio.Queue(loop=loop)

        while True:
            data = await self.queue.get()

            executor = ProcessPoolExecutor()
            manager = Manager()
            stop_event = manager.Event()
            self.stop_event = stop_event

            hard_future = loop.run_in_executor(executor, hard_work, stop_event)

            try:
                await asyncio.wait_for(hard_future, timeout=999999, loop=loop)
            except asyncio.TimeoutError:
                print('Time out')
                stop_event.set()
            else:
                print('Done')


def get_app():
    app = Application(handler_args={'keepalive_timeout': 1})

    app.router.add_get('/', index)
    app.router.add_get('/start', start)
    app.router.add_get('/stop', stop)


    async def start_worker(app):
        app.loop.set_debug(True)

        worker = Worker()
        app['worker'] = worker
        app.loop.create_task(worker.start(app.loop))

    app.on_startup.append(start_worker)

    return app


def main():
    app = get_app()
    run_app(app, host='0.0.0.0', port=5000)


if __name__ == '__main__':
    main()

Make first /start request. Wait for keep-alive timeout. Make second /stop request. aiohttp will not handle second /stop request.

import time
import asyncio

import aiohttp


async def do_requests():
    session = aiohttp.ClientSession()

    print('GET /start')
    response = await session.get('http://127.0.0.1:5000/start')
    print('Response:', await response.text())

    wait = 3
    print('Sleep in {} seconds'.format(wait))
    await asyncio.sleep(wait)

    print('GET /stop')
    response = await session.get('http://127.0.0.1:5000/stop')
    print('Response:', await response.text())


def main():
    loop = asyncio.get_event_loop()
    loop.run_until_complete(do_requests())


if __name__ == '__main__':
    main()

Your environment

Debian 8 Jessie
Python 3.5.3
aiohttp == 2.0.7

The text was updated successfully, but these errors were encountered:

fafhrd91 · 2017-05-12T17:34:37Z

I do not think this is aiohttp problem.

here is what is going on:

requests opens connection and make /start request
connection remains open
aiohttp drops connection after 75 seconds
requests could not detect connection dropped and use it to send data. aiohttp does not get any requests

to confirm, you can make /stop request with curl after 80 seconds
or you can force close connection on aiohttp side.

fafhrd91 · 2017-05-12T17:35:38Z

try aiohttp.client

voronin-opensoft · 2017-05-12T17:57:27Z

This works with curl cause it closes connections. Same bug can be reproduced with Chrome and Firefox cause they use keep-alive. We can set small keep-alive timeout for easier bug reproducing.
If we use app = Application(handler_args={'keepalive_timeout': None}) there is no bug.

voronin-opensoft · 2017-05-12T18:11:26Z

Yes. We can reproduce it using aiohttp.ClientSession()

fafhrd91 · 2017-05-12T18:11:44Z

I see. It seems related how tcp works in general.

fafhrd91 · 2017-05-12T18:17:18Z

fix in #1884
I am not sure about correctness

voronin-opensoft · 2017-05-12T18:48:17Z

It works in Chrome, Firefox, requests but not in aiohttp.client

fafhrd91 · 2017-05-12T19:20:58Z

this seems bug in asyncio default event loop. works with uvloop.

@1st1 ^^^

fafhrd91 · 2017-05-12T19:23:57Z

async-tokio loop does not work as well. seems special treatment is needed for HUP event

1st1 · 2017-05-12T19:31:02Z

this seems bug in asyncio default event loop

would be great if someone could submit a bug report on bugs.python.org.

fafhrd91 · 2017-05-12T19:33:22Z

will do, but I want to play with implementation first

asvetlov · 2018-02-25T14:20:08Z

@fafhrd91 could you provide more details?

fafhrd91 · 2018-02-25T17:11:00Z

If I rember right, this is problem of the default event loop, it does not react on socket hup event after timeout or something like that. So aiohttp can not close socket until it tries to write to it something. But uvloop works right. This is reproducible.

asvetlov · 2020-10-22T07:33:11Z

The issue is stale.
Sorry, closing.

fafhrd91 added a commit that referenced this issue May 12, 2017

Write to transport rn before closing after keepalive timeout #1883

d701f80

fafhrd91 added a commit that referenced this issue May 12, 2017

Write to transport rn before closing after keepalive timeout #1883

bf49f8e

fafhrd91 mentioned this issue May 12, 2017

Could not catch socket HUP event PyO3/tokio#38

Open

kxepal mentioned this issue Mar 1, 2018

Unsolicited response received on idle HTTP channel starting with "\r\n"; err=<nil> #2782

Closed

aio-libs-bot mentioned this issue Dec 6, 2018

Timeout on one connection can cause CancelledError on other connections on DNS refresh #3431

Closed

asvetlov closed this as completed Oct 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ProcessPoolExecutor in background cause freeze after keep-alive timeout #1883

ProcessPoolExecutor in background cause freeze after keep-alive timeout #1883

voronin-opensoft commented May 11, 2017 •

edited

Loading

fafhrd91 commented May 12, 2017

fafhrd91 commented May 12, 2017

voronin-opensoft commented May 12, 2017

voronin-opensoft commented May 12, 2017

fafhrd91 commented May 12, 2017

fafhrd91 commented May 12, 2017

voronin-opensoft commented May 12, 2017

fafhrd91 commented May 12, 2017

fafhrd91 commented May 12, 2017

1st1 commented May 12, 2017

fafhrd91 commented May 12, 2017

asvetlov commented Feb 25, 2018

fafhrd91 commented Feb 25, 2018

asvetlov commented Oct 22, 2020

ProcessPoolExecutor in background cause freeze after keep-alive timeout #1883

ProcessPoolExecutor in background cause freeze after keep-alive timeout #1883

Comments

voronin-opensoft commented May 11, 2017 • edited Loading

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

fafhrd91 commented May 12, 2017

fafhrd91 commented May 12, 2017

voronin-opensoft commented May 12, 2017

voronin-opensoft commented May 12, 2017

fafhrd91 commented May 12, 2017

fafhrd91 commented May 12, 2017

voronin-opensoft commented May 12, 2017

fafhrd91 commented May 12, 2017

fafhrd91 commented May 12, 2017

1st1 commented May 12, 2017

fafhrd91 commented May 12, 2017

asvetlov commented Feb 25, 2018

fafhrd91 commented Feb 25, 2018

asvetlov commented Oct 22, 2020

voronin-opensoft commented May 11, 2017 •

edited

Loading