Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I use it in linux and windows and the program gets stuck after 1-2 days of running, see the process is not killed, it just hibernates #155

Open
dyuzhou opened this issue May 6, 2022 · 0 comments

Comments

@dyuzhou
Copy link

dyuzhou commented May 6, 2022

I use uvicorn to start my Fastapi program, there is a long task in it, I use aiomultiprocess to speed up my httpx requests, but it always stops after the program runs for a while, I can't find the problem, nest Tried some configurations but nothing helped

Details

import httpx
from aiomultiprocess import Pool
from elasticsearch import AsyncElasticsearch
from fastapi import APIRouter, status, Depends, BackgroundTasks

from models import Article

api = APIRouter()

async def requests(data):
async with httpx.AsyncClient() as sess:
resp = await sess.post(settings.URL, data=data)
return resp.text

async def create_datas(data: list):
list_article = [Article(**item) for item in data]
await Article.bulk_create(list_article)

This function will be called multiple times in the service

async def run_tasks(start_id, es=None):
data = await get_datas(start_id, es) # get data from es
if data:
data_list = []
async with Pool(processes=6,
maxtasksperchild=800,
childconcurrency=6
) as pool:
async for result in pool.map(requests, data):
data_list.append(result)
await create_datas([i for i in data_list if i])

async def task_process(start, es):
while start < 140000000:
start_time = time.time()
await run_tasks(start, es)
all_time = time.time() - start_time
LOG.info(f'id: {start} - {start + 800} The time spent: {all_time}')
start += 800

@api.get('/tasks/')
async def trans2(background_tasks: BackgroundTasks, start: int, es: AsyncElasticsearch = Depends(get_es)):
if start:
background_tasks.add_task(task_process, start, es)
return my_response(data='', message='starting', code=200, sta=status.HTTP_200_OK)

  • OS: Windows or ubuntu
  • Python version: 3.8
  • aiomultiprocess version: 0.9.0

How can I configure it so that it can run stably?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant