Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Close async generators created by Executor on Golem shutdown #482

Merged
merged 1 commit into from
Jun 22, 2021

Conversation

azawlocki
Copy link
Contributor

@azawlocki azawlocki commented Jun 22, 2021

Resolves #481

The following scenario leads to a deadlock:

async with Golem(...) as golem:

    completed = golem.execute_tasks(...)
   
    async for task in completed:
        break  
        # or raise exception, or return from function

On shutdown, Golem is waiting for all registered Jobs to be marked as finished. However, a Job created by Executor.submit() (called from execute_tasks()) is marked as finished only after all tasks are retrieved from the associated async generator (here: completed).

To fix this, in this PR we register each async generator created by Executor.submit() in a Golem instance used by the Executor. When the instance shuts down, it calls aclose() on all registered generators before waiting for the registered jobs.

@azawlocki azawlocki requested a review from a team June 22, 2021 07:31
@azawlocki azawlocki self-assigned this Jun 22, 2021
@azawlocki azawlocki requested a review from kmazurek June 22, 2021 07:32
@azawlocki azawlocki force-pushed the az/close-golem-generators branch from 6db8f9d to 9232e50 Compare June 22, 2021 07:53
@azawlocki azawlocki requested a review from shadeofblue June 22, 2021 07:53
@azawlocki azawlocki merged commit e78ae28 into b0.6 Jun 22, 2021
@azawlocki azawlocki deleted the az/close-golem-generators branch June 22, 2021 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants