Executor shutdown takes long time (like 30 minutes)

[mbenke]

Example:

[2021-04-01 13:30:41,677 ERROR yapapi.summary] Computation failed, reason: (408)
... [destroying activities]
[2021-04-01 13:30:44,287 INFO yapapi.executor] Waiting for 28 workers to finish...
[2021-04-01 13:30:52,033 DEBUG yapapi.executor] Terminating agreements...
...
[2021-04-01 13:36:12,718 DEBUG yapapi.executor] 1 service still running: {<Task pending coro=<Executor._submit.<locals>.find_offers() running at /root/miniconda3/envs/rdkit\
/lib/python3.7/site-packages/yapapi/executor/__init__.py:445> wait_for=<Future cancelled>>}
[2021-04-01 13:36:12,718 DEBUG yapapi.executor] Cancelling task: <Task pending coro=<Executor._submit.<locals>.find_offers() running at /root/miniconda3/envs/rdkit/lib/pyth\
on3.7/site-packages/yapapi/executor/__init__.py:445> wait_for=<Future cancelled>>
...
[NB lots of ProposalReceived/ProposalResponded/ProposalFailed here]
...
[2021-04-01 14:02:44,688 DEBUG yapapi.executor] Executor is shutting down...
[2021-04-01 14:02:44,759 DEBUG yapapi.events] ShutdownFinished(exc_info=None)
[2021-04-01 14:02:44,760 INFO yapapi.summary] Total cost: 12.15269563619010337
[2021-04-01 14:02:44,760 INFO yapapi.summary] Executor has shut down
[2021-04-01 14:02:44,761 DEBUG yapapi.executor.utils] AsyncWrapper's worker task cancelled

It occurs randomly, but often enough to be noticable (and painfil in long-running computations)

[filipgolem]

So, find_offers is still running, it is then cancelled and there are lots of Proposal* events before it's stopped.
golemfactory/yapapi#306 may fix this problem.

[mbenke]

Full log:

yagna-issue101.log.gz

[azawlocki]

A fix is here: golemfactory/yapapi#308

[mateuszsrebrny]

@azawlocki so - it should be fixed, right? should I close the issue now?

[mateuszsrebrny]

the fix is merged, packages are released closing the issue