Blue/ensure handlers are async generators

[shadeofblue]

closes #655

[johny-b]

I made start in simple_service not-an-asyncgen and that's how my logs look like:

[2021-09-17T13:08:32.782+0200 INFO yapapi] Using log file `simple-service-yapapi-2021-09-17_13.08.32.log`; in case of errors look for additional information there
yapapi version: 0.7.0-dev
Using subnet: None, payment driver: zksync, and network: rinkeby

Starting 1 instance...
instances: []
[2021-09-17T13:08:38.361+0200 INFO yapapi.summary] [Job 1] Agreement proposed to provider 'EBGolem-1'
[2021-09-17T13:08:38.785+0200 INFO yapapi.summary] Received proposals from 3 providers so far
[2021-09-17T13:08:39.131+0200 INFO yapapi.summary] [Job 1] Agreement confirmed by provider 'EBGolem-1'
instances: []
[2021-09-17T13:08:41.586+0200 INFO yapapi.services] <SimpleService: simple-service-1> commissioned
[2021-09-17T13:08:41.590+0200 INFO yapapi.summary] [Job 1] Task started on provider 'EBGolem-1', task data: Service: SimpleService
[2021-09-17T13:08:41.882+0200 INFO yapapi.summary] [Job 1] Terminated agreement with EBGolem-1
[2021-09-17T13:08:43.289+0200 INFO yapapi.summary] [Job 1] Accepted invoice from 'EBGolem-1', amount: 0
instances: ['simple-service-1: starting on EBGolem-1']
instances: ['simple-service-1: starting on EBGolem-1']
[2021-09-17T13:08:50.791+0200 INFO yapapi.summary] Received proposals from 5 providers so far
instances: ['simple-service-1: starting on EBGolem-1']
[2021-09-17T13:08:56.794+0200 INFO yapapi.summary] Received proposals from 7 providers so far
[2021-09-17T13:08:59.797+0200 INFO yapapi.summary] Received proposals from 11 providers so far
instances: ['simple-service-1: starting on EBGolem-1']
instances: ['simple-service-1: starting on EBGolem-1']
instances: ['simple-service-1: starting on EBGolem-1']
^CShutting down gracefully, please wait a short while or press Ctrl+C to exit immediately...
[2021-09-17T13:09:10.386+0200 INFO yapapi.executor] Golem is shutting down...
[2021-09-17T13:09:10.386+0200 INFO yapapi.executor] All jobs have finished
[2021-09-17T13:09:10.387+0200 INFO yapapi.executor] Waiting for Golem services to finish...
[2021-09-17T13:09:10.397+0200 INFO yapapi.summary] Total cost: 0
[2021-09-17T13:09:10.397+0200 ERROR yapapi.summary] Error when shutting down Golem engine: CancelledError()
Task exception was never retrieved
future: <Task finished name='Task-15' coro=<Cluster.spawn_instance() done, defined at /home/jbetley/yapapi/yapapi/services.py:728> exception=TypeError("__init__() missing 1 required positional argument: 'job_id'")>
Traceback (most recent call last):
  File "/home/jbetley/yapapi/yapapi/services.py", line 786, in spawn_instance
    await task
  File "/home/jbetley/yapapi/yapapi/engine.py", line 573, in worker_task
    await run_worker(agreement, activity, work_context)
  File "/home/jbetley/yapapi/yapapi/services.py", line 761, in _worker
    await self._engine.process_batches(
  File "/home/jbetley/yapapi/yapapi/engine.py", line 588, in process_batches
    script: Script = await batch_generator.__anext__()
  File "/home/jbetley/yapapi/yapapi/services.py", line 640, in _run_instance
    handler = self._get_handler(instance)
  File "/home/jbetley/yapapi/yapapi/services.py", line 599, in _get_handler
    raise ServiceError(
yapapi.services.ServiceError: Service handler: `<bound method SimpleService.start of <SimpleService: simple-service-1>>` must be an asynchronous generator.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jbetley/yapapi/yapapi/services.py", line 801, in spawn_instance
    self.emit(events.WorkerFinished(agr_id=agreement_id, exc_info=sys.exc_info()))
TypeError: __init__() missing 1 required positional argument: 'job_id'

Note the Ctrl+C.
I think the important change would be to avoid instance being stuck in starting state forever.

Also it would be nice to make the error visible before the program stops, but that's less important.

Also, what's with the __init__() missing 1 required positional argument: 'job_id'? I understand this is probably not this PR, but maybe we should check this?

[shadeofblue]

I think the important change would be to avoid instance being stuck in starting state forever.

well, I think that's beyond the scope of this pull request... generally, what I wanted to achieve is to give a developer a chance to notice their error without having to guess....

and what you're observing is, I think, a symptom of yet another issue (I dare say regression) in instance handling where a failure doesn't cause the instance to be terminated ... I'm unsure whether that had been caused by @zakaprov 's changes to the script handling or by something else but I believe it's a new behavior...

[shadeofblue]

@johny-b added an issue #666

[johny-b]

Just a minor suggestion of a more-readable error.