Set default project for WorkerJob

[pmrv]

The WorkerJob needs an input project assigned, but the default is None and it's not checked before the worker job starts running, which leads to silent death of the worker if you forget to assign one. I've set the default now to be the project of the worker itself, because that seems to me the common use case. If that's too specific we should instead add a validate_ready_to_run method that does this check.

[pmrv]

Mh, I've noticed now that this creates the problem that the worker is waiting for itself to finish, which is also a bit borked...

[jan-janssen]

That is strange - I initially had an implementation where the worker was watching a specific project, but by now it should only filter based on the job_id. So the worker should not wait for itself.

[pmrv]

That is strange - I initially had an implementation where the worker was watching a specific project, but by now it should only filter based on the job_id. So the worker should not wait for itself.

Ah, I think I just got confused by the worker log. I had tried to create & submit my worker job, submit all the smaller calculations and then immediately set the worker status to collect, hoping that this would shut down the worker once all the jobs are finished. Alas, when the worker started running on the node it changed its status away from collect to running and therefore did not stop once it was done. This is a little bit of a pain point, but I'll make a separate issue for this + some refactors.

The new default works for me, if you don't have objections I'll merge this later today.

[jan-janssen]

I guess what you are looking for is the wait_for_worker() function. It stops the worker once all calculation are finished and no new jobs are submitted plus the timeout.

[jan-janssen]

Looks good to me

[pmrv]

I guess what you are looking for is the wait_for_worker() function. It stops the worker once all calculation are finished and no new jobs are submitted plus the timeout.

The issue was that I didn't want to block the notebook, but continue working on other things, but I haven't come up with anything better yet.