allow setting a "custom disconnect delay" of a worker, by clicking on…

[koo5]

… a button in worker list. The custom delay is taken into account when pruning workers.

Description

a new dropdown in the workers page, and a new variable custom_disconnect_delay in scheduler.py/Worker

Motivation and Context

My usecase are long running tasks with possibly intermittent connectivity. Therefore i configure luigi something like this:

[scheduler]
remove_delay = 88888888
worker_disconnect_delay = 88888888

[core]
rpc-retry-attempts = 88888888

But when i am certain that a worker is not coming back, i need to inform the scheduler about this. Rather than adding a special "remove" button, this PR lets you override the disconnect delay value, and automatic pruning does the rest.

This PR is somewhat preliminary:

• the gui needs some tweaks, specifically, the null value is displayed as an empty string upon page load.
• on('click', '#btn-set-worker-custom-disconnect-delay-**** could probably be abstracted
• maybe Worker.custom_disconnect_delay would be better stored in Worker.info?

I played with this code and it seems to do the job.

[dlstadther]

I like the idea here and I'm happy to review the python. However, I'm not much use when it comes to the concerns you mention with the UI

• the gui needs some tweaks, specifically, the null value is displayed as an empty string upon page load.
• on('click', '#btn-set-worker-custom-disconnect-delay-**** could probably be abstracted
• maybe Worker.custom_disconnect_delay would be better stored in Worker.info?

[dlstadther]

Now that Luigi is Python 3.6+, consider using .format() strings or f-strings, instead of the legacy %-strings.

[dlstadther]

Related to you question of if custom_disconnect_delay should live as an attribute of Worker... It does seem a little odd here since all the other attributes of this class are related to tracking work.

Given that the value this would be overriding, config.worker_disconnect_delay, is a scheduler config value, that perhaps this configuration should be a scheduler rpc update. Thoughts? (it is already a bit confusing the overlap between scheduler and worker config).

[dlstadther]

thoughts on custom disconnect delay defaulting to the scheduler config, such that this value isn't ever null? (assuming this is the null value you mention was displaying as an empty string)

[koo5]

You lost me at

this configuration should be a scheduler rpc update.

However, i found a reasonably elegant way to save the frontend from dealing with the possible Null value. worker_list can obtain the current value.

I don't see a better way right now. I'm not sure if there's a good way to make Scheduler construct Workers with a configuration value. Also, it seems that it would be complicated to make Worker weakref its Scheduler (to obtain scheduler config in, say, a property getter).

koo5@4c220d6

Also, nevermind about using Worker.info, as that seems to be for passing information from workers to scheduler.