Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allow setting a "custom disconnect delay" of a worker, by clicking on… #3225

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

allow setting a "custom disconnect delay" of a worker, by clicking on… #3225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions luigi/scheduler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,10 +353,14 @@ def __init__(self, worker_id, last_active=None):
self.info = {}
self.disabled = False
self.rpc_messages = []
self.custom_disconnect_delay = None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to you question of if custom_disconnect_delay should live as an attribute of Worker... It does seem a little odd here since all the other attributes of this class are related to tracking work.

Given that the value this would be overriding, config.worker_disconnect_delay, is a scheduler config value, that perhaps this configuration should be a scheduler rpc update. Thoughts? (it is already a bit confusing the overlap between scheduler and worker config).


def add_info(self, info):
self.info.update(info)

def set_custom_disconnect_delay(self, delay):
self.custom_disconnect_delay = delay

def update(self, worker_reference, get_work=False):
if worker_reference:
self.reference = worker_reference
Expand All @@ -366,7 +370,12 @@ def update(self, worker_reference, get_work=False):

def prune(self, config):
# Delete workers that haven't said anything for a while (probably killed)
if self.last_active + config.worker_disconnect_delay < time.time():
if self.custom_disconnect_delay:
disconnect_delay = self.custom_disconnect_delay
else:
disconnect_delay = config.worker_disconnect_delay
if self.last_active + disconnect_delay < time.time():
logger.debug("Worker %s timed out (no contact for >=%ss)", self, disconnect_delay)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that Luigi is Python 3.6+, consider using .format() strings or f-strings, instead of the legacy %-strings.

return True

def get_tasks(self, state, *statuses):
Expand Down Expand Up @@ -718,7 +727,6 @@ def _prune_workers(self):
remove_workers = []
for worker in self._state.get_active_workers():
if worker.prune(self._config):
logger.debug("Worker %s timed out (no contact for >=%ss)", worker, self._config.worker_disconnect_delay)
remove_workers.append(worker.id)

self._state.inactivate_workers(remove_workers)
Expand Down Expand Up @@ -963,6 +971,10 @@ def disable_worker(self, worker):
def set_worker_processes(self, worker, n):
self._state.get_worker(worker).add_rpc_message('set_worker_processes', n=n)

@rpc_method()
def set_worker_custom_disconnect_delay(self, worker, n):
self._state.get_worker(worker).set_custom_disconnect_delay(n)

@rpc_method()
def send_scheduler_message(self, worker, task, content):
if not self._config.send_messages:
Expand Down Expand Up @@ -1471,6 +1483,7 @@ def worker_list(self, include_running=True, **kwargs):
state=worker.state,
first_task_display_name=self._first_task_display_name(worker),
num_unread_rpc_messages=len(worker.rpc_messages),
custom_disconnect_delay=worker.custom_disconnect_delay,
**worker.info
) for worker in self._state.get_active_workers()]
workers.sort(key=lambda worker: worker['started'], reverse=True)
Expand Down
64 changes: 41 additions & 23 deletions luigi/static/visualiser/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,35 +167,53 @@ <h4 class="modal-title" id="disiableWorkerLabel">Disable worker?</h4>
{{/is_disabled}}
<div class="box-header with-border">
<h3 class="box-title">{{name}}</h3>
<div class="box-tools pull-right">
{{#num_unread_rpc_messages}}
<span class="label-unread-worker-messages">{{num_unread_rpc_messages}} unread message(s)</span>
{{/num_unread_rpc_messages}}
{{^is_disabled}}
{{#workers}}
<div class="btn-group">
<button type="button" class="btn btn-sm btn-default dropdown-toggle btn-set-workers" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Workers: <span id="label-n-workers" data-worker="{{name}}">{{workers}}</span> <span class="caret"></span>
<div class="pull-right">
<span class="btn-group">
<button type="button" class="btn btn-sm btn-default dropdown-toggle btn-set-worker-custom-disconnect-delay" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Custom disconnect delay: <span id="label-custom-disconnect-delay" data-worker="{{name}}">{{custom_disconnect_delay}}</span> <span class="caret"></span>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts on custom disconnect delay defaulting to the scheduler config, such that this value isn't ever null? (assuming this is the null value you mention was displaying as an empty string)

</button>
<ul class="dropdown-menu">
<li><a href="#" id="btn-increment-workers" data-worker="{{name}}">
<i class="glyphicon glyphicon-plus"></i> Add 1 worker
<li><a href="#" id="btn-set-worker-custom-disconnect-delay-no" data-worker="{{name}}">
no
</a></li>
<li><a href="#" id="btn-decrement-workers" data-worker="{{name}}">
<i class="glyphicon glyphicon-minus"></i> Remove 1 worker
<li><a href="#" id="btn-set-worker-custom-disconnect-delay-60" data-worker="{{name}}">
60s
</a></li>
<li><a href="#" id="btn-set-workers" data-toggle="modal" data-target="#setWorkersModal" data-worker="{{name}}">
<i class="glyphicon glyphicon-pencil"></i> Set workers ...
<li><a href="#" id="btn-set-worker-custom-disconnect-delay-3600" data-worker="{{name}}">
3600s
</a></li>
</ul>
</div>
{{/workers}}
<div class="button-tooltip" data-toggle="tooltip" title="Disable Worker">
<button type="button" class="btn btn-sm btn-danger btn-disable-worker" data-toggle="modal" data-target="#disableWorkerModal" data-worker="{{name}}">
<i class="fa fa-fire-extinguisher"></i>
</button>
</div>
{{/is_disabled}}
</span>
<span class="box-tools">
{{#num_unread_rpc_messages}}
<span class="label-unread-worker-messages">{{num_unread_rpc_messages}} unread message(s)</span>
{{/num_unread_rpc_messages}}
{{^is_disabled}}
{{#workers}}
<div class="btn-group">
<button type="button" class="btn btn-sm btn-default dropdown-toggle btn-set-workers" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
Workers: <span id="label-n-workers" data-worker="{{name}}">{{workers}}</span> <span class="caret"></span>
</button>
<ul class="dropdown-menu">
<li><a href="#" id="btn-increment-workers" data-worker="{{name}}">
<i class="glyphicon glyphicon-plus"></i> Add 1 worker
</a></li>
<li><a href="#" id="btn-decrement-workers" data-worker="{{name}}">
<i class="glyphicon glyphicon-minus"></i> Remove 1 worker
</a></li>
<li><a href="#" id="btn-set-workers" data-toggle="modal" data-target="#setWorkersModal" data-worker="{{name}}">
<i class="glyphicon glyphicon-pencil"></i> Set workers ...
</a></li>
</ul>
</div>
{{/workers}}
<div class="button-tooltip" data-toggle="tooltip" title="Disable Worker">
<button type="button" class="btn btn-sm btn-danger btn-disable-worker" data-toggle="modal" data-target="#disableWorkerModal" data-worker="{{name}}">
<i class="fa fa-fire-extinguisher"></i>
</button>
</div>
{{/is_disabled}}
</span>
</div>
</div>
<div class="box-body">
Expand Down
7 changes: 7 additions & 0 deletions luigi/static/visualiser/js/luigi.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,13 @@ var LuigiAPI = (function() {
});
};

LuigiAPI.prototype.setWorkerCustomDisconnectDelay = function(workerId, n, callback) {
var data = {worker: workerId, n: n};
jsonRPC(this.urlRoot + "/set_worker_custom_disconnect_delay", data, function(response) {
callback();
});
};

LuigiAPI.prototype.sendSchedulerMessage = function(workerId, taskId, content, callback) {
var data = {worker: workerId, task: taskId, content: content};
jsonRPC(this.urlRoot + "/send_scheduler_message", data, function(response) {
Expand Down
33 changes: 33 additions & 0 deletions luigi/static/visualiser/js/visualiserApp.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1041,6 +1041,21 @@ function visualiserApp(luigi) {
});
}

/**
* Updates the custom disconnect delay of a worker
* @param worker: the id of the worker
* @param n: delay, in seconds
*/
function updateWorkerCustomDisconnectDelay(worker, n) {
// the spinner is just for visual feedback
var $label = $('#workerList').find('#label-custom-disconnect-delay[data-worker="' + worker + '"]');
$label.html('<i class="fa fa-spinner fa-spin" aria-hidden="true"></i>');

luigi.setWorkerCustomDisconnectDelay(worker, n, function() {
$label.text(n);
});
}

/**
* Updates the number of units of a given resource available in the scheduler
* @param resource: the name of the resource
Expand Down Expand Up @@ -1428,6 +1443,24 @@ function visualiserApp(luigi) {
$event.preventDefault();
});

$('#workerList').on('click', '#btn-set-worker-custom-disconnect-delay-no', function($event) {
var worker = $(this).data("worker");
updateWorkerCustomDisconnectDelay(worker, null);
$event.preventDefault();
});

$('#workerList').on('click', '#btn-set-worker-custom-disconnect-delay-60', function($event) {
var worker = $(this).data("worker");
updateWorkerCustomDisconnectDelay(worker, 60);
$event.preventDefault();
});

$('#workerList').on('click', '#btn-set-worker-custom-disconnect-delay-3600', function($event) {
var worker = $(this).data("worker");
updateWorkerCustomDisconnectDelay(worker, 3600);
$event.preventDefault();
});

$('#workerList').on('show.bs.modal', '#setWorkersModal', function($event) {
$('#setWorkersButton').data('worker', $($event.relatedTarget).data('worker'));
var $input = $(this).find('#setWorkersInput').on('keypress', function($event) {
Expand Down