Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Scannable support for Schedulers/Workers #1050

Closed
smaldini opened this issue Feb 1, 2018 · 1 comment
Closed

Add Scannable support for Schedulers/Workers #1050

smaldini opened this issue Feb 1, 2018 · 1 comment
Labels
type/enhancement A general enhancement
Milestone
3.2.0.M2

Comments

@smaldini
Copy link
Contributor

smaldini commented Feb 1, 2018

And link the scannable inside the operators using those.
@smaldini smaldini added this to the 3.2.0.RELEASE milestone Feb 1, 2018
@smaldini smaldini added the type/enhancement A general enhancement label Feb 1, 2018
@simonbasle
Copy link
Member

A few questions:

  1. Should we try to introduce a link between Worker and their PARENT Scheduler where there is none (eg by storing a WeakReference<Scheduler> in these Worker implementations that miss a ref to their original scheduler)?
    Often the Worker will have a reference to an Executor at best.
  2. Introspection of the Executor themselves is limited. One has to recognize a particular implementation with instanceof, cast and call the available getters. Is that something we want to do?

simonbasle added a commit that referenced this issue Feb 9, 2018
@simonbasle
fix #1050 Make Scheduler/Worker Scannable, meaningful toString
13abab5 
Most Schedulers are capable of reporting their NAME, CAPACITY (degree of
parallelism/maximum thread pool size), BUFFERED (number of pending
tasks), TERMINATED/CANCELLED (is the scheduler shutdown).

Additionally, Workers that keep a link to their Scheduler will report it
as their PARENT.

Schedulers delegate NAME resolution to their improved toString() method.

Added an util method to "scan" an Executor, recognizing
ThreadPoolExecutor and extracting info out of it.
simonbasle added a commit that referenced this issue Feb 9, 2018
@simonbasle
follow-up to #1050 Let operators report their Scheduler via RUN_ON (wip)
40010a8
@simonbasle simonbasle mentioned this issue Feb 9, 2018
Merged
simonbasle added a commit that referenced this issue Feb 12, 2018
@simonbasle
follow-up to #1050 Let operators report their Scheduler via RUN_ON (wip)
128c45e
simonbasle added a commit that referenced this issue Feb 12, 2018
@simonbasle
follow-up to #1050 Let operators report their Scheduler via RUN_ON
7dc3faf 
The new `Attr` RUN_ON allows operators that take a Scheduler or a Worker
(except aliases like `Flux#delayElements`) to report which Scheduler or
Worker they run on.
simonbasle added a commit that referenced this issue Feb 13, 2018
@simonbasle
fix #1050 Make Scheduler/Worker implementations Scannable
58ed955 
Most Schedulers concrete implementations are capable of reporting their
NAME, CAPACITY (degree of parallelism), BUFFERED (number of available
workers for parallel/elastic), TERMINATED/CANCELLED (is the scheduler
shutdown). Schedulers delegate NAME resolution to their improved
`toString()` methods.

Parallel and Elastic allow iterating over their backing executors via
the `inners()` method from `Scannable`.

Most Workers are capable of reporting their NAME (including the name of
the parent in some cases, but mostly a toString of their executor),
CAPACITY (maximum task queue size or maximum thread pool size),
BUFFERED (pending and active task count) and TERMINATED/CANCELLED.

Added an util method to "scan" an Executor, recognizing
ThreadPoolExecutor and extracting info out of it. All the schedulers
that defaulted to a singleThreadExecutor now use an introspectable
ThreadPoolExecutor with a maxPoolSize of 1.
simonbasle added a commit that referenced this issue Feb 13, 2018
@simonbasle
follow-up to #1050 Let operators report their Scheduler via RUN_ON
824bd7e 
The new `Attr` RUN_ON allows operators that take a Scheduler or a Worker
(except aliases like `Flux#delayElements`) to report which Scheduler or
Worker they run on.
simonbasle added a commit that referenced this issue Feb 14, 2018
@simonbasle
fix #1050 Make Scheduler/Worker implementations Scannable
08d55d4 
Most Schedulers concrete implementations are capable of reporting their
NAME, CAPACITY (degree of parallelism), BUFFERED (number of available
workers for parallel/elastic), TERMINATED/CANCELLED (is the scheduler
shutdown). Schedulers delegate NAME resolution to their improved
`toString()` methods.

Parallel and Elastic allow iterating over their backing executors via
the `inners()` method from `Scannable`.

Most Workers are capable of reporting their NAME (including the name of
the parent in some cases, but mostly a toString of their executor),
CAPACITY (maximum task queue size or maximum thread pool size),
BUFFERED (pending and active task count) and TERMINATED/CANCELLED.

Added an util method to "scan" an Executor, recognizing
ThreadPoolExecutor and extracting info out of it. All the schedulers
that defaulted to a singleThreadExecutor now use an introspectable
ThreadPoolExecutor with a maxPoolSize of 1.
simonbasle added a commit that referenced this issue Feb 14, 2018
@simonbasle
follow-up to #1050 Let operators report their Scheduler via RUN_ON
60d7219 
The new `Attr` RUN_ON allows operators that take a Scheduler or a Worker
(except aliases like `Flux#delayElements`) to report which Scheduler or
Worker they run on.
simonbasle added a commit that referenced this issue Feb 14, 2018
@simonbasle
follow-up to #1050 Let operators report their Scheduler via RUN_ON
dcc36e5 
The new `Attr` RUN_ON allows operators that take a Scheduler or a Worker
(except aliases like `Flux#delayElements`) to report which Scheduler or
Worker they run on.
@simonbasle simonbasle modified the milestones: 3.2.0.RELEASE, 3.2.0.M2 May 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement A general enhancement
Projects
None yet
Milestone
3.2.0.M2
Development

No branches or pull requests
2 participants
@smaldini @simonbasle