Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] New executor API prototype #30222

Closed
wants to merge 26 commits into from
Closed

[WIP] New executor API prototype #30222

wants to merge 26 commits into from

Conversation

ericl
Copy link
Contributor

@ericl ericl commented Nov 11, 2022 •
edited
Loading

Why are these changes needed?

This PR prototypes a new execution backend for Datasets that supports flexible switching between bulk synchronous and pipelined execution modes.

This PR is for internal discussion only. A REP will follow that defines the motivation and formal design.

Code organization:

  • execution/interfaces.py -- boundary interfaces between the execution module and the rest of Datasets
    • RefBundle, PhysicalOperator, Executor, OneToOneOperator, AllToAllOperator, ExchangeOperator
  • execution/operators.py -- operator implementation
    • InputDataBuffer, MapOperator, ...
  • execution/bulk_executor.py -- bulk executor implementation
    • BulkExecutor

TODO:

  • prototype basic synchronous execution
  • prototype PipelinedExecutor implementation
  • prototype integration with ComputeStrategy
  • prototype AllToAll operator implementation
  • prototype Datasets plan integration

@ericl
wip
85f3936 
Signed-off-by: Eric Liang <[email protected]>

write docs

Signed-off-by: Eric Liang <[email protected]>

wip

Signed-off-by: Eric Liang <[email protected]>

wip

Signed-off-by: Eric Liang <[email protected]>
@ericl
add basic execution test
ba64aa3 
Signed-off-by: Eric Liang <[email protected]>
@ericl
typo
cdf4adc 
Signed-off-by: Eric Liang <[email protected]>
@ericl
doc compute strategy
ed36a06 
Signed-off-by: Eric Liang <[email protected]>
@ericl
refactor to be unit testable
6128965 
Signed-off-by: Eric Liang <[email protected]>
@ericl
add progress bar
ed1718d 
Signed-off-by: Eric Liang <[email protected]>
@ericl
add basic priority policy
d3b61fc 
Signed-off-by: Eric Liang <[email protected]>
@ericl
remove sleep
c7f9756 
Signed-off-by: Eric Liang <[email protected]>
@ericl
cache desc
7b0422f 
Signed-off-by: Eric Liang <[email protected]>
@ericl
rename buffer to exchange
44f88ac 
Signed-off-by: Eric Liang <[email protected]>
@ericl
update docs
d25c183 
Signed-off-by: Eric Liang <[email protected]>
@ericl
add execution option support for pipeline execution
a4a8951 
Signed-off-by: Eric Liang <[email protected]>
@ericl
remove affinity print
4565314 
Signed-off-by: Eric Liang <[email protected]>
@ericl
fetch local false
5993253 
Signed-off-by: Eric Liang <[email protected]>
@ericl
remove sleep
ee74977 
Signed-off-by: Eric Liang <[email protected]>
@ericl
move reporting up
9a39851 
Signed-off-by: Eric Liang <[email protected]>
@ericl
add bundle sizes
ca43867 
Signed-off-by: Eric Liang <[email protected]>
@ericl
add actor pool
612f8cc 
Signed-off-by: Eric Liang <[email protected]>
@ericl
fix num gpus
f23df9c 
Signed-off-by: Eric Liang <[email protected]>
@ericl
add spread
b3b6b07 
Signed-off-by: Eric Liang <[email protected]>
@ericl
Merge remote-tracking branch 'upstream/master' into executor-refactor
f318134
@ericl
wip
076dd2f 
Signed-off-by: Eric Liang <[email protected]>
@ericl
wip
3cb5440 
Signed-off-by: Eric Liang <[email protected]>
@ericl
wip
72dab9f 
Signed-off-by: Eric Liang <[email protected]>
@ericl
passing basic tests
87ba983 
Signed-off-by: Eric Liang <[email protected]>
@ericl ericl closed this Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scottjlee scottjlee Awaiting requested review from scottjlee scottjlee will be requested when the pull request is marked ready for review scottjlee is a code owner

@bveeramani bveeramani Awaiting requested review from bveeramani bveeramani will be requested when the pull request is marked ready for review bveeramani is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen will be requested when the pull request is marked ready for review raulchen is a code owner

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang stephanie-wang will be requested when the pull request is marked ready for review stephanie-wang is a code owner

@omatthew98 omatthew98 Awaiting requested review from omatthew98 omatthew98 will be requested when the pull request is marked ready for review omatthew98 is a code owner

@alexeykudinkin alexeykudinkin Awaiting requested review from alexeykudinkin alexeykudinkin will be requested when the pull request is marked ready for review alexeykudinkin is a code owner

@srinathk10 srinathk10 Awaiting requested review from srinathk10 srinathk10 will be requested when the pull request is marked ready for review srinathk10 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ericl