Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

combine streaming and distributed in copy into #8128

Closed
2 tasks done
youngsofun opened this issue Oct 11, 2022 · 0 comments
Closed
2 tasks done

combine streaming and distributed in copy into #8128

youngsofun opened this issue Oct 11, 2022 · 0 comments
Labels
C-feature Category: feature

Comments

@youngsofun
Copy link
Member

youngsofun commented Oct 11, 2022

Summary

sketch:

//path_stream = list(dirs) 
while paths = path_stream.collect(n):
  splits = split(paths)
  for (part, node) in distribute(cluster.nodes, splits)
      node.fragment_exchanges.try_send(part) 

some considerations:

  1. reuse fragment_exchanges to send parts
  2. reuse static_exchanges too.
  3. fragment_exchanges may return detailed progress to help with fair distribution of
  4. add a setting enable_distributed_copy
  5. get head N of list(dirs), before starting progress, if < N, no need for streaming.

related issues:

distributed copy

streaming copy

@BohuTANG BohuTANG added the C-feature Category: feature label Oct 11, 2022
@BohuTANG BohuTANG closed this as completed Jun 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature
Projects
None yet
Development

No branches or pull requests

2 participants