Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

init RSS framework #551

Merged
merged 1 commit into from
Aug 21, 2024
Merged

init RSS framework #551

merged 1 commit into from
Aug 21, 2024

Conversation

richox
Copy link
Collaborator

@richox richox commented Aug 21, 2024

add RSS framework, for easier cooperating with 3rdparty remote shuffle service.

for supporting a new 3rdparty RSS serivice, one needs to:

  1. implement an RssPartitionWriter derived from RssPartitionWriterBase, which writes encoded columnar data to a specified shuffled partition.
  2. implement an RssShuffleWriter derived from BlazeRssShuffleWriterBase, which creates instances of RssPartitionWriter.
  3. implement an RssShuffleReader derived from BlazeRssShuffleReaderBase, which implements reading columnar/unsafe row data of specified shuffle, partition and maps.
  4. implement an RssShuffleManager derived from BlazeRssShuffleManagerBase, which registers/unregisters a shuffle, and cooperates with RssShuffleWriter/Reader.

@lihao712 lihao712 merged commit a69338b into master Aug 21, 2024
520 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants