Investigate and propose parallel sort-merge join

[qw4990]

Description

In this PR(#14238), we introduce the shuffle executor.
We may use this shuffle executor to implement the parallel sort-merge join by nesting some sort and merge-join executors into a shuffle executor:

ShuffleExec
    - MergeJoinExec
        - SortExec
    - MergeJoinExec
        - SortExec
    ...

Compared with the hash-join we implement now, whose building phase is not parallel, it may utilize more CPU resources to get better performance.

• Modify the original shuffle to let it support multiple datasources and pass all existed tests
• Introduce a new variable to control concurrency of merge_join
• Support shuffle + merge_join
• Add some correct tests and benchmarks about shuffle + merge_join

Score

6600

Mentor

@qw4990 (Slack ID: Zhang Yuanjia)

Recommend Skills

• Golang

[hftsin]

/pick-up

[ti-challenge-bot]

It is not a pickable issue!

More

Tip : If you want this issue to be picked, you need to add a challenge-program label to it.

Warning: None

[hftsin]

@qw4990 can you add a challenge-program label to this issue?

[ti-challenge-bot]

This issue does not belong to any SIG.

More

Tip : Currently, we only support sig labels starting with sig/, maybe you should add this type of label.

Warning: None

[huang-b]

/pick-up

[ti-challenge-bot]

You do not have a team yet, and the current challenge program only supports team participation.

[huang-b]

/pickup