-
Notifications
You must be signed in to change notification settings - Fork 992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sort-merge join #4538
Comments
Closed
8 tasks
Some initial work on this has been made in #4539. As I am not going to pursue it anytime soon, and change is rather complex. I will close this for now. If there is a demand, we can always reopen. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We are now using binary search algorithm when doing join/merge of two tables.
Despite binary search merge (call it
bmerge
) is a single threaded implementation now, as of now it is still 4 times faster than the next fastest solution, on a 5GB, 1e8 rows data (not taking GPU into account).I am proposing to have an alternative join algorithm, sort-merge join (call it
smerge
), that could be used if tables are already sorted by join keys. In sort-merge join, sorting join keys is the most expensive part.After looking at the timings in #4539 it make sense to use
smerge
instead ofbmerge
in many other cases, not just when tables are sorted. Although makingbmerge
parallel would reduce this timings differences, at least up to currently supported int32 row limit, #3957. I assume sort-merge join to scale little better. In the end it makes a linear scan over sorted integers, so the bigger the data (and chunk sizes), the better utilization of threads.The text was updated successfully, but these errors were encountered: