Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Query Partitioning #1094

Open
Shelnutt2 opened this issue Dec 13, 2018 · 1 comment
Open

Implement Query Partitioning #1094

Shelnutt2 opened this issue Dec 13, 2018 · 1 comment

Comments

@Shelnutt2
Copy link
Member

Shelnutt2 commented Dec 13, 2018

For high level applications such as presto or spark having a query partitioner that can break a query into optimally sized sub arrays will be beneficial. Ideally the partition function can take one or more sub arrays as input and the number of desired partition and return a list of new subarrays to query based off of.

Implementing heuristics so for sparse arrays the partitions can be balanced is important. Currently Presto and spark implement their own naive partitioning which can result in unbalanced reads on a sparse array.

@Shelnutt2 Shelnutt2 added this to the 1.6.0 milestone Dec 13, 2018
@tdenniston
Copy link
Contributor

An experimental partitioner was added in #1197 . See also #1225 -- decision needs to be taken on an API for this or not.

@tdenniston tdenniston modified the milestones: 1.6.0, 1.7.0 Jun 13, 2019
@tdenniston tdenniston modified the milestones: 1.7.0, 1.8.0 Nov 12, 2019
@joe-maley joe-maley removed this from the 1.8.0 milestone Oct 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants