-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DataFusion weekly project plan (Andrew Lamb) - April 22, 2024 #10172
Comments
9 tasks
8 tasks
Next week: #10283 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Follow on to #10002 (I didn't make one for week of April 15 2024)
It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)
My Highlights from Last 2 Weeks
BuiltInScalarFunction
was removed): Move coalesce to datafusion-functions and remove BuiltInScalarFunction #10098BuiltInAggregateFunction
as well: @jayzhan211 is leading the charge [Epic] UnifyAggregateFunction
Interface (remove built in list ofAggregateFunction
s), improve the system #8708trim
,concat
,upper
etc got faster -- see [EPIC] A collection of improvements to optimize the speed of scalar functions #10171. thanks @JasonLi-cn 🙏Optimizer
, part of improving planning performance37.1.0
maintenance release: Release DataFusion37.1.0
(non breaking API release) #9904UNNEST
with multiple arrays thanks to @jonahgao feat: supportunnest
multiple arrays #100440.45.0
datafusion-sqlparser-rs#1162My (personal) goals
1000 projects are built using DataFusion 📈
My (personal) plans for this week
LogicalPlan
during OptimizerPasses #9637 --ParquetExec
more (so it is clear / easier to understand what is already supported and what else could be useful API in ParquetExec to pass in RowSelections toParquetExec
(enable custom indexes, finer grained pushdown) #9929)Project Queue (list of future projects)
ParquetExec
(enable custom indexes, finer grained pushdown) #9929 (likely not this week)Projects I plan to help actively help review / plan
StringViewArray
andBinaryViewArray
arrow-rs#5374Algorithm for (my) prioritizing PR reviews
Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I use to prioritize my own time
Priority:
The top strategically important projects in my head are:
BuiltInScalarFunction
) #8045Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation or the projects named above
If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)
Background
The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:
The text was updated successfully, but these errors were encountered: