DataFusion weekly project plan (Andrew Lamb) - April 1, 2024

[alamb]

Follow on to #9796

It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)

My Highlights from Last Week

• 37.0.0 release is looking 🚀 Release DataFusion 37.0.0 #9682

• The community made great progress on unparsing expressions: Complete support for Expr --> String  #9726

• "extract scalar functions from the core" continues at good pace way [Epic] Port BuiltInFunctons to datafusion-functions-* crates #9285 (thanks @jayzhan211 and @erenavsarogullari and @Omega359 for keeping the 🚋 running here)

• Submitted SIGMOD final draft : Finalize SIGMOD 2024 paper ~(if accepted)~ #8373

• Ran our first in person meetup (it would be great if others wanted to organize meetups in other geographies): UPDATE: VENUE IS FULL Invitation: Mar 25, 2024 DataFusion meetup before the DataCouncil March 2024 in Austin TX #8522

• We have submitted the proposal for a new top level project for DataFusion to the ASF board: [DISCUSS] Move Apache Arrow Datafusion to a new top level Apache project #6475 (comment)

My (personal) goals

1000 projects are built using DataFusion 📈

My (personal) plans for this week

• Make at least two more PRs for subtasks [EPIC] Stop copying LogicalPlan during OptimizerPasses #9637
• Catchup on review backlog
• Writeup ideas on custom / user defined indexes (I mentioned this to @westonpace in person but haven't written it down yet)

Project Queue (list of future projects)

Projects I plan to help actively help review / plan

• [EPIC] Implement StringViewArray and BinaryViewArray arrow-rs#5374
• [Epic] Port BuiltInFunctons to datafusion-functions-* crates #9285
• [EPIC] Support converting Exprs and LogicalPlans --> SQL Strings #9494
• Supporting using parallel parquet writer outside of Datafusion query execution #9493 with @wiedld

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I use to prioritize my own time

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality (note this is the lowest)

The top strategically important projects in my head are:

• Anything that makes it easier to use DataFusion as a user (docs, examples, better APIs, etc), including Getting started guide for new users (who want to use DataFusion in their project)
• Anything that improves DataFusion's quality such as bug fixes, and improved / expanded tests, etc
• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation or the projects named above

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

Background

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

[matthewmturner]

Really interested to see your thoughts on custom indices, its something were currently working on to improve performance of listing table.

[alamb]

Review queue

Arrow

• IPC format support for StringViewArray and BinaryViewArray arrow-rs#5525
• feat: support reading and writingStringView and BinaryView in parquet (part 2) arrow-rs#5557 (review)
• Upgrade object_store dependency to use chrono 0.4.34 arrow-rs#5578
• Add try_filter_leaves to propagate error from filter closure arrow-rs#5575 (review)
• Update py03 from 0.20 to 0.21 arrow-rs#5566
• Optionally require alignment when reading IPC, respect alignment when writing arrow-rs#5554

DataFusion

• Make FirstValue an UDAF, Change AggregateUDFImpl::accumulator signature, support ORDER BY for UDAFs #9874
• refactor: make all udf function impls public #9903

[alamb]

Arrow:

• feat: support reading and writingStringView and BinaryView in parquet (part 2) arrow-rs#5557
• Support list_view_array & large_list_view layout and basic construction arrow-rs#5576

DataFusion

• perf: Use Arc<str> instead of Cow<&'a> in the analyzer #9824
• Simplify Expr::map_children #9876
• Consistent LogicalPlan subquery handling in TreeNode::apply and TreeNode::visit  #9913
• feature: Add a WindowUDFImpl::simplify() API #9906
• Validate partitions columns in CREATE EXTERNAL TABLE if table already exists. #9912
• feat: eliminate redundant sorts on monotonic expressions #9813
• Optimization: concat function #9732
• feat: Use file cache to list partitions if available #9655

[seddonm1]

Sorry for the location and possibly dumb question, but shouldn't all functions be exported here: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs#L138

I am probably using the library incorrectly but (for example) LENGTH will not be registered by the central context here: https://github.com/apache/arrow-datafusion/blob/2f550032140d42d1ee6d8ed86f7790766fa7302e/datafusion/core/src/execution/context/mod.rs#L1450

[alamb]

Hi @seddonm1 👋

Sorry for the location and possibly dumb question, but shouldn't all functions be exported here: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs#L138

Yes that is the intention

I am probably using the library incorrectly but (for example) LENGTH will not be registered by the central context here:

https://github.com/apache/arrow-datafusion/blob/2f550032140d42d1ee6d8ed86f7790766fa7302e/datafusion/core/src/execution/context/mod.rs#L1450

I think length is registered as an alias of character_length

[seddonm1]

Thanks @alamb . If you look at the first link I cannot see anywhere where the length alias is actually registered. My queries which previously worked started failing after this update. I haven't investigated if more functions have also not been registered.

[Omega359]

aliases for character_length are defined in https://github.com/apache/arrow-datafusion/blob/4bd7c137e0e205140e273a7c25824c94b457c660/datafusion/functions/src/unicode/character_length.rs#L45 and is used during udf registration @ https://github.com/apache/arrow-datafusion/blob/4bd7c137e0e205140e273a7c25824c94b457c660/datafusion/core/src/execution/context/mod.rs#L2137

[alamb]

Arrow:

• feat: support reading and writingStringView and BinaryView in parquet (part 2) arrow-rs#5557

DataFusion

• Move AggregateExpr, PhysicalExpr and PhysicalSortExpr to physical-expr-core #9926 (review)
• test: fix intermittent failure in cte.slt #9934
• Move cbrt, cos, cosh, degrees to datafusion-functions #9938
• Add Expr->String for Exists, Sort #9936
• Remove OwnedTableReference and OwnedSchemaReference #9933
• Prune out constant expressions from output ordering. #9947
• Optimize COUNT(1): Change the sentinel value's type for COUNT(*) to Int64 #9944
• chore(deps): update substrait requirement from 0.28.0 to 0.29.0 #9942
• feat: eliminate redundant sorts on monotonic expressions #9813
• Optimization: concat function #9732
• feat: Use file cache to list partitions if available #9655

[alamb]

DataFusion 37.0.0 is released: #9682 🌮

[seddonm1]

Thanks @Omega359 . I was thrown as the aliases are not registered here https://github.com/apache/arrow-datafusion/blob/3ae029988754c3fd3eb000abd4b76e643b9cbc7b/datafusion/execution/src/registry.rs#L174

[alamb]

BTW something cool for next week -- @milenkovicm WASM UDFs #9326 / #9326 (comment) / https://github.com/milenkovicm/wasaffi

[alamb]

Really interested to see your thoughts on custom indices, its something were currently working on to improve performance of listing table.

In case anyone else is following along, here are some items related to indexing:

• Indexing Support in DataFusion? #9963
• API in ParquetExec to pass in RowSelections to ParquetExec (enable custom indexes, finer grained pushdown) #9929

While I had this on the brain, @matthewmturner I also filed #9964 as I suspect others will be interested in helping make ListingTable faster too

[alamb]

Reviews:
@jayzhan211 is starting to pull aggregate functions out of the core: #9960

[alamb]

Next week #10002