DataFusion weekly project plan (Andrew Lamb) - Jan 29, 2024

[alamb]

Follow on to Jan 22, 2024

Boilerplate Introduction

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)

My (personal) goals for this project

1000 projects are built using DataFusion

My (personal) plans for this week

• (LOOKING FOR A REVIEW HERE) Get first step of breaking out functions ://github.com/Create datafusion-functions crate, extract encode and decode to  #8705 reviewed, address comments, file follow on tickets
• Continue to improve ClickBench performance Faster String Aggregation (this is my personal fun project)

Project Queue (list of future projects)

• Improved Statistics: Epic: Statistics improvements #8227 (specifically, Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)

Projects I plan to help actively help review / plan

• More advanced pruning predicates Don't error on unknown column when pruning if predicate can still be proven false #7869 with @appletreeisyellow
• Improve planning speed [Epic] A collection of issues to improve planning performance / speed / efficiency #5637 with @matthewmturner
• Hardening Recursive CTEs Add support for recursive CTEs #462 with @matthewgapp and @jonahgao

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I use to prioritize my own time

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (my employer)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality (note this is the lowest)

The top strategically important projects in my head are:

• Anything that makes it easier to use DataFusion as a user (docs, examples, better APIs, etc), including Getting started guide for new users (who want to use DataFusion in their project)
• Anything that improves DataFusion's quality such as improved / expanded tests, etc
• Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Improving performance of aggregation - especially high cardinality groupings, where we still lag state of the art by 2x

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation or the projects named above

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

[alamb]

Review Queue:

DataFusion

• [task #8201] Port tests in expr.rs to sqllogictest, finish the left c… #9014
• Properly encode STRING_AGG, NTH_VALUE in physical plan protobufs #9027 (review)

[alamb]

Review Queue:

DataFusion

• Change remove from swap to shift in index map #9049
• fix issue where upper and lower functions only work correctly on ascii character  #9054
• Return proper number of expressions for nth_value_agg #9044
• Split physical_plan_tpch into separate benchmarks #9043
• ScalarUDF with zero arguments should be provided with one null array as parameter #9031
• Remove single_file_output option from FileSinkConfig and Copy statement #9041
• Speedup DFSchema::merge using HashSet indices #9020
• Feat/make dfschema wrap schemaref #8905
• Remove some recursive cloning from logical planning #9050
• Add a make_date function #9040
• move InList related simplify to one place #9037
• Minor: small updates to bench.sh #9035
• Chore: explicitly list out all Expr types in TypeCoercionRewriter::mutate #9038
• Fix instr Function to Correctly Handle Unicode Characters in Arrow DataFusion #9051
• feat: support the ergonomics of getting list slice with stride  #8946

[alamb]

DataFusion

• Add documentation for streaming usecase #9070
• Handle invalid types for negation #9066
• Add extension context to SessionState. #9063
• feat: support array_reverse #9023

[alamb]

DataFusion

• chore(placeholder): update error message and add tests #9073 (review)
• Add benchmarks for to_timestamp and make_date functions #9086
• feat: track memory consumers for GreedyMemoryPoolState #9015
• Unnest with single expression #9069
• Support FixedSizeList type coercion #8902
• Support join filter for SortMergeJoin #9080

[alamb]

DataFusion

• Fix update_expr for projection pushdown #9096
• Cache output equivalence_properties in ProjectionExec #9097
• feat: issue #8969 adding position function #8988 (review)
• Remove External Table Backwards Compatibility Options #9105 (review)
• WIP [Performance] Optimize DFSchema search by field #9104
• feat: improve make_date performance #9112
• Cleanup regex_expressions.rs to remove _regexp_match function #9106
• Add to_date function #9019
• Add http(s) support to the command line #8753
• Consider table scan filter during analysis of optimize projections #9131
• Add regexp_like, improve docs and examples for regexp_match` #9137
• Add ForeignKey constraint type #8566
• Change ScalarValue::Struct to ArrayRef #7893
• Transform with payload #8664
• Consolidate TreeNode transform and rewrite APIs #8891
• Fix incorrect results for NOT IN subqueries with nulls #8271

Watching

• Support Copy with Remote Object Stores in datafusion-cli #9064
• Support compute return types from argument values (not just their DataTypes) #8985
• Feat/make dfschema wrap schemaref #8905

[alamb]

Next week in #9121