Blog post with DataFusion Jun - Sep 2023

[alamb]

Is your feature request related to a problem or challenge?

We have had good luck writing up quarterly updates for DataFusion, most recently:
https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/

(see #5812)

Describe the solution you'd like

It would be great to write another about what has happened in the last few months of DataFusion

Things I expect will be good to highlight (🤞 ):

• Improved Struct/array support (@izveigor ❤️ )
• better group by performance with many distinct groups
• better insert performance

Others?

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

Ideas of Major items to include in this post

1. User defined window functions: Blog post about user defined window functions #6781
2. faster aggregatge performance -- Improve the performance of Aggregator, grouping, aggregation #4973
3. Support for ARRAY / Lists -- General ticket for Array/List data type #6863 etc (thanks @izveigor and @jayzhan211 )

[Dandandan]

Improved join performance would maybe be another thing to highlight. Maybe we show a benchmark with improvements (TCP-H, ClickBench, ...) from version 25 -> 28.

• Hash Join Vectorized collision checking #6724
• Improve performance/memory usage of HashJoin datastructure (5-15% improvement on selected TPC-H queries) #6679

[alamb]

There has been major work on INSERT and COPY as well, thanks to @devinjdangelo : #6569

[alamb]

Also #7400 spilling group by from @kazuyukitanimura

[alamb]

Another topic: the new library user guide: https://arrow.apache.org/datafusion/library-user-guide/index.html

[alamb]

FYI this is very much on my list, but I need to focus on the SIGMOD paper for a while. If someone else has the time and inclination to start a PR I would be most appreciative

[alamb]

Realistically I am very tied up with #6782 and so won't have time to work on a blog post until after that is submitted (end of Nov). If someone else has time to work on this it would be very much apprecaited

[alamb]

This is going to have to be more like a 2023 retrospective 🤔

[alamb]

I am starting to draft this now

[alamb]

Here is a PR with a draft (still needs more work): apache/arrow-site#457

[alamb]

The blog post is now published! https://arrow.apache.org/blog/2024/01/19/datafusion-34.0.0/

[alamb]

Let's capture other items to highlight here #9602