Release DataFusion 42.1.0

[alamb]

Is your feature request related to a problem or challenge?

There are a few bugs introduced as part of 42.0.0 that were hit by downstream consumers

For example, @rtyler and @ion-elgreco have hit #12687 I think

delta-io/delta-rs#2886 (comment)

Describe the solution you'd like

Consider releasing 42.1.0 patch release with some code fixes. Ideas:

• Field Metadata Lost on COUNT DISTINCT queries resulting in Internal Error: Physical input schema should be the same as the one converted from logical input schema #12687
• Field Metadata Lost on CROSS JOIN #12734
• Field Metadata Lost on UNION #12735
• physical-plan: Cast nested group values back to dictionary if necessary #12586

Describe alternatives you've considered

We could also just wait for the next release

• Release DataFusion 43.0.0 #12470

Additional context

No response

[alamb]

@andygrove i wonder what your thoughts are on this idea?

[andygrove]

Sounds good to me

[brancz]

Any chance we could include #12586 ?

[andygrove]

I can start looking at this today. Is there still interest in the idea of maintaining a LTS version of DataFusion? Would 42 be a good foundation for that?

[alamb]

Is there still interest in the idea of maintaining a LTS version of DataFusion? Would 42 be a good foundation for that?

I can help with this, but I am already stretched pretty thin with other things

In terms of creating an LTS branch, perhaps we can target 43 of 44 (where I am hoping we'll be able to complete

• [DISCUSSION] Make DataFusion the fastest engine for querying parquet data in ClickBench #12821
• [Epic] Unify WindowFunction Interface (remove built in list of BuiltInWindowFunction s) #8709
• [EPIC] Automatically generate all function documentation from code #12740

[alamb]

I have created the https://github.com/apache/datafusion/tree/maint-42.x branch

It would be super helpful if someone could make cherry-pick PR(s) with the changes mentioned above targeting that branch (I promise to review them quickly!)

[andygrove]

I have created the https://github.com/apache/datafusion/tree/maint-42.x branch

It would be super helpful if someone could make cherry-pick PR(s) with the changes mentioned above targeting that branch (I promise to review them quickly!)

We do already have the https://github.com/apache/datafusion/tree/branch-42 branch

[andygrove]

Here is some info on the current branching policy: https://github.com/apache/datafusion/tree/main/dev/release#branching-policy

[alamb]

We do already have the https://github.com/apache/datafusion/tree/branch-42 branch

Sorry - I have removed the maint-42 branch

Let's make the PRs to the branch-42 branch then!

[alamb]

Since no one made any backport PRs I think we are moving on to just releasing 43.0.0 #12470 -- if anyone is interested in helping do maintenance releases we can considering doing them, but otherwise I think we only have capacity to do releases off main

[matthewmturner]

I should have picked this up - apologies.

[Xuanwo]

Hello, I'm interested in helping with this as well. We need to cherry-pick four PRs, correct? Since @matthewmturner is also interested in helping, perhaps we could each prepare two PRs.

Cc @alamb, please let me know if I have misunderstood anything. I jumped from delta-io/delta-rs#2886 and am less familiar with the context of this issue.

[Xuanwo]

By the way, I noticed that #12470 is scheduled for October 20, 2024. Have I joined too late?

[matthewmturner]

imo there is still value in doing a 42.1.0 release - for example we have prepped for upgrade to datafusion 42 and i dont anticipate us wanting to do another round of getting ready for 43 quite yet.

i would be happy to split the work @Xuanwo

@alamb assuming you are still okay with cutting this release is the expectation we make prs to branch-42?

[alamb]

imo there is still value in doing a 42.1.0 release - for example we have prepped for upgrade to datafusion 42 and i dont anticipate us wanting to do another round of getting ready for 43 quite yet.

i would be happy to split the work @Xuanwo

@alamb assuming you are still okay with cutting this release is the expectation we make prs to branch-42?

Thank you @Xuanwo and @matthewmturner

I would be happy to support this release (aka do the process to create the candidate and run voting on the mailing list).

Creating PRs to branch-42 would be ideal -- perhaps just link them to this ticket and feel free to mention me

Once we have the changes on branch-42 I'll do the process to make the release candidate

[Xuanwo]

Great, I will take the first two. @matthewmturner let me know if you have some preferences.

Looking forward to working with you. This will be my first contribution to DF.

[matthewmturner]

@Xuanwo no preference, ill take the second two

[matthewmturner]

@Xuanwo i just did item 3 and i think it also covered item 2 from the list

[Xuanwo]

@Xuanwo i just did item 3 and i think it also covered item 2 from the list

That's great!

[Xuanwo]

Hello @matthewmturner, would you like me to take care of #12586, or are you already working on it?

[Xuanwo]

Hello @matthewmturner, would you like me to take care of #12586, or are you already working on it?

Oh, I have seen #12976. So quickly. Bravo!

[Xuanwo]

Thank you all for the excellent collaboration. I'm going to get some sleep now (it's 1 A.M. here). I will check the updates tomorrow and hope everything runs smoothly.

[alamb]

Thank you all for the excellent collaboration. I'm going to get some sleep now (it's 1 A.M. here). I will check the updates tomorrow and hope everything runs smoothly.

Pleasant dreams 💤

[alamb]

Ok, I think all the fixes have been applied, and I will now proceed to making a release candidate / voting thread for this release

Thanks again @matthewmturner and @Xuanwo for your help

[alamb]

Working on creating a release candidate

[alamb]

The RC is ready and the vote thread started: https://lists.apache.org/thread/xnmhm0gxc8tbdr3k4mft3vf8w61klhnp

[alamb]

The release was approved: https://lists.apache.org/thread/5rn0ghcj1zsfcqqwfws6yd929vjqvnwf 🎉

However, I had trouble uploading it to crates.io as it appears I am not an owner of datafusion-expr-common -- @andygrove can you either make me an owner or perhaps upload the crates? 🙏

 (cd datafusion/expr-common && cargo publish)
...

   Uploading datafusion-expr-common v42.1.0 (/Users/andrewlamb/Downloads/apache-datafusion-42.1.0/datafusion/expr-common)
error: failed to publish to registry at https://crates.io

Caused by:
  the remote server responded with an error (status 403 Forbidden): this crate exists but you don't seem to be an owner. If you believe this is a mistake, perhaps you need to accept an invitation to be an owner before publishing.

[alamb]

(it has been done -- thank you @andygrove )

[alamb]

The release is now available on crates.io: https://crates.io/crates/datafusion/42.1.0