Release Datafusion 6.0.0 #890

houqp · 2021-08-15T19:10:36Z

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We had some oversights in the 5.0.0 release (#771) causing us not able to release the python binding and datafusion-cli.

Describe the solution you'd like

Release Datafusion 5.1.0 with an improved process to support python binding and cli releasse.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
see #887, #883 and #837

mmuru · 2021-09-03T03:43:42Z

@houqp: Do we have an ETA on the python binding release? Thanks.

houqp · 2021-09-03T05:03:47Z

@mmuru probably 2-3 weeks, need to get some legal issues resolved in #920 before we can cut a release tarball and start the voting process.

mmuru · 2021-09-03T18:12:40Z

@houqp: Thanks for the update. I am trying to build python wheel locally but I noticed in Cargo.toml file the dependency listed
datafusion = { git = "https://github.com/apache/arrow-datafusion.git", rev = "4d61196dee8526998aee7e7bb10ea88422e5f9e1" }
It did not get updated. Will maturin develop pick up the latest datafusion code? Please, can you clarify it? Thanks again.

houqp · 2021-09-04T05:08:56Z

@mmuru it will not, I just sent #967 to handle the datafusion update.

mmuru · 2021-09-05T04:16:25Z

@houqp: Thanks for the clarification and quick turnaround fix. I verified your changes and found two issues.

the requirement.txt is locked to Python 3.8 version. I think, this file should not be checked in the source since the python development version could be different. In my case, it was python 3.7 version.
Fixed the issue Creating dataframe with Recordbatch using pyarrow.Table.to_batches gives "type16 not valid error" when schema includes date32[day] type #949. Added support for pyarrow datatypes such as date32, date64 and Timestamp.
I would like to submit my changes as part of update datafusion to 5.1.0 for python binding #967 PR. Please, let me know.

houqp · 2021-09-05T22:22:41Z

the requirement.txt is locked to Python 3.8 version.

The requirements.txt is supposed to work for all python versions supported by the python binding. if it's broken for python 3.7, could you file a separate issue with the error message? We can continue the troubleshooting there.

I would like to submit my changes as part of update datafusion to 5.1.0 for python binding #967 PR. Please, let me know.

That's great, I recommend you send a separate PR based off the branch in #967 for this, or collaborate on #969 to get that issue addressed.

mmuru · 2021-09-06T23:14:54Z

@houqp: Sure, created #975. Ping me if you need more information.

houqp · 2021-09-30T04:29:43Z

@andygrove @alamb @Dandandan @jorgecarleitao @nevi-me given that we have had many major breaking changes merged in since the 5.x release, I am thinking maybe it's better to skip 5.1 and go 6.0 after #1010 gets merged. What do you think?

alamb · 2021-09-30T10:41:31Z

merged in since the 5.x release

I think using 6.0 is a good idea. I also don't think we need to wait for #1010 to be merged for a release, if we need to get the python binding / cli out sooner

houqp · 2021-10-02T08:00:32Z

sounds good, I think think I can try help push #873 to the finish line after you have arrow 6 released.

alamb · 2021-10-02T10:22:54Z

sounds good, I think think I can try help push #873 to the finish line after you have arrow 6 released.

It sounds like we are aiming to release arrow 6.0 in 2 weeks or so

tupshin · 2021-10-19T23:58:06Z

arrow 6 has been released. Any ETA on this one? I'm really looking forward to an up to date python API, in particular

houqp · 2021-10-20T02:21:24Z

@tupshin I pinged #873 again, once that's merged, we could kick off the release process, which usually takes 3-5 days.

jimexist · 2021-10-28T05:03:13Z

related Homebrew/homebrew-core#88184

tupshin · 2021-11-05T20:27:06Z

Not to nag, but I see 873 is merged. How we doing?

houqp · 2021-11-06T01:44:37Z

I am working on the changelog and the release PR, should be out this weekend.

alamb · 2021-11-06T09:55:21Z

FYI see #1253

houqp · 2021-11-14T02:03:47Z

rc0 tag pushed, working on automation to package and sign python wheels now. once that's done, i will send out the request for vote email.

houqp · 2021-11-17T06:41:57Z

Vote passed and I have pushed the release tags into Github. The release steps requires PMC member access. @alamb @andygrove @jorgecarleitao @kszucs could one of you follow the steps in https://github.com/apache/arrow-datafusion/tree/master/dev/release#finalize-the-release to complete the release?

The remaining steps are:

run ./dev/release/release-tarball.sh 6.0.0 0
publish to crates.io
publish to pypi

houqp · 2021-11-17T06:43:48Z

@jimexist we should be able to update datafusion-cli in homebrew as well.

jimexist · 2021-11-17T06:49:27Z

Homebrew/homebrew-core#89562

alamb · 2021-11-17T11:27:43Z

Vote passed and I have pushed the release tags into Github. The release steps requires PMC member access. @alamb @andygrove @jorgecarleitao @kszucs could one of you follow the steps in https://github.com/apache/arrow-datafusion/tree/master/dev/release#finalize-the-release to complete the release?

@houqp I will do so now. Thank you for all the work in this regard

houqp · 2021-11-17T19:27:40Z

Thanks @alamb ! I just noticed I forgot to add (cd datafusion-cli && cargo publish) in the release doc, could you help run that command as well? I will send a PR to get the doc updated later today.

@andygrove @jorgecarleitao @kou @kszucs @xhochy we will need your help to publish the python binding to PyPI since only you are listed as maintainers of the PyPI package. The steps are documented at https://github.com/apache/arrow-datafusion/tree/master/dev/release#publish-on-pypi

xhochy · 2021-11-17T20:26:35Z

I'd rather give more people access to PyPI ;)

kou · 2021-11-17T20:27:16Z

I'm trying.

I found a typo in the document:

diff --git a/dev/release/README.md b/dev/release/README.md
index 2127dc23..73b3eb1a 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -304,7 +304,7 @@ PyPI, in order to conform to Apache Software Foundation governance standards.
 First, download all official python release artifacts:
 
 ```shell
-svn co https://dist.apache.org/repos/dist/release/arrow/apache-arrow-datafusion-5.1.0-rc0/python ./python-artifacts
+svn co https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-5.1.0/python ./python-artifacts
 ```
 
 Use [twine](https://pypi.org/project/twine/) to perform the upload.

https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-6.0.0/python/ uses 0.4.0 not 6.0.0. Is it OK?

houqp · 2021-11-17T21:07:39Z

I'd rather give more people access to PyPI ;)

+1 :D

Good catch @kou , I will include that fix in my docs PR. The version diff is expected because we want them to be decoupled so we can release major version change in the python binding without forcing a major version bump in datafusion.

kou · 2021-11-17T21:19:45Z

OK. I've published them: https://pypi.org/project/datafusion/0.4.0/

I found one more typo:

diff --git a/dev/release/README.md b/dev/release/README.md
index 2127dc23..fcf090e3 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -310,7 +310,7 @@ svn co https://dist.apache.org/repos/dist/release/arrow/apache-arrow-datafusion-
 Use [twine](https://pypi.org/project/twine/) to perform the upload.
 
 ```shell
-twine upload ./python-artifactl/*.{tar.gz,whl}
+twine upload ./python-artifacts/*.{tar.gz,whl}
 ```
 
 ### Call the vote

houqp · 2021-11-17T21:27:30Z

Thank you @kou ! I will include that fix in my docs PR as well :)

alamb · 2021-11-18T11:59:57Z

Thanks @alamb ! I just noticed I forgot to add (cd datafusion-cli && cargo publish) in the release doc, could you help run that command as well? I will send a PR to get the doc updated later today.

Hi @houqp

I tried to publish datafusion-cli and I got the following error. It looks like datafusion-cli relies on ballista somehow

(arrow_dev) alamb@MacBook-Pro:~/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli$ cargo publish
    Updating crates.io index
warning: manifest has no description.
See https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata for more info.
   Packaging datafusion-cli v5.1.0-SNAPSHOT (/Users/alamb/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli)
error: failed to prepare local package for uploading

Caused by:
  failed to select a version for the requirement `ballista = "^0.6.0"`
  candidate versions found which didn't match: 0.5.0, 0.3.0, 0.2.5, ...
  location searched: crates.io index
  required by package `datafusion-cli v5.1.0-SNAPSHOT (/Users/alamb/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli)`

houqp · 2021-11-18T19:41:18Z

oh yeah, it supports ballista as a way to perform remote query execution. @alamb looks like we haven't published the ballista crates yet? could you do that first by following https://github.com/apache/arrow-datafusion/tree/master/dev/release#publish-on-cratesio?

alamb · 2021-11-18T21:15:16Z

could you do that first by following https://github.com/apache/arrow-datafusion/tree/master/dev/release#publish-on-cratesio?

Done (updated instructions in #1331)

Turns out I still can't upload datafusion-cli package:

(arrow_dev) alamb@MacBook-Pro:~/Downloads/apache-arrow-datafusion-6.0.0$ (cd datafusion-cli && cargo publish)
....
    Finished dev [unoptimized + debuginfo] target(s) in 1m 20s
   Uploading datafusion-cli v5.1.0-SNAPSHOT (/Users/alamb/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli)
error: failed to publish to registry at https://crates.io

Caused by:
  the remote server responded with an error: invalid upload request: invalid length 7, expected at most 5 keywords per crate at line 1 column 3441

~~I will file a ticket~~ Tracked by #1332

houqp added the enhancement New feature or request label Aug 15, 2021

houqp self-assigned this Aug 15, 2021

houqp added this to the 5.1.0 milestone Aug 20, 2021

houqp changed the title ~~Release Datafusion 5.1.0~~ Release Datafusion 6.0.0 Oct 9, 2021

houqp closed this as completed Nov 17, 2021

alamb mentioned this issue Nov 18, 2021

Update release instructions #1331

Merged

houqp mentioned this issue Nov 19, 2021

fix and update release instruction #1333

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release Datafusion 6.0.0 #890

Release Datafusion 6.0.0 #890

houqp commented Aug 15, 2021 •

edited

Loading

mmuru commented Sep 3, 2021

houqp commented Sep 3, 2021

mmuru commented Sep 3, 2021

houqp commented Sep 4, 2021

mmuru commented Sep 5, 2021

houqp commented Sep 5, 2021

mmuru commented Sep 6, 2021

houqp commented Sep 30, 2021

alamb commented Sep 30, 2021 •

edited

Loading

houqp commented Oct 2, 2021

alamb commented Oct 2, 2021

tupshin commented Oct 19, 2021 •

edited

Loading

houqp commented Oct 20, 2021

jimexist commented Oct 28, 2021

tupshin commented Nov 5, 2021

houqp commented Nov 6, 2021

alamb commented Nov 6, 2021

houqp commented Nov 14, 2021

houqp commented Nov 17, 2021 •

edited

Loading

houqp commented Nov 17, 2021

jimexist commented Nov 17, 2021

alamb commented Nov 17, 2021

houqp commented Nov 17, 2021

xhochy commented Nov 17, 2021

kou commented Nov 17, 2021 •

edited

Loading

houqp commented Nov 17, 2021 •

edited

Loading

kou commented Nov 17, 2021

houqp commented Nov 17, 2021

alamb commented Nov 18, 2021

houqp commented Nov 18, 2021

alamb commented Nov 18, 2021 •

edited

Loading

Release Datafusion 6.0.0 #890

Release Datafusion 6.0.0 #890

Comments

houqp commented Aug 15, 2021 • edited Loading

mmuru commented Sep 3, 2021

houqp commented Sep 3, 2021

mmuru commented Sep 3, 2021

houqp commented Sep 4, 2021

mmuru commented Sep 5, 2021

houqp commented Sep 5, 2021

mmuru commented Sep 6, 2021

houqp commented Sep 30, 2021

alamb commented Sep 30, 2021 • edited Loading

houqp commented Oct 2, 2021

alamb commented Oct 2, 2021

tupshin commented Oct 19, 2021 • edited Loading

houqp commented Oct 20, 2021

jimexist commented Oct 28, 2021

tupshin commented Nov 5, 2021

houqp commented Nov 6, 2021

alamb commented Nov 6, 2021

houqp commented Nov 14, 2021

houqp commented Nov 17, 2021 • edited Loading

houqp commented Nov 17, 2021

jimexist commented Nov 17, 2021

alamb commented Nov 17, 2021

houqp commented Nov 17, 2021

xhochy commented Nov 17, 2021

kou commented Nov 17, 2021 • edited Loading

houqp commented Nov 17, 2021 • edited Loading

kou commented Nov 17, 2021

houqp commented Nov 17, 2021

alamb commented Nov 18, 2021

houqp commented Nov 18, 2021

alamb commented Nov 18, 2021 • edited Loading

houqp commented Aug 15, 2021 •

edited

Loading

alamb commented Sep 30, 2021 •

edited

Loading

tupshin commented Oct 19, 2021 •

edited

Loading

houqp commented Nov 17, 2021 •

edited

Loading

kou commented Nov 17, 2021 •

edited

Loading

houqp commented Nov 17, 2021 •

edited

Loading

alamb commented Nov 18, 2021 •

edited

Loading