Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Establish Spark 3.x & Scala 2.13 cross publish #573

Merged
merged 16 commits into from
Oct 26, 2021

Conversation

pomadchin
Copy link
Member

@pomadchin pomadchin commented Oct 18, 2021

TODO:

  • check that 3.2.0 version would still properly work with Spark 3.1.x, is it even possible or we would delay this PR merge due to compatability reasons. incompatible , see Establish Spark 3.x & Scala 2.13 cross publish #573 (comment) (it was expected but still worth checking)
  • Cross publish projects for different Spark versions
  • Drop irrec-regex-gen test dependency
  • Scala 2.13 cross build
  • Green CI
  • Drop sbt-github-actions (we have a bit more complicated release process now)
  • Scala 2.13 / 2.12 cross publish
  • Updated docs

This PR slightly changes the release process:

  • By default, frameless artifacts depend on the most recent Spark version
  • Suffix -spark-{major}{minor} is added to artifacts that are released for the previous Spark version(s).

Artifact names examples:

  • frameless-dataset (the latest Spark dependency)
  • frameless-dataset-spark31 (Spark 3.1.x dependency)
  • frameless-dataset-spark30 (Spark 3.0.x dependency)

Closes #554
Closes #570
Closes #553

@codecov
Copy link

codecov bot commented Oct 18, 2021

Codecov Report

Merging #573 (a14d99a) into master (8b18af2) will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #573      +/-   ##
==========================================
+ Coverage   95.13%   95.14%   +0.01%     
==========================================
  Files          65       65              
  Lines        1131     1134       +3     
  Branches       11        5       -6     
==========================================
+ Hits         1076     1079       +3     
  Misses         55       55              
Flag Coverage Δ
2.12.15 95.14% <100.00%> (+0.01%) ⬆️
2.13.6 96.39% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...scala/frameless/functions/AggregateFunctions.scala 100.00% <ø> (ø)
...taset/src/main/scala/frameless/functions/Lit.scala 94.44% <ø> (ø)
...t/src/main/scala/frameless/functions/package.scala 100.00% <ø> (ø)
...et/src/main/scala/frameless/ops/SmartProject.scala 100.00% <ø> (ø)
cats/src/main/scala/frameless/cats/implicits.scala 73.91% <100.00%> (ø)
dataset/src/main/scala/frameless/TypedColumn.scala 100.00% <100.00%> (ø)
...ataset/src/main/scala/frameless/TypedDataset.scala 100.00% <100.00%> (ø)
...c/main/scala/frameless/TypedDatasetForwarded.scala 74.28% <100.00%> (ø)
...la/frameless/functions/NonAggregateFunctions.scala 100.00% <100.00%> (ø)
...taset/src/main/scala/frameless/functions/Udf.scala 87.23% <100.00%> (+0.27%) ⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b18af2...a14d99a. Read the comment docs.

@pomadchin pomadchin force-pushed the upd/spark-3.2.0 branch 5 times, most recently from 6a248b4 to de2d72a Compare October 18, 2021 13:45
@pomadchin
Copy link
Member Author

@cchantep what are your thoughts about merging this one? I would like to suggest to cut a release with 3.1.x in deps before merging this in.

@pomadchin pomadchin changed the title [WIP] Bump Spark version up to 3.2.0 Bump Spark version up to 3.2.0 Oct 18, 2021
Copy link
Collaborator

@cchantep cchantep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As-is it would break compatibility (e.g. functions.{shiftrightunsigned,bitwise_not}), so unless a cross-version build is set up, I won't merge that.

@pomadchin pomadchin force-pushed the upd/spark-3.2.0 branch 2 times, most recently from 56c3fc8 to bf27a12 Compare October 18, 2021 16:29
@pomadchin
Copy link
Member Author

pomadchin commented Oct 18, 2021

@cchantep bf27a12 makes the code source compatible with 3.1.x, is it enough for you or you want to establish 3.1 / 3.2 cross version releases as well?

.github/workflows/ci.yml Outdated Show resolved Hide resolved
@pomadchin pomadchin force-pushed the upd/spark-3.2.0 branch 2 times, most recently from 1911a94 to 844fe3e Compare October 19, 2021 00:05
@pomadchin pomadchin changed the title Bump Spark version up to 3.2.0 [WIP] Establish Spark 3.x crosspublish Oct 20, 2021
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
build.sbt Show resolved Hide resolved
@pomadchin pomadchin force-pushed the upd/spark-3.2.0 branch 2 times, most recently from 6faff65 to cafd1e3 Compare October 23, 2021 21:39
@pomadchin pomadchin force-pushed the upd/spark-3.2.0 branch 2 times, most recently from 1190634 to 9ed1527 Compare October 24, 2021 15:36
@pomadchin pomadchin changed the title [WIP] Establish Spark 3.x & Scala 2.13 cross publish Establish Spark 3.x & Scala 2.13 cross publish Oct 24, 2021
@pomadchin pomadchin force-pushed the upd/spark-3.2.0 branch 4 times, most recently from e288560 to 815a004 Compare October 24, 2021 15:58
README.md Show resolved Hide resolved
build.sbt Show resolved Hide resolved
build.sbt Outdated Show resolved Hide resolved
Co-authored-by: Cédric Chantepie <[email protected]>
@pomadchin
Copy link
Member Author

Merging! Let's see how the snapshot deployment goes (hopefully well!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Spark 3.2.0 upgrade Scala 2.13 release
2 participants