Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AMORO-1449]: Extract common spark module to reuse code between different spark versions #1858

Merged
merged 6 commits into from
Aug 23, 2023

Conversation

baiyangtx
Copy link
Contributor

Why are the changes needed?

fix #1449

Brief change log

  • Add a common spark module.
  • Extract common test framework to common spark module
  • Extract properties to reuse between spark versions.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

@github-actions github-actions bot added module:mixed-spark Spark module for Mixed Format module:core Core module type:build labels Aug 16, 2023
@codecov
Copy link

codecov bot commented Aug 16, 2023

Codecov Report

Patch coverage has no change and project coverage change: +0.01% 🎉

Comparison is base (76da3c4) 50.90% compared to head (1cdfb78) 50.91%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1858      +/-   ##
============================================
+ Coverage     50.90%   50.91%   +0.01%     
  Complexity     3804     3804              
============================================
  Files           477      479       +2     
  Lines         25631    25635       +4     
  Branches       2608     2608              
============================================
+ Hits          13047    13052       +5     
+ Misses        11417    11416       -1     
  Partials       1167     1167              
Flag Coverage Δ
core 51.37% <ø> (+0.01%) ⬆️
trino 48.67% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...com/netease/arctic/spark/ArcticTableStoreType.java 0.00% <ø> (ø)
...a/com/netease/arctic/spark/SparkSQLProperties.java 0.00% <ø> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions github-actions bot added module:mixed-hive Hive moduel for Mixed Format module:mixed-trino trino module for Mixed Format labels Aug 22, 2023
Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@zhoujinsong zhoujinsong merged commit 2840f3b into apache:master Aug 23, 2023
@baiyangtx baiyangtx deleted the spark-common-module branch August 23, 2023 05:43
ShawHee pushed a commit to ShawHee/arctic that referenced this pull request Dec 29, 2023
…rent spark versions (apache#1858)

* extract common spark module

* disable spark unified catalog tests

* common defined

* fix review comments

* fix unit test failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:core Core module module:mixed-hive Hive moduel for Mixed Format module:mixed-spark Spark module for Mixed Format module:mixed-trino trino module for Mixed Format type:build
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement]: Extract Spark Common Module to reuse code between different spark versions.
2 participants