Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatible with spark3 and spark2 #41

Merged
merged 13 commits into from
Jan 4, 2022
Merged

Conversation

Nicole00
Copy link
Contributor

compatible with spark2 and scala 2.11
compatible with spark3 and scala 2.12

@Nicole00 Nicole00 added wip Solution: work in progress do not review PR: not ready for the code review yet doc affected PR: improvements or additions to documentation labels Dec 14, 2021
@Nicole00 Nicole00 force-pushed the spark3.0 branch 2 times, most recently from ec62aa2 to 858d134 Compare December 23, 2021 07:30
@codecov-commenter
Copy link

codecov-commenter commented Dec 30, 2021

Codecov Report

Merging #41 (80184bf) into master (657b83b) will increase coverage by 13.61%.
The diff coverage is 11.11%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master      #41       +/-   ##
=============================================
+ Coverage     36.57%   50.19%   +13.61%     
- Complexity       48       74       +26     
=============================================
  Files            24       16        -8     
  Lines          2127     1291      -836     
  Branches        386      246      -140     
=============================================
- Hits            778      648      -130     
+ Misses         1200      525      -675     
+ Partials        149      118       -31     
Impacted Files Coverage Δ
...com/vesoft/exchange/common/CheckPointHandler.scala 0.00% <ø> (ø)
...cala/com/vesoft/exchange/common/ErrorHandler.scala 0.00% <0.00%> (ø)
...ala/com/vesoft/exchange/common/GraphProvider.scala 67.85% <ø> (ø)
...cala/com/vesoft/exchange/common/MetaProvider.scala 68.75% <ø> (ø)
...ain/scala/com/vesoft/exchange/common/Package.scala 8.33% <0.00%> (ø)
.../vesoft/exchange/common/config/SchemaConfigs.scala 70.96% <ø> (ø)
...om/vesoft/exchange/common/config/SinkConfigs.scala 76.92% <ø> (ø)
...m/vesoft/exchange/common/processor/Processor.scala 67.20% <ø> (ø)
...ft/exchange/common/processor/ReloadProcessor.scala 0.00% <0.00%> (ø)
...a/com/vesoft/exchange/common/utils/HDFSUtils.scala 0.00% <ø> (ø)
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 657b83b...80184bf. Read the comment docs.

@Nicole00 Nicole00 removed do not review PR: not ready for the code review yet wip Solution: work in progress labels Jan 4, 2022
@Nicole00
Copy link
Contributor Author

Nicole00 commented Jan 4, 2022

Exchange support Spark2.2 with Scala 2.11, Spark 2.4 with Scala 2.11 and Scala 2.12, Spark 3.0 with Scala 2.12.

We have tested these three versions of Exchange, there are the support situation of the data sources:

datasource/spark version nebula-exchange_spark_2.2 nebula-exchange_spark_2.4 nebula-exchange_spark_3.0
csv Y Y Y
json Y Y Y
parquet Y Y Y
orc Y Y Y
mysql Y Y Y
clickhouse Y Y Y
hbase Y Y Y
hive Y Y Y
neo4j Y N
maxcompute N Y N
pulsar Y  Not tested
kafka Y  Not tested

@@ -0,0 +1,383 @@
/* Copyright (c) 2020 vesoft inc. All rights reserved.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2021?

Copy link
Contributor Author

@Nicole00 Nicole00 Jan 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file exists since 2020

@@ -0,0 +1,257 @@
/* Copyright (c) 2020 vesoft inc. All rights reserved.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc affected PR: improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants