[Feature][connector-v2] add tablestore source and sink #3309

liugddx · 2022-11-05T07:56:47Z

Purpose of this pull request

The tablestore source connector reuse JDBC connector.

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist

liugddx · 2022-11-06T14:22:43Z

env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  Jdbc {
    url = "jdbc:ots:https://p00spwbz7y1w.cn-shanghai.ots.aliyuncs.com/p00spwbz7y1w"
    driver = "com.alicloud.openservices.tablestore.jdbc.OTSDriver"
    user = "xxxxxxxx"
    password = "xxxxx"
    query = "select * from table_test"
  }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {


  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform
}

sink {
  Console {
    parallelism = 1
  }

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

liugddx · 2022-11-07T05:03:15Z

This is the test data.

liugddx · 2022-11-07T05:43:32Z

flink engine test result

liugddx · 2022-11-07T05:54:04Z

spark engine test result.

# Conflicts: # seatunnel-connectors-v2/pom.xml

liugddx · 2022-11-09T14:06:54Z

sink test result:

source data
flink result
spark result

liugddx · 2022-11-09T14:08:20Z

sink connector test conf

env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  Jdbc {
    url = "jdbc:ots:https://p00spwbz7y1w.cn-shanghai.ots.aliyuncs.com/p00spwbz7y1w"
    driver = "com.alicloud.openservices.tablestore.jdbc.OTSDriver"
    user = "xxx"
    password = "xxx"
    query = "select * from source"
  }


  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {

  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform
}

sink {
  Tablestore {
    end_point = "https://p00spwbz7y1w.cn-shanghai.ots.aliyuncs.com"
    instance_name = "xxx"
    access_key_id = "xxx"
    access_key_secret = "xxx"
    table = "sink"
    primary_keys = ["pk_1","pk_2","pk_3"]
  }

# Conflicts: # plugin-mapping.properties

liugddx · 2022-11-09T14:14:55Z

@hailin0 @Hisoka-X @EricJoy2048 PTAL,thanks

ic4y · 2022-11-11T02:31:43Z

...tor-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSource.java

@@ -139,7 +130,7 @@ private SeaTunnelRowType initTableField(Connection conn) {
        ArrayList<String> fieldNames = new ArrayList<>();
        try {
            PreparedStatement ps = conn.prepareStatement(jdbcSourceOptions.getJdbcConnectionOptions().getQuery());
-            ResultSetMetaData resultSetMetaData = ps.getMetaData();
+            ResultSetMetaData resultSetMetaData = ps.executeQuery().getMetaData();


why need add executeQuery

executeQuery has a great impact on the performance of other databases, it is recommended to use JdbcDialect to achieve

ic4y · 2022-11-11T02:49:00Z

...ain/java/org/apache/seatunnel/connectors/seatunnel/tablestore/sink/TablestoreSinkClient.java

+        initialize = true;
+    }
+
+    public synchronized void write(RowPutChange rowPutChange) throws IOException {


no synchronized needed

ic4y · 2022-11-11T02:49:29Z

...ain/java/org/apache/seatunnel/connectors/seatunnel/tablestore/sink/TablestoreSinkClient.java

+        }
+    }
+
+    public synchronized void close() throws IOException {


ic4y · 2022-11-11T02:56:38Z

seatunnel-dist/pom.xml

+                    <groupId>org.apache.seatunnel</groupId>
+                    <artifactId>connector-tablestore</artifactId>
+                    <version>${project.version}</version>
+                    <scope>provided</scope>                </dependency>


keep the format

ic4y · 2022-11-11T02:58:56Z

...main/java/org/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java

+        if (config.hasPath(TablestoreConfig.DEFAULT_BATCH_INTERVAL_MS)) {
+            this.batchIntervalMs = config.getInt(TablestoreConfig.DEFAULT_BATCH_INTERVAL_MS);
+        }
+        if (config.hasPath(TablestoreConfig.PRIMARY_KEYS)) {


already checked in checkAllExists

EricJoy2048

Please add OptionRule reference #3337.

EricJoy2048 · 2022-11-11T03:22:44Z

docs/en/connector-v2/sink/Jdbc.md

Please add changed log

EricJoy2048 · 2022-11-11T03:23:19Z

docs/en/connector-v2/sink/Tablestore.md

+
+The primaryKeys of Tablestore.
+
+### common options


Suggested change

### common options

### common options [ config ]

liugddx · 2022-11-11T14:40:48Z

@EricJoy2048 @ic4y done. PTAL again,thanks.

EricJoy2048 · 2022-11-12T03:07:41Z

...in/java/org/apache/seatunnel/connectors/seatunnel/tablestore/sink/TablestoreSinkFactory.java

+
+    @Override
+    public OptionRule optionRule() {
+        return OptionRule.builder().required(END_POINT, TABLE, INSTANCE_NAME, ACCESS_KEY_ID, ACCESS_KEY_SECRET, PRIMARY_KEYS, SeaTunnelSchema.SCHEMA).build();


Lost the option options?

Lost the option options?

What is the meaning of it?

Done . @EricJoy2048 @ic4y

.../main/java/org/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java

...main/java/org/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java Co-authored-by: Eric <[email protected]>

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java Co-authored-by: Eric <[email protected]>

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java Co-authored-by: Eric <[email protected]>

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java Co-authored-by: Eric <[email protected]>

ic4y

Nice

liugddx marked this pull request as draft November 5, 2022 07:56

liugddx closed this Nov 5, 2022

liugddx force-pushed the feature-tablestore branch from c0cf8b5 to da7490e Compare November 5, 2022 13:07

add tablestore

e94a14d

liugddx reopened this Nov 6, 2022

liugddx added 3 commits November 6, 2022 21:09

fix unable to release link problem

d036fc6

remove schema

77c9ad4

add doc

77dd396

liugddx added 12 commits November 7, 2022 18:28

fix convert error

cbf1a6d

fix deadlink error

29c6897

fix code style error

6e608a0

fix code style error

53b5abb

revert table sink

8359747

fix deadlink

0785f3b

add tablestore sink

0de05f5

Merge remote-tracking branch 'upstream/dev' into feature-tablestore

7a0593f

# Conflicts: # seatunnel-connectors-v2/pom.xml

fix type error

72f75d4

add tablestore sink

51a56a7

fix tablestore sink bug

b736f23

fix tablestore sink bug

448a619

Merge remote-tracking branch 'upstream/dev' into feature-tablestore

a86225b

# Conflicts: # plugin-mapping.properties

liugddx marked this pull request as ready for review November 9, 2022 14:14

Hisoka-X requested a review from ic4y November 10, 2022 02:50

ic4y reviewed Nov 11, 2022

View reviewed changes

EricJoy2048 reviewed Nov 11, 2022

View reviewed changes

liugddx added 2 commits November 11, 2022 22:20

Merge remote-tracking branch 'upstream/dev' into feature-tablestore

f594236

fix cr error

40d2018

liugddx added 2 commits November 11, 2022 22:48

fix cr error

5896145

fix cr error

d5a839a

EricJoy2048 reviewed Nov 12, 2022

View reviewed changes

liugddx added 3 commits November 12, 2022 21:36

add optional params

7e84bd7

Merge remote-tracking branch 'upstream/dev' into feature-tablestore

adfc22e

fix cr error

5df8159

EricJoy2048 reviewed Nov 14, 2022

View reviewed changes

liugddx and others added 9 commits November 14, 2022 16:34

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

de19951

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

1be80a9

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

3e5c3eb

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

f2e4c6b

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

c2883a5

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreConfig.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

557fed0

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

6ff5e15

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java Co-authored-by: Eric <[email protected]>

Update seatunnel-connectors-v2/connector-tablestore/src/main/java/org…

d76fa8a

…/apache/seatunnel/connectors/seatunnel/tablestore/config/TablestoreOptions.java Co-authored-by: Eric <[email protected]>

fix cr error

ec0786c

liugddx requested a review from EricJoy2048 November 14, 2022 10:12

ic4y approved these changes Nov 15, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 15, 2022

EricJoy2048 approved these changes Nov 15, 2022

View reviewed changes

EricJoy2048 merged commit ebebf0b into apache:dev Nov 15, 2022

EricJoy2048 mentioned this pull request Nov 15, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

liugddx deleted the feature-tablestore branch November 18, 2022 09:04

zhilinli123 mentioned this pull request Apr 11, 2023

[Feature][Connector-V2][Docs] Unified format for connector-v2 docs #4544

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][connector-v2] add tablestore source and sink #3309

[Feature][connector-v2] add tablestore source and sink #3309

liugddx commented Nov 5, 2022 •

edited

Loading

liugddx commented Nov 6, 2022 •

edited

Loading

liugddx commented Nov 7, 2022 •

edited

Loading

liugddx commented Nov 7, 2022

liugddx commented Nov 7, 2022

liugddx commented Nov 9, 2022 •

edited

Loading

liugddx commented Nov 9, 2022 •

edited

Loading

liugddx commented Nov 9, 2022

ic4y Nov 11, 2022

ic4y Nov 14, 2022

ic4y Nov 11, 2022

ic4y Nov 11, 2022

ic4y Nov 11, 2022

ic4y Nov 11, 2022

EricJoy2048 left a comment

EricJoy2048 Nov 11, 2022

EricJoy2048 Nov 11, 2022

liugddx commented Nov 11, 2022

EricJoy2048 Nov 12, 2022

liugddx Nov 12, 2022

liugddx Nov 14, 2022

ic4y left a comment

[Feature][connector-v2] add tablestore source and sink #3309

[Feature][connector-v2] add tablestore source and sink #3309

Conversation

liugddx commented Nov 5, 2022 • edited Loading

Purpose of this pull request

Check list

liugddx commented Nov 6, 2022 • edited Loading

liugddx commented Nov 7, 2022 • edited Loading

liugddx commented Nov 7, 2022

liugddx commented Nov 7, 2022

liugddx commented Nov 9, 2022 • edited Loading

liugddx commented Nov 9, 2022 • edited Loading

liugddx commented Nov 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricJoy2048 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liugddx commented Nov 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ic4y left a comment

Choose a reason for hiding this comment

liugddx commented Nov 5, 2022 •

edited

Loading

liugddx commented Nov 6, 2022 •

edited

Loading

liugddx commented Nov 7, 2022 •

edited

Loading

liugddx commented Nov 9, 2022 •

edited

Loading

liugddx commented Nov 9, 2022 •

edited

Loading