Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Connector-V2][Cassandra] Add Cassandra Source And Sink Connector #3229

Merged
merged 31 commits into from
Nov 8, 2022
Merged

[Feature][Connector-V2][Cassandra] Add Cassandra Source And Sink Connector #3229

Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
c697e0a
add cassandra connector
Oct 29, 2022
89a8adf
add cassandra connector
Oct 29, 2022
1cb1906
add cassandra connector
Oct 29, 2022
1962d94
update
Oct 29, 2022
b305857
update
Oct 30, 2022
655e6dc
Merge branch 'apache:dev' into new-connector-cassandra
FWLamb Oct 30, 2022
fb4b852
update
Oct 30, 2022
9927a0e
Merge remote-tracking branch 'origin/new-connector-cassandra' into ne…
Oct 30, 2022
4b33676
update
Oct 30, 2022
865bf5d
Merge branch 'apache:dev' into new-connector-cassandra
FWLamb Oct 30, 2022
5d318a0
update
Oct 30, 2022
1f4f435
Merge branch 'apache:dev' into new-connector-cassandra
FWLamb Oct 31, 2022
cab1739
Merge branch 'apache:dev' into new-connector-cassandra
FWLamb Oct 31, 2022
fced491
update
Oct 31, 2022
7166ad6
update
Oct 31, 2022
413b34c
update
Nov 1, 2022
e0a2a18
update
Nov 1, 2022
803efc5
update
Nov 1, 2022
5fcbfa7
update
Nov 3, 2022
9e9bfc1
update
Nov 3, 2022
b3eb8e6
update
Nov 3, 2022
752be51
update
Nov 4, 2022
a9cd693
update
Nov 4, 2022
8fc4420
update
Nov 4, 2022
89c9a49
Merge branch 'apache:dev' into new-connector-cassandra
FWLamb Nov 4, 2022
4dd7fee
update
Nov 4, 2022
816de34
update
Nov 4, 2022
6f464c1
update
Nov 4, 2022
5f5469c
Merge branch 'dev' into new-connector-cassandra
ic4y Nov 7, 2022
2c9af78
Merge branch 'dev' into new-connector-cassandra
EricJoy2048 Nov 8, 2022
7b5181f
Merge branch 'dev' into new-connector-cassandra
EricJoy2048 Nov 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 52 additions & 50 deletions docs/en/Connector-v2-release-state.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,53 +9,55 @@ SeaTunnel uses a grading system for connectors to help you understand what to ex

## Connector V2 Health

| Connector Name | Type | Status | Support Version |
|-------------------------------------------------------------|--------|--------|-----------------|
| [Asset](connector-v2/sink/Assert.md) | Sink | Beta | 2.2.0-beta |
| [ClickHouse](connector-v2/source/Clickhouse.md) | Source | Beta | 2.2.0-beta |
| [ClickHouse](connector-v2/sink/Clickhouse.md) | Sink | Beta | 2.2.0-beta |
| [ClickHouseFile](connector-v2/sink/ClickhouseFile.md) | Sink | Beta | 2.2.0-beta |
| [Console](connector-v2/sink/Console.md) | Sink | Beta | 2.2.0-beta |
| [DataHub](connector-v2/sink/Datahub.md) | Sink | Alpha | 2.2.0-beta |
| [DingTalk](connector-v2/sink/dingtalk.md) | Sink | Alpha | 2.2.0-beta |
| [Elasticsearch](connector-v2/sink/Elasticsearch.md) | Sink | Beta | 2.2.0-beta |
| [Email](connector-v2/sink/Email.md) | Sink | Alpha | 2.2.0-beta |
| [Enterprise WeChat](connector-v2/sink/Enterprise-WeChat.md) | Sink | Alpha | 2.2.0-beta |
| [FeiShu](connector-v2/sink/Feishu.md) | Sink | Alpha | 2.2.0-beta |
| [Fake](connector-v2/source/FakeSource.md) | Source | Beta | 2.2.0-beta |
| [FtpFile](connector-v2/sink/FtpFile.md) | Sink | Alpha | 2.2.0-beta |
| [Greenplum](connector-v2/sink/Greenplum.md) | Sink | Alpha | 2.2.0-beta |
| [Greenplum](connector-v2/source/Greenplum.md) | Source | Alpha | 2.2.0-beta |
| [HdfsFile](connector-v2/sink/HdfsFile.md) | Sink | Beta | 2.2.0-beta |
| [HdfsFile](connector-v2/source/HdfsFile.md) | Source | Beta | 2.2.0-beta |
| [Hive](connector-v2/sink/Hive.md) | Sink | Beta | 2.2.0-beta |
| [Hive](connector-v2/source/Hive.md) | Source | Beta | 2.2.0-beta |
| [Http](connector-v2/sink/Http.md) | Sink | Beta | 2.2.0-beta |
| [Http](connector-v2/source/Http.md) | Source | Beta | 2.2.0-beta |
| [Hudi](connector-v2/source/Hudi.md) | Source | Alpha | 2.2.0-beta |
| [Iceberg](connector-v2/source/Iceberg.md) | Source | Alpha | 2.2.0-beta |
| [IoTDB](connector-v2/source/IoTDB.md) | Source | Beta | 2.2.0-beta |
| [IoTDB](connector-v2/sink/IoTDB.md) | Sink | Beta | 2.2.0-beta |
| [Jdbc](connector-v2/source/Jdbc.md) | Source | Beta | 2.2.0-beta |
| [Jdbc](connector-v2/sink/Jdbc.md) | Sink | Beta | 2.2.0-beta |
| [Kudu](connector-v2/source/Kudu.md) | Source | Alpha | 2.2.0-beta |
| [Kudu](connector-v2/sink/Kudu.md) | Sink | Alpha | 2.2.0-beta |
| [LocalFile](connector-v2/sink/LocalFile.md) | Sink | Beta | 2.2.0-beta |
| [LocalFile](connector-v2/source/LocalFile.md) | Source | Beta | 2.2.0-beta |
| [MongoDB](connector-v2/source/MongoDB.md) | Source | Beta | 2.2.0-beta |
| [MongoDB](connector-v2/sink/MongoDB.md) | Sink | Beta | 2.2.0-beta |
| [Neo4j](connector-v2/sink/Neo4j.md) | Sink | Alpha | 2.2.0-beta |
| [OssFile](connector-v2/sink/OssFile.md) | Sink | Alpha | 2.2.0-beta |
| [OssFile](connector-v2/source/OssFile.md) | Source | Beta | 2.2.0-beta |
| [Phoenix](connector-v2/sink/Phoenix.md) | Sink | Alpha | 2.2.0-beta |
| [Phoenix](connector-v2/source/Phoenix.md) | Source | Alpha | 2.2.0-beta |
| [Pulsar](connector-v2/source/pulsar.md) | Source | Beta | 2.2.0-beta |
| [Redis](connector-v2/sink/Redis.md) | Sink | Beta | 2.2.0-beta |
| [Redis](connector-v2/source/Redis.md) | Source | Alpha | 2.2.0-beta |
| [Sentry](connector-v2/sink/Sentry.md) | Sink | Alpha | 2.2.0-beta |
| [Socket](connector-v2/sink/Socket.md) | Sink | Alpha | 2.2.0-beta |
| [Socket](connector-v2/source/Socket.md) | Source | Alpha | 2.2.0-beta |
| [Kafka](connector-v2/source/kafka.md) | Source | Alpha | 2.3.0-beta |
| [Kafka](connector-v2/sink/Kafka.md) | Sink | Alpha | 2.3.0-beta |
| [S3File](connector-v2/source/S3File.md) | Source | Alpha | 2.3.0-beta |
| [S3File](connector-v2/sink/S3File.md) | Sink | Alpha | 2.3.0-beta |
| Connector Name | Type | Status | Support Version |
|-------------------------------------------------------------------|--------|--------|-----------------|
| [Asset](connector-v2/sink/Assert.md) | Sink | Beta | 2.2.0-beta |
| [ClickHouse](connector-v2/source/Clickhouse.md) | Source | Beta | 2.2.0-beta |
| [ClickHouse](connector-v2/sink/Clickhouse.md) | Sink | Beta | 2.2.0-beta |
| [ClickHouseFile](connector-v2/sink/ClickhouseFile.md) | Sink | Beta | 2.2.0-beta |
| [Console](connector-v2/sink/Console.md) | Sink | Beta | 2.2.0-beta |
| [DataHub](connector-v2/sink/Datahub.md) | Sink | Alpha | 2.2.0-beta |
| [DingTalk](connector-v2/sink/dingtalk.md) | Sink | Alpha | 2.2.0-beta |
| [Elasticsearch](connector-v2/sink/Elasticsearch.md) | Sink | Beta | 2.2.0-beta |
| [Email](connector-v2/sink/Email.md) | Sink | Alpha | 2.2.0-beta |
| [Enterprise WeChat](connector-v2/sink/Enterprise-WeChat.md) | Sink | Alpha | 2.2.0-beta |
| [FeiShu](connector-v2/sink/Feishu.md) | Sink | Alpha | 2.2.0-beta |
| [Fake](connector-v2/source/FakeSource.md) | Source | Beta | 2.2.0-beta |
| [FtpFile](connector-v2/sink/FtpFile.md) | Sink | Alpha | 2.2.0-beta |
| [Greenplum](connector-v2/sink/Greenplum.md) | Sink | Alpha | 2.2.0-beta |
| [Greenplum](connector-v2/source/Greenplum.md) | Source | Alpha | 2.2.0-beta |
| [HdfsFile](connector-v2/sink/HdfsFile.md) | Sink | Beta | 2.2.0-beta |
| [HdfsFile](connector-v2/source/HdfsFile.md) | Source | Beta | 2.2.0-beta |
| [Hive](connector-v2/sink/Hive.md) | Sink | Beta | 2.2.0-beta |
| [Hive](connector-v2/source/Hive.md) | Source | Beta | 2.2.0-beta |
| [Http](connector-v2/sink/Http.md) | Sink | Beta | 2.2.0-beta |
| [Http](connector-v2/source/Http.md) | Source | Beta | 2.2.0-beta |
| [Hudi](connector-v2/source/Hudi.md) | Source | Alpha | 2.2.0-beta |
| [Iceberg](connector-v2/source/Iceberg.md) | Source | Alpha | 2.2.0-beta |
| [IoTDB](connector-v2/source/IoTDB.md) | Source | Beta | 2.2.0-beta |
| [IoTDB](connector-v2/sink/IoTDB.md) | Sink | Beta | 2.2.0-beta |
| [Jdbc](connector-v2/source/Jdbc.md) | Source | Beta | 2.2.0-beta |
| [Jdbc](connector-v2/sink/Jdbc.md) | Sink | Beta | 2.2.0-beta |
| [Kudu](connector-v2/source/Kudu.md) | Source | Alpha | 2.2.0-beta |
| [Kudu](connector-v2/sink/Kudu.md) | Sink | Alpha | 2.2.0-beta |
| [LocalFile](connector-v2/sink/LocalFile.md) | Sink | Beta | 2.2.0-beta |
| [LocalFile](connector-v2/source/LocalFile.md) | Source | Beta | 2.2.0-beta |
| [MongoDB](connector-v2/source/MongoDB.md) | Source | Beta | 2.2.0-beta |
| [MongoDB](connector-v2/sink/MongoDB.md) | Sink | Beta | 2.2.0-beta |
| [Neo4j](connector-v2/sink/Neo4j.md) | Sink | Alpha | 2.2.0-beta |
| [OssFile](connector-v2/sink/OssFile.md) | Sink | Alpha | 2.2.0-beta |
| [OssFile](connector-v2/source/OssFile.md) | Source | Beta | 2.2.0-beta |
| [Phoenix](connector-v2/sink/Phoenix.md) | Sink | Alpha | 2.2.0-beta |
| [Phoenix](connector-v2/source/Phoenix.md) | Source | Alpha | 2.2.0-beta |
| [Pulsar](connector-v2/source/pulsar.md) | Source | Beta | 2.2.0-beta |
| [Redis](connector-v2/sink/Redis.md) | Sink | Beta | 2.2.0-beta |
| [Redis](connector-v2/source/Redis.md) | Source | Alpha | 2.2.0-beta |
| [Sentry](connector-v2/sink/Sentry.md) | Sink | Alpha | 2.2.0-beta |
| [Socket](connector-v2/sink/Socket.md) | Sink | Alpha | 2.2.0-beta |
| [Socket](connector-v2/source/Socket.md) | Source | Alpha | 2.2.0-beta |
| [Kafka](connector-v2/source/kafka.md) | Source | Alpha | 2.3.0-beta |
| [Kafka](connector-v2/sink/Kafka.md) | Sink | Alpha | 2.3.0-beta |
| [S3File](connector-v2/source/S3File.md) | Source | Alpha | 2.3.0-beta |
| [S3File](connector-v2/sink/S3File.md) | Sink | Alpha | 2.3.0-beta |
| [Cassandra](connector-v2/source/Cassandra.md) | Source | Alpha | 2.3.0-beta |
| [Cassandra](connector-v2/sink/Cassandra.md) | Sink | Alpha | 2.3.0-beta |
98 changes: 98 additions & 0 deletions docs/en/connector-v2/sink/Cassandra.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Cassandra

> Cassandra sink connector

## Description

Write data to Apache Cassandra.

## Key features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [schema projection](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-------------------|--------|----------|---------------|
| host | String | Yes | - |
| keyspace | String | Yes | - |
| table | String | Yes | - |
| username | String | No | - |
| password | String | No | - |
| datacenter | String | No | datacenter1 |
| consistency_level | String | No | LOCAL_ONE |
| fields | String | No | LOCAL_ONE |
| batch_size | String | No | 5000 |
| batch_type | String | No | UNLOGGER |
| async_write | String | No | true |

### host [string]

`Cassandra` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as
`"cassandra1:9042,cassandra2:9042"`.

### keyspace [string]

The `Cassandra` keyspace.

### table [String]

The `Cassandra` table name.

### username [string]

`Cassandra` user username.

### password [string]

`Cassandra` user password.

### datacenter [String]

The `Cassandra` datacenter, default is `datacenter1`.

### consistency_level [String]

The `Cassandra` write consistency level, default is `LOCAL_ONE`.

### fields [array]

The data field that needs to be output to `Cassandra` , if not configured, it will be automatically adapted
according to the sink table `schema`.

### batch_size [number]

The number of rows written through [Cassandra-Java-Driver](https://github.com/datastax/java-driver) each time,
default is `5000`.

### batch_type [String]

The `Cassandra` batch processing mode, default is `UNLOGGER`.

### async_write [boolean]

Whether `cassandra` writes in asynchronous mode, default is `true`.

## Examples

```hocon
sink {
Cassandra {
host = "localhost:9042"
username = "cassandra"
password = "cassandra"
datacenter = "datacenter1"
keyspace = "test"
}
}
```

## Changelog

### next version

- Add Cassandra Sink Connector



FWLamb marked this conversation as resolved.
Show resolved Hide resolved
82 changes: 82 additions & 0 deletions docs/en/connector-v2/source/Cassandra.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Cassandra

> Cassandra source connector

## Description

Read data from Apache Cassandra.

## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [schema projection](../../concept/connector-v2-features.md)
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-------------------------|--------|----------|---------------|
| host | String | Yes | - |
| keyspace | String | Yes | - |
| cql | String | Yes | - |
| username | String | No | - |
| password | String | No | - |
| datacenter | String | No | datacenter1 |
| consistency_level | String | No | LOCAL_ONE |

### host [string]

`Cassandra` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as
`"cassandra1:9042,cassandra2:9042"`.

### keyspace [string]

The `Cassandra` keyspace.

### cql [String]

The query cql used to search data though Cassandra session.

### username [string]

`Cassandra` user username.

### password [string]

`Cassandra` user password.

### datacenter [String]

The `Cassandra` datacenter, default is `datacenter1`.

### consistency_level [String]

The `Cassandra` write consistency level, default is `LOCAL_ONE`.

## Examples

```hocon
source {
Cassandra {
host = "localhost:9042"
username = "cassandra"
password = "cassandra"
datacenter = "datacenter1"
keyspace = "test"
cql = "select * from source_table"
result_table_name = "source_table"
}
}
```

## Changelog

### next version

- Add Cassandra Source Connector


EricJoy2048 marked this conversation as resolved.
Show resolved Hide resolved

3 changes: 3 additions & 0 deletions plugin-mapping.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,4 +139,7 @@ seatunnel.source.S3File = connector-file-s3
seatunnel.sink.S3File = connector-file-s3
seatunnel.source.Amazondynamodb = connector-amazondynamodb
seatunnel.sink.Amazondynamodb = connector-amazondynamodb
seatunnel.source.Cassandra = connector-cassandra
seatunnel.sink.Cassandra = connector-cassandra
seatunnel.sink.StarRocks = connector-starrocks

52 changes: 52 additions & 0 deletions seatunnel-connectors-v2/connector-cassandra/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>seatunnel-connectors-v2</artifactId>
<groupId>org.apache.seatunnel</groupId>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>connector-cassandra</artifactId>

<properties>
<cassandra.driver.version>4.14.0</cassandra.driver.version>
</properties>

<dependencies>
<dependency>
<groupId>com.datastax.oss</groupId>
<artifactId>java-driver-core</artifactId>
<version>${cassandra.driver.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-common</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>


</project>
Loading