Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2][Elasticsearch] Support Elasticsearch source #2821

Merged
merged 26 commits into from
Nov 4, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
a96b361
[Feature][Connector-V2]improve elasticseach connector directory struc…
Aug 28, 2022
c7050d7
[Feature][Connector-V2]new connector of Elasticsearch source(#2553)
Sep 20, 2022
66f9ad9
[Feature][Connector-V2]improve elasticseach connector directory struc…
Aug 28, 2022
b6688c2
[Feature][Connector-V2]new connector of Elasticsearch source(#2553)
Sep 20, 2022
a8f96fe
Merge branch 'dev-20220919' of github.com:iture123/incubator-seatunne…
Sep 22, 2022
3e4538e
Merge branch 'dev' of github.com:iture123/incubator-seatunnel into de…
Sep 24, 2022
c878685
[Feature][Connector-V2]resolve conflict(#2553)
Sep 24, 2022
5cba46f
[Feature][Connector-V2]remove checkstyle:MagicNumber(#2553)
Sep 27, 2022
83e288b
[Feature][Connector-V2]use lombok to simple code(#2553)
Sep 28, 2022
c70f2af
[Feature][Connector-V2]add 'synchronized (output.getCheckpointLock())…
Sep 28, 2022
5a02fc9
[Feature][Connector-V2][Elasticsearce]simplify Config.class path(#2553)
Oct 15, 2022
ad71a2a
[Feature][Connector-V2][Elasticsearce]improve elasticsearch record de…
Oct 15, 2022
d39c5cb
[Feature][Connector-V2][Elasticsearce]EsRestClient#getFieldTypeMappin…
Oct 16, 2022
ed2ba50
[Feature][Connector-V2][Elasticsearce]improve restore split(#2553)
Oct 16, 2022
a184fac
[Feature][Connector-V2][Elasticsearce]improve Elasticsearc.md(#2553)
Oct 16, 2022
dbb44ad
[Feature][Connector-V2][Elasticsearce]fix NPP when ES version <= 6.x(…
Oct 16, 2022
62f299f
[Feature][Connector-V2][Elasticsearce]improve elasticsearch source(#2…
Oct 22, 2022
d6f9282
Merge branch 'dev' of github.com:iture123/incubator-seatunnel into de…
Oct 22, 2022
4a3b560
[Feature][Connector-V2][Elasticsearce]fix jackson conflict in spark a…
Oct 22, 2022
388226b
[Feature][Connector-V2][Elasticsearce]pom.xml remove dependency seatu…
Oct 30, 2022
d53b879
[Feature][Connector-V2][Elasticsearce]improve ElasticsearchSinkWriter…
Oct 30, 2022
b51b058
[Feature][Connector-V2][Elasticsearce]improve ElasticsearchSinkWriter…
Oct 30, 2022
b6c318d
[Feature][Connector-V2][Elasticsearce]improve ElasticsearchIT
Oct 31, 2022
6eb559f
[Feature][Connector-V2][Elasticsearce]improve ElasticsearchSourceReader
Oct 31, 2022
0ccdca0
[Feature][Connector-V2][Elasticsearce]improve ElasticsearchSourceSpli…
Oct 31, 2022
4232854
[Feature][Connector-V2][Elasticsearce]ElasticsearchSourceSplit use lo…
Oct 31, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions docs/en/connector-v2/source/Elasticsearch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Elasticsearch

> Elasticsearch source connector

## Description

Used to read data from Elasticsearch.

support version >= 2.x and < 8.x.

## Key features

- [x] [batch](../../concept/connector-v2-features.md)
- [ ] [stream](../../concept/connector-v2-features.md)
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [schema projection](../../concept/connector-v2-features.md)
- [ ] [parallelism](../../concept/connector-v2-features.md)
- [ ] [support user-defined split](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-------------|--------| -------- |---------------|
| hosts | array | yes | - |
| username | string | no | - |
| password | string | no | - |
| index | string | yes | - |
| source | array | yes | - |
| scroll_time | string | no | 1m |
| scroll_size | int | no | 100 |



### hosts [array]
Elasticsearch cluster http address, the format is `host:port`, allowing multiple hosts to be specified. Such as `["host1:9200", "host2:9200"]`.

### username [string]
x-pack username.

### password [string]
x-pack password.

### index [string]
Elasticsearch index name, support * fuzzy matching.

### source [array]
The fields of index.
You can get the document id by specifying the field `_id`.If sink _id to other index,you need specify an alias for _id due to the Elasticsearch limit.

### scroll_time [String]
Amount of time Elasticsearch will keep the search context alive for scroll requests.

### scroll_size [int]
Maximum number of hits to be returned with each Elasticsearch scroll request.

## Examples
simple
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add complex example

```hocon
Elasticsearch {
hosts = ["localhost:9200"]
index = "seatunnel-*"
source = ["_id","name","age"]
}
```
complex
```hocon
Elasticsearch {
hosts = ["elasticsearch:9200"]
index = "st_index"
schema = {
fields {
c_map = "map<string, tinyint>"
c_array = "array<tinyint>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_decimal = "decimal(2, 1)"
c_bytes = bytes
c_date = date
c_timestamp = timestamp
}
}
}
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 changes: 2 additions & 1 deletion plugin-mapping.properties
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ seatunnel.sink.OssFile = connector-file-oss
seatunnel.source.Pulsar = connector-pulsar
seatunnel.source.Hudi = connector-hudi
seatunnel.sink.DingTalk = connector-dingtalk
seatunnel.sink.elasticsearch = connector-elasticsearch
seatunnel.source.Elasticsearch = connector-elasticsearch
seatunnel.sink.Elasticsearch = connector-elasticsearch
seatunnel.source.IoTDB = connector-iotdb
seatunnel.sink.IoTDB = connector-iotdb
seatunnel.source.Neo4j = connector-neo4j
Expand Down
19 changes: 8 additions & 11 deletions seatunnel-connectors-v2/connector-elasticsearch/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,8 @@
<modelVersion>4.0.0</modelVersion>

<artifactId>connector-elasticsearch</artifactId>

<properties>
<elasticsearch-rest-client.version>7.5.1</elasticsearch-rest-client.version>
<jackson.databind.version>2.12.6</jackson.databind.version>
<jackson-datatype-jsr310.version>2.13.3</jackson-datatype-jsr310.version>
</properties>

<dependencies>
Expand All @@ -42,15 +39,15 @@
<version>${elasticsearch-rest-client.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.databind.version}</version>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-common</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
<version>${jackson-datatype-jsr310.version}</version>
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-format-json</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

</project>
</project>
Loading