Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added connector hub support. Fixes #3 #4

Merged
merged 1 commit into from
Apr 3, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,5 @@

# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*
target
.okhttpcache
57 changes: 44 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,66 @@

# Introduction



# Transformations


## Archive

The Archive transformation is used to help preserve all of the data for a message when archived to S3.




### Note

This transform works by copying the key, value, topic, and timestamp to new record where this is all contained in the value of the message. This will allow connectors like Confluent's S3 connector to properly archive the record.


### Configuration

| Name | Type | Importance | Default Value | Validator | Documentation|
| ---- | ---- | ---------- | ------------- | --------- | -------------|

#### Examples

##### Standalone Example

#### Standalone Example
This configuration is used typically along with [standalone mode](http://docs.confluent.io/current/connect/concepts.html#standalone-workers).

```properties
transforms=Archive
transforms.Archive.type=com.github.jcustenborder.kafka.connect.archive.Archive
# The following values must be configured.
name=Connector1
connector.class=org.apache.kafka.some.SourceConnector
tasks.max=1
transforms=tran
transforms.tran.type=com.github.jcustenborder.kafka.connect.archive.Archive
```

#### Distributed Example
##### Distributed Example

This configuration is used typically along with [distributed mode](http://docs.confluent.io/current/connect/concepts.html#distributed-workers).
Write the following json to `connector.json`, configure all of the required values, and use the command below to
post the configuration to one the distributed connect worker(s).

```json
{
"name": "connector1",
"config": {
"connector.class": "com.github.jcustenborder.kafka.connect.archive.Archive",
"transforms": "Archive",
"transforms.Archive.type": "com.github.jcustenborder.kafka.connect.archive.Archive",
}
"name" : "Connector1",
"connector.class" : "org.apache.kafka.some.SourceConnector",
"transforms" : "tran",
"transforms.tran.type" : "com.github.jcustenborder.kafka.connect.archive.Archive"
}
```

Use curl to post the configuration to one of the Kafka Connect Workers. Change `http://localhost:8083/` the the endpoint of
one of your Kafka Connect worker(s).

Create a new instance.
```bash
curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors
```

Update an existing instance.
```bash
curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/TestSinkConnector1/config
```


39 changes: 39 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
<name>kafka-connect-transform-archive</name>
<url>https://github.com/jcustenborder/kafka-connect-transform-archive</url>
<inceptionYear>2017</inceptionYear>
<description>Kafka Connect transformation used to help with archiving data to s3. This transformation copies the timestamp, topic, key, and value to a struct in the value of the record.</description>
<licenses>
<license>
<name>Apache License 2.0</name>
Expand Down Expand Up @@ -74,4 +75,42 @@
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>io.confluent</groupId>
<artifactId>kafka-connect-maven-plugin</artifactId>
<version>0.9.0</version>
<executions>
<execution>
<goals>
<goal>kafka-connect</goal>
</goals>
<configuration>
<ownerUsername>jcustenborder</ownerUsername>
<confluentControlCenterIntegration>true
</confluentControlCenterIntegration>
<documentationUrl>
https://jcustenborder.github.io/kafka-connect-documentation/
</documentationUrl>
<ownerName>Jeremy Custenborder</ownerName>
<dockerNamespace>jcustenborder</dockerNamespace>
<dockerName>kafka-connect-docker</dockerName>
<pluginTypes>
<pluginType>transform</pluginType>
</pluginTypes>
<tags>
<tag>Archive</tag>
<tag>S3</tag>
</tags>
<title>Archive Transformation</title>
<supportUrl>${pom.issueManagement.url}</supportUrl>
<supportSummary>Support provided through community involvement.
</supportSummary>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>