Kafka Connector

Contents

• Overview
• Summary
• Source Files
• Configuration
  • Shared Configuration
  • Connector-Specific Configuration
• Diagnostics
• Metrics

Overview

• KafkaConnector is intended for consuming a single Kafka topic

Summary

Supports Datastream Updates
Checkpoint Type
Diagnostics Aware

Source Files

1. com.linkedin.datastream.connectors.kafka.KafkaConnector

2. com.linkedin.datastream.connectors.kafka.AbstractKafkaConnector

Configuration

• All Connector config properties must be prefixed with brooklin.server.connector.<connectorName>

• connectorName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames)

Shared Configuration

Configuration properties shared among all Kafka Connectors in Brooklin are documented on this page.

Connector-Specific Configuration

Configuration properties specific to KafkaConnector are listed below.

Property	Description	Default
isGroupIdHashingEnabled	A flag indicating whether Kafka consumer group ID should be hashed If true, the consumer group ID for a datastream is set to <clusterName>.<groupIdMD5Hash>, where: <clusterName>: Brooklin cluster name <groupIdMD5Hash>: The MD5 hash of metadata.system.taskPrefix property of the datastream If false, the consumer group ID for a datastream is set to <sourceConnectionString>-to-<destConnectionString>, where: <sourceConnectionString>: source.connectionString of the datastream <destConnectionString>: destination.connectionString of the datastream	false
whiteListedClusters	A comma-separated list of Kafka brokers, e.g. host1:port1,host2:port2 Any datastream using this Connector must specify one of these Kafka brokers in its source.connectionString	(None)

Diagnostics

Metrics