-
Notifications
You must be signed in to change notification settings - Fork 0
Kafka & Iceberg
Cesar Celis Hernandez edited this page Dec 2, 2022
·
12 revisions
To use Iceberg Format in Kafka
- Get Kafka Running:
# Window 1:
# Download files:
rm -rf ~/kafka
mkdir ~/kafka
cd ~/kafka; wget https://dlcdn.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
tar -xzf kafka_2.13-3.3.1.tgz
cd kafka_2.13-3.3.1;
pwd;
# Window 1:
# Kafka with ZooKeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
# Window 2:
# Start the Kafka broker service:
bin/kafka-server-start.sh config/server.properties
- Get
kafka-connect-iceberg-sink
fromgetindata
based onmemiiso/debezium-server-iceberg
cd ~
rm -rf ~/kafka-connect-iceberg-sink
mkdir ~/kafka-connect-iceberg-sink
cd ~/kafka-connect-iceberg-sink
wget https://github.com/getindata/kafka-connect-iceberg-sink/releases/download/0.2.0/kafka-connect-iceberg-sink-0.2.0-plugin.zip
unzip kafka-connect-iceberg-sink-0.2.0-plugin.zip
rm kafka-connect-iceberg-sink-0.2.0-plugin.zip
cd kafka-connect-iceberg-sink
- From
getindata
you got ajar
file:
$ ls kafka-connect-iceberg-sink-0.2.0.jar
kafka-connect-iceberg-sink-0.2.0.jar
- Run
Debezium
container withDocker
:
docker run -it --name connect --net=host -p 8083:8083 \
-e GROUP_ID=1 \
-e CONFIG_STORAGE_TOPIC=my-connect-configs \
-e OFFSET_STORAGE_TOPIC=my-connect-offsets \
-e BOOTSTRAP_SERVERS=host.docker.internal:9092 \
-e CONNECT_TOPIC_CREATION_ENABLE=true \
-v ~/.aws/credentials:/kafka/.aws/credentials \
-v /Users/cniackz/kafka-connect-iceberg-sink/kafka-connect-iceberg-sink/kafka-connect-iceberg-sink-0.2.0.jar:/kafka/connect/kafka-connect-iceberg-sink-0.2.0.jar \
debezium/connect
- run
PostgreSQL
onDocker
:
docker run -d --name postgres -e POSTGRES_PASSWORD=postgres \
-p 5432:5432 postgres -c wal_level=logical