Make a replication of a source Kafka (Not Event Streams) to an EventStreams Kafka. As part of the demo, we will backup topics and consumer lags.
- Working instance of Event Streams.
- A source Kafka cluster (e.g. Strimzi).
- Take note of Source Kafka details. If you do not have your own Kafka instance, you can use this strimzi instance.
Take note of the topic name, consumer group, consumer lags and last offset message.
-
Create a MM2 CR.
Download the sample yaml file from here.
Edit the yaml and replace all instances of <student-id> with your actual student-id.
Take a look and understand the following important fields (no changes needed)sync.group.offsets.enabled: should be true to replicate consumer lags. Source bootstrapServer: the bootstrap URL of the source strimzi server. Destination bootstrapServer: the bootstrap server of the destination Kafka. topicsPattern: the topic in strimzi that will be replicated. Wild card permitted. groupsPattern: the consumer groups that should be replicated. The following fields are useful but are not used in our labs: replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy. This will maintain the topics name is source and target without adding any prefix.
-
Create the MM2 instance.
Login to the Openshift Console. Go to "Installed Operators" -> IBM Event Streams -> Kafka Mirror Maker 2.
Click on "Create Kafka MirrorMaker2:.
Paste the CR (yaml file) you created in the above step. Click on create. -
This will create a MirrorMaker2 instance and it will also start replicating topics and consumer lags.
-
Verify the replication (topics).
Check the EventStreams web console and a new topic should have been created. Look for the replicated topic.
A prefix will be added to the topic name.
- Verify the replication (consumer lags)
Check consumer lags in Event Streams.
- Go to Event Streams home page -> Topics -> Click on the replicated topic -> Click on “Consumer Groups”.
The new consumer group should appear and the offset lag should match with that of strimzi.
- Do a random check on the timestamp of a message.
Great. You have sucessfully replicated a topic and consumer group lags from Strimzi to Event Streams.