Consume Kafka Messages from Fetcher and Index Them into OpenSearch

[olga-casian]

Update the Data Processor service to consume messages from Kafka produced by the Fetcher service and index them into OpenSearch. This will enable real-time indexing and searching of XKCD comic data.

Tasks:

1. Set Up Kafka Consumer in Data Processor:

   • Implement a Kafka consumer in the Data Processor service.
   • Configure the consumer to subscribe to the relevant Kafka topic.

2. Process Kafka Messages:

   • Parse and process the incoming messages from Kafka.
   • Ensure data integrity and handle any potential errors.

3. Index Data into OpenSearch:

   • Implement functionality to index the processed data into OpenSearch.
   • Ensure the data is correctly formatted and searchable in OpenSearch.

4. Testing:

   • Write unit and integration tests to validate Kafka message consumption and OpenSearch indexing.
   • Test edge cases such as message failures and reprocessing.

5. Update Documentation:

   • Document the Kafka consumer setup and configuration in the Data Processor README.
   • Include instructions on monitoring and troubleshooting the Kafka consumer and OpenSearch indexing process.

Acceptance Criteria:

• Kafka consumer is implemented and configured in the Data Processor service.
• Messages from Kafka are processed and indexed into OpenSearch.
• Tests pass successfully, validating the end-to-end flow from Kafka to OpenSearch.
• Documentation is updated with Kafka consumer and OpenSearch indexing instructions.