Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DLQ: Kafka sources and/or sinks #4267

Open
franky-m opened this issue Mar 12, 2024 · 1 comment
Open

DLQ: Kafka sources and/or sinks #4267

franky-m opened this issue Mar 12, 2024 · 1 comment
Labels
enhancement New feature or request plugin - sink A plugin to write data to a destination.

Comments

@franky-m
Copy link
Contributor

Is your feature request related to a problem? Please describe.
At present, the DLQ only supports S3 sources. To my understanding possible sinks are currently: text files (dlq_file), S3 storage and opensearch.

Describe the solution you’d like
We would like to use a DLQ with Kafka as a source. Ideally, the DLQ sink would be Kafka as well, i.e., the entire failed message would be copied into a separate DLQ-Kafka topic.

Alternatively, only the Key of the failed Kafka-message could be written to the sink (in which case it could also be written nicely to opensearch, dlq_file or S3 storage, even if the message is larger or has a broken json structure).

@dlvenable
Copy link
Member

@franky-m , If I understand your request, you are looking to write failed events into a Kafka-based DLQ. Is that correct?

We have an existing issue discussing the idea of a pipeline-level DLQ - #3857. As part of that concept, we would also allow any sink to be usable as a DLQ sink. In this way, we don't need to re-implement the sink for the DLQ purposes.

We also have some incomplete work to support a kafka sink.

If you are interested in helping pick up any of these tasks, please let us know. I'd be happy to help you get started and point you to some relevant code to change.

@dlvenable dlvenable added plugin - sink A plugin to write data to a destination. enhancement New feature or request and removed untriaged labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin - sink A plugin to write data to a destination.
Projects
Development

No branches or pull requests

2 participants