You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
At present, the DLQ only supports S3 sources. To my understanding possible sinks are currently: text files (dlq_file), S3 storage and opensearch.
Describe the solution you’d like
We would like to use a DLQ with Kafka as a source. Ideally, the DLQ sink would be Kafka as well, i.e., the entire failed message would be copied into a separate DLQ-Kafka topic.
Alternatively, only the Key of the failed Kafka-message could be written to the sink (in which case it could also be written nicely to opensearch, dlq_file or S3 storage, even if the message is larger or has a broken json structure).
The text was updated successfully, but these errors were encountered:
@franky-m , If I understand your request, you are looking to write failed events into a Kafka-based DLQ. Is that correct?
We have an existing issue discussing the idea of a pipeline-level DLQ - #3857. As part of that concept, we would also allow any sink to be usable as a DLQ sink. In this way, we don't need to re-implement the sink for the DLQ purposes.
We also have some incomplete work to support a kafka sink.
If you are interested in helping pick up any of these tasks, please let us know. I'd be happy to help you get started and point you to some relevant code to change.
Is your feature request related to a problem? Please describe.
At present, the DLQ only supports S3 sources. To my understanding possible sinks are currently: text files (
dlq_file
), S3 storage and opensearch.Describe the solution you’d like
We would like to use a DLQ with Kafka as a source. Ideally, the DLQ sink would be Kafka as well, i.e., the entire failed message would be copied into a separate DLQ-Kafka topic.
Alternatively, only the Key of the failed Kafka-message could be written to the sink (in which case it could also be written nicely to opensearch,
dlq_file
or S3 storage, even if the message is larger or has a broken json structure).The text was updated successfully, but these errors were encountered: