You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Data Prepper supports creating a pipeline which uses the pipeline sink to send events to multiple downstream pipelines. However, the Event is not being copied. Thus, the same Event object is passed into multiple threads. This produces unexpected behavior.
To Reproduce
Create a pipeline which modifies an Event in a downstream pipeline.
Expected behavior
Modifying an Event in one pipeline should not modify it in others. The PipelineConnector should duplicate the Events.
Additional context
The Event model contains metadata, making it hard to "copy" an event. We solved this in peer-forwarder with a quick fix which is can be seen in this PR. However, this approach is not highly extensible since it requires a lot of knowledge duplicated throughout the code.
The text was updated successfully, but these errors were encountered:
Describe the bug
Data Prepper supports creating a pipeline which uses the pipeline sink to send events to multiple downstream pipelines. However, the Event is not being copied. Thus, the same Event object is passed into multiple threads. This produces unexpected behavior.
To Reproduce
Create a pipeline which modifies an Event in a downstream pipeline.
Expected behavior
Modifying an Event in one pipeline should not modify it in others. The
PipelineConnector
should duplicate the Events.Additional context
The
Event
model contains metadata, making it hard to "copy" an event. We solved this in peer-forwarder with a quick fix which is can be seen in this PR. However, this approach is not highly extensible since it requires a lot of knowledge duplicated throughout the code.The text was updated successfully, but these errors were encountered: