You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Data Prepper supports a file sink and has plans to support an S3 Sink (#1048). Sinks like these can benefit from a sink codec concept similar to the source codec (#1532).
Describe the solution you'd like
Create an interface in data-prepper-api which can transform events into an output format for consumption by participating sinks.
Proposed interface
interface OutputCodec {
/**
* Called by the sink when a new destination is in use.
*/
void start(OutputStream outputStream);
/**
* Called by the sink before saving the destination to the external sink.
*/
void complete(OutputStream outputStream);
/**
* Writes an event into the underlying stream.
*/
void writeEvent(Event event, OutputStream outputStream);
/**
* Returns the expected file extension for this type of codec.
* For example, json, csv.
*/
String getExtension();
}
I included the start and complete interfaces because some codecs like JSON array need to create an initial wrapping and then close that.
Additionally, the getExtension method can be used to determine the file name when the sink is writing to a file or file-like system (e.g. S3).
Plugin names
Use the same name as the corresponding input plugin when appropriate. Thus, we will have an input plugin for newline and an output plugin for newline. The Data Prepper plugin framework permits this because they implement different interfaces.
Project structure
Include these new output codecs in the same Gradle projects as the input codec.
For example, both the input and output newline codecs should be in the data-prepper-plugins/newline-codecs project.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Data Prepper supports a file sink and has plans to support an S3 Sink (#1048). Sinks like these can benefit from a sink codec concept similar to the source codec (#1532).
Describe the solution you'd like
Create an interface in
data-prepper-api
which can transform events into an output format for consumption by participating sinks.Proposed interface
I included the
start
andcomplete
interfaces because some codecs like JSON array need to create an initial wrapping and then close that.Additionally, the
getExtension
method can be used to determine the file name when the sink is writing to a file or file-like system (e.g. S3).Plugin names
Use the same name as the corresponding input plugin when appropriate. Thus, we will have an input plugin for
newline
and an output plugin fornewline
. The Data Prepper plugin framework permits this because they implement different interfaces.Project structure
Include these new output codecs in the same Gradle projects as the input codec.
For example, both the input and output
newline
codecs should be in thedata-prepper-plugins/newline-codecs
project.The text was updated successfully, but these errors were encountered: