Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a generic codec structure for sinks #2403

Closed
dlvenable opened this issue Mar 27, 2023 · 0 comments · Fixed by #2842, #2845, #2881, #2928 or #2986
Closed

Support a generic codec structure for sinks #2403

dlvenable opened this issue Mar 27, 2023 · 0 comments · Fixed by #2842, #2845, #2881, #2928 or #2986
Labels
enhancement New feature or request

Comments

@dlvenable
Copy link
Member

dlvenable commented Mar 27, 2023

Is your feature request related to a problem? Please describe.

Data Prepper supports a file sink and has plans to support an S3 Sink (#1048). Sinks like these can benefit from a sink codec concept similar to the source codec (#1532).

Describe the solution you'd like

Create an interface in data-prepper-api which can transform events into an output format for consumption by participating sinks.

Proposed interface

interface OutputCodec {

  /**
   * Called by the sink when a new destination is in use.
   */
  void start(OutputStream outputStream);

  /**
   * Called by the sink before saving the destination to the external sink.
   */
  void complete(OutputStream outputStream);

  /**
   * Writes an event into the underlying stream.
   */
  void writeEvent(Event event, OutputStream outputStream);

  /**
   * Returns the expected file extension for this type of codec.
   * For example, json, csv.
   */
  String getExtension();
}

I included the start and complete interfaces because some codecs like JSON array need to create an initial wrapping and then close that.

Additionally, the getExtension method can be used to determine the file name when the sink is writing to a file or file-like system (e.g. S3).

Plugin names

Use the same name as the corresponding input plugin when appropriate. Thus, we will have an input plugin for newline and an output plugin for newline. The Data Prepper plugin framework permits this because they implement different interfaces.

Project structure

Include these new output codecs in the same Gradle projects as the input codec.

For example, both the input and output newline codecs should be in the data-prepper-plugins/newline-codecs project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment