Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support encoding JSON #832

Closed
Tracked by #4460
dlvenable opened this issue Jan 6, 2022 · 5 comments · Fixed by #4514
Closed
Tracked by #4460

Support encoding JSON #832

dlvenable opened this issue Jan 6, 2022 · 5 comments · Fixed by #4514
Assignees
Labels
enhancement New feature or request plugin - processor A plugin to manipulate data in the data prepper pipeline.
Milestone

Comments

@dlvenable
Copy link
Member

Is your feature request related to a problem? Please describe.

In some scenarios, pipelines should be able to encode objects as a JSON string. Data Prepper should provide a feature for this.

Describe the solution you'd like

Provide a processor for encoding JSON - encode_json. It should be the inverse of the JSON parsing processor from #831.

Example

Given the following configuration:

processor:
  encode_json:
    source: my_object
    target: json_string

Given this input event:

"my_object" : {
  "key1" : "value1",
  "key2" : "value2",
}

The event is changed to:

"my_object" : {
  "key1" : "value1",
  "key2" : "value2",
}
"json_string" : "{\"key1\" : \"value1\", \"key2\" : \"value2\"}"
@dlvenable
Copy link
Member Author

Per the request in #852, it would be good to keep these different encoders similar.

Two possible options to consider:

  1. Create an abstract base class for encoding and a standard encoding configuration model. Extend this for both the encode_json and encode_message_pack plugins.

This might look like:

public abstract class AbstractEncoder implements Processor {
   ...
   abstract String encode(Map<String, Object> target);
}

class EncoderConfig {
   private String source;
   private String target;
   ...
}

@DataPrepperPlugin(name = "encode_json", pluginConfigurationType = EncoderConfig.class)
public class JsonEncoder extends AbstractEncoder{
...
}

@DataPrepperPlugin(name = "encode_message_pack", pluginConfigurationType = EncoderConfig.class)
public class MessagePackEncoder extends AbstractEncoder{
...
}

  1. Create a single encoder processor which supports various encodings via a plugin.

This could be configured by authors like:

processor:
  encoder:
    encoding: json
    source: my_object
    target: json_string
  encoder:
    encoding: message_pack
    source: my_object
    target: message_pack_string

@dlvenable dlvenable added enhancement New feature or request plugin - processor A plugin to manipulate data in the data prepper pipeline. labels Jan 25, 2022
@cmanning09 cmanning09 modified the milestones: v1.3, v2.0 Feb 3, 2022
@dlvenable dlvenable removed this from the v2.0 milestone Jun 25, 2022
@sharraj
Copy link

sharraj commented Jan 4, 2024

We should not have processor names with actions in it. Ideally processor name should be "json" with action as "encode json". There can be cases where we need same json processor with say action "decode json".

@kkondaka
Copy link
Collaborator

kkondaka commented May 2, 2024

@dlvenable @sharraj How about this?

processor:
   - write_json:
        serialization_type: "string" (or XMLstring or something else in future)
        source: my_object
        target: target_name

@dlvenable
Copy link
Member Author

@kkondaka , I like write_json.

I'm unclear on what serialization_type would mean. What would it be other than a string? The output from the processor should always be a JSON string.

@kkondaka
Copy link
Collaborator

kkondaka commented May 7, 2024

@dlvenable it could be XML, for example

@kkondaka kkondaka mentioned this issue May 7, 2024
2 tasks
@github-project-automation github-project-automation bot moved this from Unplanned to Done in Data Prepper Tracking Board May 9, 2024
@dlvenable dlvenable added this to the v2.8 milestone May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin - processor A plugin to manipulate data in the data prepper pipeline.
Projects
Development

Successfully merging a pull request may close this issue.

4 participants