[BUG] : unable to set field_delimiter_regex

[efloresb-tibco]

Describe the bug
While trying to set field_delimiter_regex, I see the following error Caused by: java.lang.IllegalArgumentException: field_delimiter_regex and field_split_characters cannot both be defined., I am not setting field_split_characters, I assume it is using the default one.

To Reproduce

1. Use the configuration mentioned in the documentation
   field_delimiter_regex: "&\\{2\\}"
2. Star the docker container

docker run --rm --name data-prepper -p 2021:2021-v $(pwd)/pipeline.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml -v $(pwd)/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml -v data-prepper-out:/usr/share/log opensearchproject/data-prepper:latest

4. See error

Caused by: java.lang.IllegalArgumentException: field_delimiter_regex and field_split_characters cannot both be defined.
	at org.opensearch.dataprepper.plugins.processor.keyvalue.KeyValueProcessor.<init>(KeyValueProcessor.java:45) ~[key-value-processor-2.2.1.jar:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.dataprepper.plugin.PluginCreator.newPluginInstance(PluginCreator.java:40) ~[data-prepper-core-2.2.1.jar:?]
	... 35 more

Pipeline config

log-pipeline:
  source:
    http:
  processor:
    - key_value:
        field_delimiter_regex: "&\\{2\\}"
        source: "log"
  sink:
    - file:
        path: /usr/share/log/output-file

[graytaylor0]

This is a bug. The field_split_characters is given a default value of & here (

data-prepper/data-prepper-plugins/key-value-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/keyvalue/KeyValueProcessorConfig.java

Line 37 in 05d229a

private String fieldSplitCharacters = DEFAULT_FIELD_SPLIT_CHARACTERS;

) and so it fails the validation here (https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/key-value-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/keyvalue/KeyValueProcessor.java#L48).

You should be able to get around this until the bug is fixed by changing your config to

log-pipeline:
  source:
    http:
  processor:
    - key_value:
        field_split_characters: null
        field_delimiter_regex: "&\\{2\\}"
        source: "log"
  sink:
    - file:
        path: /usr/share/log/output-file

[bbgu1]

@graytaylor0 The workaround doesn't work, as field_split_characters is set to be @NotEmpty. Setting it to null or empty string causes the plug-in initialization to fail with error "fieldSplitCharacters must not be empty". So I am blocked and can't see a way to use field_delimiter_regex at all.

Why not just set the field_delimiter_regex to take precedence, i.e. ignoring the field_split_characters when a regex is configured?

[efloresb-tibco]

@bbgu1, I used substitute_string processor as a workaround and then the key_value, idk if that helps your scenario.

    - substitute_string:
          entries:
            - source: "log"
              from: \s(?=[a-z\\_\\-]+=)
              to: "\u0007"
    - key_value:
        source: "log"
        destination: "parsed-log"
        field_split_characters: "\u0007"
        delete_value_regex: "\""

[dlvenable]

PR to resolve this:

#4358