[pkg/ottl] Add function for parsing key value pairs

[dpaasman00]

Component(s)

pkg/ottl, processor/transform

Is your feature request related to a problem? Please describe.

I'd like to be able to parse key value pairs using OTTL similar to the Stanza key value parser operator, but currently there is no function in OTTL that achieves this.

Describe the solution you'd like

Add a new converter function that returns a pcommon.Map of the key value pairs parsed from a target string.

ParseKeyValuePairs(target, Optional[delimiter], Optional[pair_delimiter])

The function parameters would be as follows:

• target: StringGetter Returns a string containing key value pairs to be parsed from such as "pkg=ottl func=keyvalue"
• delimiter: Optional[string] A string containing the delimiter value to use to split the key value pair, such as "=". Default can be =.
• pair_delimiter: Optional[string]A string containing the delimiter value to use to split pairs in the target string. Default can be white space.

Implementation can follow the Stanza package key value parser operator and use strings.FieldsFunc() when white space is the pair_delimiter, strings.Split() for non white space pair_delimiters, and strings.SplitN() for splitting the key and value in a given pair.

Describe alternatives you've considered

A possible alternative is implementing this function as an editor function. This option would allow users to immediately overwrite the source of the key value pairs with the newly parsed version. However, there is a precedent already set with the ParseJSON converter function to just return the parsed value. While it may be useful for this function to also replace the source string, I don't think it's necessary for it to do so and just returning a map of the pairs is enough.

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dpaasman00]

I'd like to implement these functions once design is agreed on.

[TylerHelmuth]

@dpaasman00 can you share an example payload you'd be parsing?

[dpaasman00]

Sure, here's an example payload where the delimiter is = and the pair delimiter would be ;:

CMD=0; VER=10; MSGType= RECONNECT;              LEN=15; ERR=0; FLG=0 SPID=3040; SCH=112; SPH=10709; SM=0 RPID=2; RCH=0; RPH=0; RM=0

After parsing, the function would output these values in a map like:

{"CMD": "0", "VER": "10", "MSGType": "RECONNECT", "LEN": "15", ...}

Some other examples and potential use cases can be seen in the test cases for the Stanza Parser here.

Let me know if I can expand on any of this at all!

[TylerHelmuth]

@dpaasman00 are these values in the log body or somewhere else? I assume the data is coming from a receiver that is not the filelogreceiver?

[dpaasman00]

@TylerHelmuth Entirely depends on the situation. Typically we'd expect the raw string to be on the log body, but it could be on the attributes as well. In terms of receivers, the data can come from the filelogreceiver(especially in the case the pairs are located in the body) but this is also known to occur with the TCP, Splunk TCP, and Fluent Forward receivers.

Something we've seen in customer environments occasionally is cases where they are using very low powered machines and want to do as much parsing as possible at the gateway/aggregator level, where they have more compute power. So we'd like to support parsing key-values even when we are receiving data from receivers like the OTLP receiver.

[TylerHelmuth]

I ask the question bc I want to make sure the use case exists somewhere that Stanza is not used. The TCP receiver can use stanza operators and therefore you wouldn't need the transformprocessor, but data from the fluentforward and OTLP receivers would need the transformprocessor.

[TylerHelmuth]

I agree with the function parameters. Could we shorten the name to be ParseKeyValue?

@evan-bradley please take a look at this proposal.

[dpaasman00]

Totally understand and no problem changing the name.

[evan-bradley]

This all sounds good to me. I'm fine with having the name as ParseKeyValue so it lines up with the name of the Stanza operator.