Allow creation of multiple tags/fields with regex processor

[sslupsky]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Allow creation of multiple tags/fields when using the regex processor

Current behavior:

When using named subgroups, it is possible to create a tag/field using the result_key parameter. Presently, if you want to create multiple tag/fields from multiple subgroups, you need to create (instantiate?) another processor. For example, in the following configuration a regex is used to extract the gateway, uid and channel from the path. In order to create a new tag, three processors are required:

[[processors.regex.tags]]
  key = "path"
  pattern = "^/motescan/(?P<gateway>scannet[0-9]+)/csv/l(?P<uid>[0-9A-Fa-f]+)[0-9]+\.csv_channel(?P<channel>[A-Za-z]?[A-Za-z0-9]+)\.csv$"
  replacement = "${gateway}"
  result_key = "gateway"

[[processors.regex.tags]]
  key = "path"
  pattern = "^/motescan/(?P<gateway>scannet[0-9]+)/csv/l(?P<uid>[0-9A-Fa-f]+)[0-9]+\.csv_channel(?P<channel>[A-Za-z]?[A-Za-z0-9]+)\.csv$"
  replacement = "${uid}"
  result_key = "uid"

[[processors.regex.tags]]
  key = "path"
  pattern = "^/motescan/(?P<gateway>scannet[0-9]+)/csv/l(?P<uid>[0-9A-Fa-f]+)[0-9]+\.csv_channel(?P<channel>[A-Za-z]?[A-Za-z0-9]+)\.csv$"
  replacement = "${channel}"
  result_key = "channel"

Desired behavior:

It would improve resource usage if only one processor was used as the regex would be executed once instead of multiple times.

Use case: [Why is this important (helps with prioritizing requests)]

Using a regex to create/extract metadata from another tag/field.

[hackery]

I was just looking to do this too. Would the following syntax be suitable?

replacements = { gateway = "${gateway}", uid = "${uid}", channel = "${channel}" }

[reimda]

@sslupsky and @hackery it looks like the next step for this is to find someone to implement the change. Are either of you able to put together a PR to change the regex processor to allow multiple regex groups to produce multiple tags/fields?

[srebhan]

@sslupsky please test the binary in PR #13971 which implements the possibility of using named groups to create a bunch of new tags/fields using the group names and values. Let me know if this is what you had in mind...