Add list to map processor #3806

Naarcha-AWS · 2023-04-18T21:32:19Z

Fixes #3755

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS · 2023-04-19T15:51:25Z

@dlvenable and @oeyh: Would y'all mind taking a look at this one?

oeyh

Looks great. Just a few small things:

_data-prepper/pipelines/configuration/processors/list-to-map.md

_data-prepper/pipelines/configuration/processors/mutate-event.md

Co-authored-by: Hai Yan <[email protected]>

cwillum

Looks good with suggestions.

_data-prepper/pipelines/configuration/processors/add-entries.md

_data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Chris Moore <[email protected]>

Signed-off-by: Naarcha-AWS <[email protected]>

_data-prepper/pipelines/configuration/processors/add-entries.md

dlvenable · 2023-04-20T13:47:42Z

_data-prepper/pipelines/configuration/processors/list-to-map.md

@@ -0,0 +1,305 @@
+---
+layout: default
+title: List to map processor


Also, per #3666 I think we should use the actual processor name.

dlvenable · 2023-04-20T13:47:58Z

_data-prepper/pipelines/configuration/processors/list-to-map.md

+
+# list_to_map
+
+The `list-to-map` processor converts a list of objects from an event, where each object contains a `key` field, into a map of target keys.


list_to_map -> Use hyphens.

Sorry, I mean, you should use underscores instead of hyphens.

natebower

@Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

natebower · 2023-04-20T13:49:20Z

_data-prepper/pipelines/configuration/processors/list-to-map.md

@@ -0,0 +1,305 @@
+---
+layout: default
+title: List to map processor


It looks like we use "List to map", "list_to_map", and "list-to-map". Please ensure consistent usage globally.

We have a discussion about the title standard going in #3666. Fixed the consistency on the page, but leaving the title the same for now to address once we come to a decision on the title convention.

_data-prepper/pipelines/configuration/processors/list-to-map.md

natebower · 2023-04-20T13:49:55Z

_data-prepper/pipelines/configuration/processors/list-to-map.md

+
+## Configuration
+
+The following table describes the configuration options used to generate target keys for the mappings:


Don't introduce a table with a colon.

_data-prepper/pipelines/configuration/processors/list-to-map.md

natebower · 2023-04-20T14:15:44Z

_data-prepper/pipelines/configuration/processors/list-to-map.md

+
+### Example: No `value_key` specified
+
+The follow example `pipeline.yaml` shows the `list_to_map` processor with no `value_key` specified. Since `key` is set to the objects `name`, the processor extracts the names of each object to use as keys in the map. 


Use since only to describe time events. Don’t use in place of because.

_data-prepper/pipelines/configuration/processors/list-to-map.md

dlvenable · 2023-04-20T15:13:39Z

Also, why does this PR have a lot of other changes beyond list_to_map?

natebower · 2023-04-20T15:20:14Z

Also, why does this PR have a lot of other changes beyond list_to_map?

If you mean my changes, then because it went through editorial review.

_data-prepper/pipelines/configuration/processors/list-to-map.md

Co-authored-by: Nathan Bower <[email protected]>

Naarcha-AWS · 2023-04-20T17:47:11Z

Also, why does this PR have a lot of other changes beyond list_to_map?

@dlvenable: I made additional changes to make the taxonomy cleaner, including:

Change nav_order so processors appear in alpha order.
Remove redundant content for Mutate event processors, where mutate event content would appear both on its single page AND the Mutate event page.
Fixed the titles for consistency, though we'll likely readdress them depending on the outcome of [DOC] Data Prepper plugin naming consistency #3666

* Add list to map processor. Signed-off-by: Naarcha-AWS <[email protected]> * Tweak one last file Signed-off-by: Naarcha-AWS <[email protected]> * Fix typo Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <[email protected]> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit e1a1f44) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add list to map processor. * Tweak one last file * Fix typo * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback * A couple more wording tweaks * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Apply suggestions from code review --------- (cherry picked from commit e1a1f44) Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

* Add list to map processor. Signed-off-by: Naarcha-AWS <[email protected]> * Tweak one last file Signed-off-by: Naarcha-AWS <[email protected]> * Fix typo Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <[email protected]> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

This reverts commit 02c8c15.

* Add list to map processor. Signed-off-by: Naarcha-AWS <[email protected]> * Tweak one last file Signed-off-by: Naarcha-AWS <[email protected]> * Fix typo Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <[email protected]> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

This reverts commit 5b20e59.

* Add list to map processor. Signed-off-by: Naarcha-AWS <[email protected]> * Tweak one last file Signed-off-by: Naarcha-AWS <[email protected]> * Fix typo Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Hai Yan <[email protected]> * Update mutate-event.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Chris Moore <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Add Chris' feedback Signed-off-by: Naarcha-AWS <[email protected]> * A couple more wording tweaks Signed-off-by: Naarcha-AWS <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Update _data-prepper/pipelines/configuration/processors/list-to-map.md Co-authored-by: Nathan Bower <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> --------- Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Hai Yan <[email protected]> Co-authored-by: Chris Moore <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

Naarcha-AWS added 2 commits April 18, 2023 16:29

Add list to map processor.

2c6aaad

Signed-off-by: Naarcha-AWS <[email protected]>

Tweak one last file

a7443aa

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS added the 3 - Tech review PR: Tech review in progress label Apr 18, 2023

Fix typo

4b94f14

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS added the data-prepper label Apr 19, 2023

oeyh reviewed Apr 19, 2023

View reviewed changes

Naarcha-AWS and others added 5 commits April 19, 2023 13:04

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

b6196cb

Co-authored-by: Hai Yan <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

115958e

Co-authored-by: Hai Yan <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

8fa5402

Co-authored-by: Hai Yan <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

9100718

Co-authored-by: Hai Yan <[email protected]>

Update mutate-event.md

bd672f2

Naarcha-AWS added 4 - Doc review PR: Doc review in progress and removed 3 - Tech review PR: Tech review in progress labels Apr 19, 2023

cwillum approved these changes Apr 19, 2023

View reviewed changes

Naarcha-AWS and others added 6 commits April 19, 2023 16:06

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

68d08dc

Co-authored-by: Chris Moore <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

aaf996a

Co-authored-by: Chris Moore <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

5242b1e

Co-authored-by: Chris Moore <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

05faaa5

Add Chris' feedback

968e4a1

Signed-off-by: Naarcha-AWS <[email protected]>

A couple more wording tweaks

29feb2d

Signed-off-by: Naarcha-AWS <[email protected]>

ariamarble approved these changes Apr 20, 2023

View reviewed changes

Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Apr 20, 2023

dlvenable requested changes Apr 20, 2023

View reviewed changes

natebower reviewed Apr 20, 2023

View reviewed changes

natebower removed the 5 - Editorial review PR: Editorial review in progress label Apr 20, 2023

Naarcha-AWS commented Apr 20, 2023

View reviewed changes

_data-prepper/pipelines/configuration/processors/list-to-map.md Outdated Show resolved Hide resolved

Naarcha-AWS and others added 4 commits April 20, 2023 12:39

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

893ed80

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

c873a45

Co-authored-by: Nathan Bower <[email protected]>

Update _data-prepper/pipelines/configuration/processors/list-to-map.md

1959861

Co-authored-by: Nathan Bower <[email protected]>

Apply suggestions from code review

e6e2076

Co-authored-by: Nathan Bower <[email protected]>

oeyh approved these changes Apr 20, 2023

View reviewed changes

Naarcha-AWS merged commit e1a1f44 into main Apr 20, 2023

Naarcha-AWS deleted the list-to-map-processor branch April 20, 2023 18:48

Naarcha-AWS added backport 2.5 PR: Backport label for 2.5 backport 2.6 PR: Backport label for 2.6 labels Apr 20, 2023

opensearch-trigger-bot bot mentioned this pull request Apr 20, 2023

[Backport 2.6] Add list to map processor #3829

Merged

opensearch-trigger-bot bot mentioned this pull request Apr 20, 2023

[Backport 2.5] Add list to map processor #3830

Merged

vagimeli added a commit that referenced this pull request Apr 25, 2023

Revert "Add list to map processor (#3806)"

89ce678

This reverts commit 02c8c15.

vagimeli added a commit that referenced this pull request Apr 25, 2023

Revert "Add list to map processor (#3806)"

6b7dc02

This reverts commit 02c8c15.

vagimeli added a commit that referenced this pull request May 4, 2023

Revert "Add list to map processor (#3806)"

41ec6b7

This reverts commit 5b20e59.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add list to map processor #3806

Add list to map processor #3806

Naarcha-AWS commented Apr 18, 2023

Naarcha-AWS commented Apr 19, 2023

oeyh left a comment

cwillum left a comment

dlvenable Apr 20, 2023

dlvenable Apr 20, 2023

dlvenable Apr 20, 2023

natebower left a comment

natebower Apr 20, 2023

Naarcha-AWS Apr 20, 2023

natebower Apr 20, 2023

natebower Apr 20, 2023

dlvenable commented Apr 20, 2023

natebower commented Apr 20, 2023

Naarcha-AWS commented Apr 20, 2023


		# list_to_map

		The `list-to-map` processor converts a list of objects from an event, where each object contains a `key` field, into a map of target keys.


		## Configuration

		The following table describes the configuration options used to generate target keys for the mappings:


		### Example: No `value_key` specified

		The follow example `pipeline.yaml` shows the `list_to_map` processor with no `value_key` specified. Since `key` is set to the objects `name`, the processor extracts the names of each object to use as keys in the map.

Add list to map processor #3806

Add list to map processor #3806

Conversation

Naarcha-AWS commented Apr 18, 2023

Checklist

Naarcha-AWS commented Apr 19, 2023

oeyh left a comment

Choose a reason for hiding this comment

cwillum left a comment

Choose a reason for hiding this comment

dlvenable Apr 20, 2023

Choose a reason for hiding this comment

dlvenable Apr 20, 2023

Choose a reason for hiding this comment

dlvenable Apr 20, 2023

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

natebower Apr 20, 2023

Choose a reason for hiding this comment

Naarcha-AWS Apr 20, 2023

Choose a reason for hiding this comment

natebower Apr 20, 2023

Choose a reason for hiding this comment

natebower Apr 20, 2023

Choose a reason for hiding this comment

dlvenable commented Apr 20, 2023

natebower commented Apr 20, 2023

Naarcha-AWS commented Apr 20, 2023