[DOC] Add drop processor #5767

vagimeli · 2023-12-01T21:17:27Z

Description

Adds drop processor documentation to close content gap

Issues Resolved

#proce

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli · 2023-12-01T21:19:19Z

Hi @macohen & @msfroh, Will you review, or direct me to the appropriate reviewer, for the ingest processor documentation? We're working through content gaps and need dev reviewers to vet technical accuracy. Thank you.

META issue for reference #4647

vagimeli · 2023-12-21T20:44:36Z

@hdhalter I've requested several tech reviews via OpenSearch community Slack channel and AWS OpenSearch Slack channel. No response to date. Who is the SDM for ingest processors for OpenSearch core? The Data Prepper team told me that this isn't their lane.

msfroh · 2023-12-21T20:59:48Z

I can take a look at this once I'm back home on Tuesday.

vagimeli · 2023-12-21T22:49:28Z

I can take a look at this once I'm back home on Tuesday.

@msfroh You've been been more than helpful with tech reviewing the other three ingest processors. We're trying to load balance the content gap reviews, and I don't want you to get caught exclusively on this and the remaining 9-10 ingest processors. Please don't prioritize this while you're OOO. Let us see if we can find someone to share the load. I'll keep you updated. Thank you again for all your help and reviews :)

_ingest-pipelines/processors/drop.md

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli · 2024-02-15T21:12:54Z

@gaobinlong Please review this processor documentation at your availability. Will you provide examples relevant to OpenSearch users? Thank you for your support!

gaobinlong · 2024-02-26T06:32:19Z

_ingest-pipelines/processors/drop.md

+The following is the syntax for the `drop` processor:
+
+```json
+{


{
"drop": {
"if": "ctx.foo == 'bar'"
}
}

gaobinlong · 2024-02-26T06:59:46Z

_ingest-pipelines/processors/drop.md

+{
+  "user_info": "Sensitive information including credit card"
+}
+```


The following response shows that the no operation happened, the document with id 1 was not indexed:

{ "_index": "testindex1", "_id": "1", "_version": -3, "result": "noop", "_shards": { "total": 0, "successful": 0, "failed": 0 } }

gaobinlong · 2024-02-26T07:02:08Z

_ingest-pipelines/processors/date.md

Should this file be removed?

Signed-off-by: Melissa Vagi <[email protected]>

Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

vagimeli · 2024-02-26T22:06:43Z

@gaobinlong Thank you for the examples. I've added them to the doc. Will you approve as the tech reviewer? Thank you.

gaobinlong · 2024-02-27T02:28:07Z

_ingest-pipelines/processors/date.md

Oh my mad, this file shouldn't be removed, I just saw this change was not related to the drop processor so I thought it shouldn't be in this PR, actually it was a small change for the date processor, so please add this file back, thanks!

This reverts commit 73296f5.

Signed-off-by: Melissa Vagi <[email protected]>

gaobinlong

Looks good to me.

natebower

@vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_ingest-pipelines/processors/drop.md

natebower · 2024-03-07T12:39:37Z

_ingest-pipelines/processors/drop.md

+
+Follow these steps to use the processor in a pipeline.
+
+**Step 1: Create a pipeline.**


Global: I believe that these do not take periods.

_ingest-pipelines/processors/drop.md

natebower · 2024-03-07T12:43:09Z

_ingest-pipelines/processors/drop.md

+```
+{% include copy-curl.html %}
+
+The following response confirms that the document with ID `1` was not indexed: 


"with the ID 1" or "with the ID of 1"?

natebower

LGTM with noted comments/changes.

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

* Add drop processor doc to address content gap Signed-off-by: Melissa Vagi <[email protected]> * Address tech review feedback Signed-off-by: Melissa Vagi <[email protected]> * Address tech review changes Signed-off-by: Melissa Vagi <[email protected]> * Delete _ingest-pipelines/processors/date.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Revert "Delete _ingest-pipelines/processors/date.md" This reverts commit 73296f5. * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update drop.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit b38ba75) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add drop processor doc to address content gap * Address tech review feedback * Address tech review changes * Delete _ingest-pipelines/processors/date.md * Revert "Delete _ingest-pipelines/processors/date.md" This reverts commit 73296f5. * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update drop.md --------- (cherry picked from commit b38ba75) Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nathan Bower <[email protected]>

* Add drop processor doc to address content gap Signed-off-by: Melissa Vagi <[email protected]> * Address tech review feedback Signed-off-by: Melissa Vagi <[email protected]> * Address tech review changes Signed-off-by: Melissa Vagi <[email protected]> * Delete _ingest-pipelines/processors/date.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Revert "Delete _ingest-pipelines/processors/date.md" This reverts commit 73296f5. * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update drop.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

Add drop processor doc to address content gap

20c5fc6

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli requested review from hdhalter, kolchfa-aws, Naarcha-AWS, ananzh, AMoo-Miki and natebower as code owners December 1, 2023 21:17

Merge branch 'main' into drop-processor

cc54f0b

vagimeli mentioned this pull request Dec 1, 2023

[META DOC] Ingest Processors Content Gaps #4647

Closed

45 tasks

vagimeli self-assigned this Dec 15, 2023

vagimeli added Needs SME Waiting on input from subject matter expert 3 - Tech review PR: Tech review in progress Content gap Blocked PR: Cannot move forward without assistance and removed Needs SME Waiting on input from subject matter expert labels Dec 15, 2023

msfroh reviewed Dec 29, 2023

View reviewed changes

_ingest-pipelines/processors/drop.md Outdated Show resolved Hide resolved

Address tech review feedback

6d0febd

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli added 4 - Doc review PR: Doc review in progress and removed Blocked PR: Cannot move forward without assistance 3 - Tech review PR: Tech review in progress labels Jan 10, 2024

vagimeli changed the title ~~[DOC] Add drop processor doc to address content gap~~ [DOC] Add drop processor Feb 15, 2024

gaobinlong reviewed Feb 26, 2024

View reviewed changes

Address tech review changes

35a2053

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli requested review from dlvenable and stephen-crawford as code owners February 26, 2024 21:29

Delete _ingest-pipelines/processors/date.md

73296f5

Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

gaobinlong reviewed Feb 27, 2024

View reviewed changes

vagimeli added 4 commits February 27, 2024 12:45

Revert "Delete _ingest-pipelines/processors/date.md"

b572c4d

This reverts commit 73296f5.

Merge branch 'main' into drop-processor

fb049e8

Signed-off-by: Melissa Vagi <[email protected]>

Merge branch 'main' into drop-processor

a3650f6

Merge branch 'main' into drop-processor

0efa50e

gaobinlong approved these changes Mar 1, 2024

View reviewed changes

natebower reviewed Mar 7, 2024

View reviewed changes

natebower approved these changes Mar 7, 2024

View reviewed changes

vagimeli and others added 6 commits March 7, 2024 09:36

Update _ingest-pipelines/processors/drop.md

b6f7902

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _ingest-pipelines/processors/drop.md

8ee3c95

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _ingest-pipelines/processors/drop.md

0046644

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _ingest-pipelines/processors/drop.md

0ac9c1d

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update _ingest-pipelines/processors/drop.md

8081f65

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

Update drop.md

4477666

Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

vagimeli added 3 - Done Issue is done/complete backport 2.12 PR: Backport label for 2.12 and removed 4 - Doc review PR: Doc review in progress labels Mar 7, 2024

Merge branch 'main' into drop-processor

e996085

vagimeli merged commit b38ba75 into main Mar 7, 2024
4 checks passed

vagimeli deleted the drop-processor branch March 7, 2024 16:45

opensearch-trigger-bot bot mentioned this pull request Mar 7, 2024

[Backport 2.12] [DOC] Add drop processor #6627

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOC] Add drop processor #5767

[DOC] Add drop processor #5767

vagimeli commented Dec 1, 2023

vagimeli commented Dec 1, 2023

vagimeli commented Dec 21, 2023 •

edited

Loading

msfroh commented Dec 21, 2023

vagimeli commented Dec 21, 2023

vagimeli commented Feb 15, 2024

gaobinlong Feb 26, 2024

gaobinlong Feb 26, 2024

gaobinlong Feb 26, 2024

vagimeli commented Feb 26, 2024

gaobinlong Feb 27, 2024

gaobinlong left a comment

natebower left a comment

natebower Mar 7, 2024

natebower Mar 7, 2024

natebower left a comment


		Follow these steps to use the processor in a pipeline.

		Step 1: Create a pipeline.

[DOC] Add drop processor #5767

[DOC] Add drop processor #5767

Conversation

vagimeli commented Dec 1, 2023

Description

Issues Resolved

Checklist

vagimeli commented Dec 1, 2023

vagimeli commented Dec 21, 2023 • edited Loading

msfroh commented Dec 21, 2023

vagimeli commented Dec 21, 2023

vagimeli commented Feb 15, 2024

gaobinlong Feb 26, 2024

Choose a reason for hiding this comment

gaobinlong Feb 26, 2024

Choose a reason for hiding this comment

gaobinlong Feb 26, 2024

Choose a reason for hiding this comment

vagimeli commented Feb 26, 2024

gaobinlong Feb 27, 2024

Choose a reason for hiding this comment

gaobinlong left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

natebower Mar 7, 2024

Choose a reason for hiding this comment

natebower Mar 7, 2024

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

vagimeli commented Dec 21, 2023 •

edited

Loading