[META DOC] Ingest Processors Content Gaps

[vagimeli]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
Close Ingest API content gaps as follows:

Ingest pipelines

• Create, get, delete, test pipeline [DOC] Add ingest processors documentation #4299 and [DOC] Update ingest pipelines content and page navigation #4285
• Example tutorial
• Accessing data in pipelines
• Conditional execution in pipelines (nested field conditionals, complex conditionals, conditionals with pipeline processor, conditionals with regular expressions) (see [DOC] Add conditional execution in pipelines #7348)
• Enrich data (set up enrich processor, add tutorials (example geolocation, example exact values, example value to a range), enrich policy definition) Not a processor in OpenSearch
• Handling pipeline failures https://github.com/opensearch-project/documentation-website/pull/4299/files

Ingest processors

• index.md [DOC] Add ingest processors documentation #4299
• append (10) [DOC] Add ingest processors documentation #4299
• bytes (20) [DOC] Add ingest processors documentation #4299
• convert (30) [DOC] Add ingest processors documentation #4299
• csv (40) [DOC] Add ingest processors documentation #4299
• date (50) [DOC] Add ingest processors documentation #4299
• date_index_name (60) [DOC] Add new documentation for date index name processor #5398
• dissect (70) [DOC] Add dissect processor documentation #5510
• dot_extender (80) [DOC]Add new dot expander processor doc #5631
• drop (90) [DOC] Add drop processor #5767
• enrich (CONFIRMED THIS IS NOT IN OPENSEARCH - Is this in the code? It's an XPACK feature. Doesn't appear in my list of processors when I run GET /_nodes/ingest?filter_path=nodes.*.ingest.processors. See also comments in the forum https://forum.opensearch.org/t/ingest-pipelines-support/7716/3.)
• fail (100) [DOC] Add fail processor documentation #5980
• foreach (110) Add foreach processor documentation #5981
• grok (120) [DOC] Add new doc for grok processor #5019
• gsub (130) [DOC] Add gsub processor documentation #5983
• html_strip (140) [DOC] Add html strip processor documentation #5984
• IP2Geo (150) Add new documentation IP2Geo processor automatic updating feature #4095
• join (160) https://github.com/opensearch-project/documentation-website/pull/5985/files
• json (170) Add JSON processor documentation #5982
• kv (180) [DOC] Add kv processor content gap doc #5781
• lowercase (190)
• pipeline (200) [DOC] Add pipeline processor documentation #5986
• rename (210) [DOC] Add rename processor documentation #5987
• remove (220)
• script (230) [DOC] Add script processor documentation #5988
• set (240) [DOC] Add set processor documentation #5989
• sort (250) [DOC] Add sort processor documentation #5990
• sparse_encoder (260)
• split (270) [DOC] Add split processor documentation #5991
• text_embedding (280)
• text_image (290)
• trim (300) [DOC] Add trim processor documentation #5993
• uppercase (310)
• urldecode (320) [DOC] Add urldecode processor documentation #5994
• user_agent (330) [DOC] Add user agent processor documentation #5995

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

[vagimeli]

@heemin32 The doc team is working through Ingest APIs content gaps. This issue is tracking those gaps. At your availability, will you review and provide your feedback on anything we've missed? Thank you.

[heemin32]

1. Enrich data (set up enrich processor, add tutorials (example geolocation, example exact values, example value to a range), enrich policy definition) is not supported in OpenSearch.

2. foreach processor is missing.

[hdhalter]

@vagimeli - Should we keep this open until all ingest processors are documented, or did you open a new issue for tracking?

[vagimeli]

@heemin32 Is the enrich processor in the code? It's an XPACK feature. Doesn't appear in my list of processors when I run GET /_nodes/ingest?filter_path=nodes.*.ingest.processors. See also comments in the forum https://forum.opensearch.org/t/ingest-pipelines-support/7716/3

[heemin32]

@heemin32 Is the enrich processor in the code? It's an XPACK feature. Doesn't appear in my list of processors when I run GET /_nodes/ingest?filter_path=nodes.*.ingest.processors. See also comments in the forum https://forum.opensearch.org/t/ingest-pipelines-support/7716/3

OpenSearch does not have enrich processor

[vagimeli]

@vamshin This is the primary PR for tracking the ingest processor content gaps. As I write the docs, I'll track each processor's PR in this primary PR. I'll tag you in those PRs so that you have visibility into the needed technical reviews by your team. Thanks for helping move these docs forward.

[hdhalter]

Hi @vagimeli , we will need documentation for the kv processor for a 2.12 feature. Can you please prioritize on that one next? Thanks! See #5651

[hdhalter]

@vagimeli - did you mean to close this? We should keep it open until all are complete.

[dblock]

[Triage -- attendees 1, 2, 3, 4, 5, 6, 7]