Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding create search pipeline step #569

Merged
merged 2 commits into from
Mar 13, 2024

Conversation

amitgalitz
Copy link
Member

Description

Added create search pipeline step by creating an abstract create pipeline step since both have same required in puts and almost exact same logic:

example:

{
                    "id": "create_search_pipeline",
                    "type": "create_search_pipeline",
                    "previous_node_inputs": {
                        "deploy_openai_model": "model_id"
                    },
                    "user_inputs": {
                        "pipeline_id": "rag_pipeline",
                        "configurations": {
                            "response_processors": [
                                {
                                    "retrieval_augmented_generation": {
                                        "tag": "openai_pipeline_demo",
                                        "description": "Demo pipeline Using OpenAI Connector",
                                        "model_id":"${{deploy_openai_model.model_id}}",
                                        "context_field_list": [
                                            "text"
                                        ],
                                        "system_prompt": "You are a helpful assistant",
                                        "user_instructions": "Generate a concise and informative answer in less than 100 words for the given question"
                                    }
                                }
                            ]
                        }
                    }
                }

Issues Resolved

resolves #545

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added the backport 2.x backport PRs to 2.x branch label Mar 12, 2024
@amitgalitz amitgalitz force-pushed the search-pipeline-step branch from 7aaf859 to 9f19dd0 Compare March 12, 2024 21:58
Copy link
Member

@dbwiddis dbwiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with some nits.

CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Member

@joshpalis joshpalis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending fixes for the nits @dbwiddis mentioned

@amitgalitz amitgalitz changed the title adding create search pipeline step Adding create search pipeline step Mar 12, 2024
Copy link
Member

@owaiskazi19 owaiskazi19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a minor comment

Signed-off-by: Amit Galitzky <[email protected]>
@amitgalitz amitgalitz force-pushed the search-pipeline-step branch from 4d06699 to d3a3708 Compare March 12, 2024 23:41
Copy link

codecov bot commented Mar 12, 2024

Codecov Report

Attention: Patch coverage is 84.12698% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 72.48%. Comparing base (7a2d9d9) to head (d3a3708).

Files Patch % Lines
...framework/workflow/AbstractCreatePipelineStep.java 80.76% 10 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #569      +/-   ##
============================================
+ Coverage     72.24%   72.48%   +0.24%     
- Complexity      653      657       +4     
============================================
  Files            78       80       +2     
  Lines          3397     3416      +19     
  Branches        269      270       +1     
============================================
+ Hits           2454     2476      +22     
+ Misses          825      822       -3     
  Partials        118      118              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@owaiskazi19
Copy link
Member

This change would also require an update to the documentation because we are introducing new param configuration here.

@amitgalitz
Copy link
Member Author

This change would also require an update to the documentation because we are introducing new param configuration here.

Good call out, and new steps

@amitgalitz amitgalitz merged commit bc2306b into opensearch-project:main Mar 13, 2024
32 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/flow-framework/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/flow-framework/backport-2.x
# Create a new branch
git switch --create backport/backport-569-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bc2306b451e85f91a62952f16fcf23c2e9a8733c
# Push it to GitHub
git push --set-upstream origin backport/backport-569-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/flow-framework/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-569-to-2.x.

amitgalitz added a commit to amitgalitz/opensearch-ai-flow-framework that referenced this pull request Mar 13, 2024
* adding create search pipeline step

Signed-off-by: Amit Galitzky <[email protected]>

* addresing nit changes

Signed-off-by: Amit Galitzky <[email protected]>

---------

Signed-off-by: Amit Galitzky <[email protected]>
@amitgalitz amitgalitz mentioned this pull request Mar 13, 2024
joshpalis pushed a commit that referenced this pull request Mar 13, 2024
Adding create search pipeline step (#569)

* adding create search pipeline step



* addresing nit changes



---------

Signed-off-by: Amit Galitzky <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x backport PRs to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Add Create Search Pipeline Step
4 participants