-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add neural search default processor for non OpenAI/Cohere scenario #1274
Merged
zane-neo
merged 15 commits into
opensearch-project:main
from
zane-neo:neural-search-default-processor
Sep 26, 2023
Merged
Add neural search default processor for non OpenAI/Cohere scenario #1274
zane-neo
merged 15 commits into
opensearch-project:main
from
zane-neo:neural-search-default-processor
Sep 26, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zane-neo
requested review from
b4sjoo,
dhrubo-os,
jngz-es,
model-collapse,
rbhavna,
wujunshen,
ylwu-amzn and
Zhangxunmt
as code owners
September 1, 2023 04:19
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:19 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:19 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:19 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:19 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:19 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:19 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:45 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:45 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:45 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:45 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:45 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:45 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:56 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:56 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:56 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:56 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:56 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 04:56 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 06:48 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 06:48 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 06:48 — with
GitHub Actions
Failure
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 1, 2023 06:48 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 25, 2023 00:26 — with
GitHub Actions
Error
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 25, 2023 00:26 — with
GitHub Actions
Error
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:26 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:26 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:26 — with
GitHub Actions
Inactive
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 25, 2023 00:47 — with
GitHub Actions
Error
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:47 — with
GitHub Actions
Inactive
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 25, 2023 00:47 — with
GitHub Actions
Failure
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:47 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:47 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 00:47 — with
GitHub Actions
Inactive
dhrubo-os
approved these changes
Sep 25, 2023
4 tasks
zane-neo
had a problem deploying
to
ml-commons-cicd-env
September 25, 2023 19:36 — with
GitHub Actions
Error
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 19:36 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 19:36 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 19:36 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 19:36 — with
GitHub Actions
Inactive
zane-neo
temporarily deployed
to
ml-commons-cicd-env
September 25, 2023 19:36 — with
GitHub Actions
Inactive
ylwu-amzn
approved these changes
Sep 26, 2023
zane-neo
added a commit
that referenced
this pull request
Sep 26, 2023
* Add Auto Release Workflow (#1306) * Add Auto Release Workflow Signed-off-by: Sicheng Song <[email protected]> * Fix release note address Signed-off-by: Sicheng Song <[email protected]> --------- Signed-off-by: Sicheng Song <[email protected]> * Bump aws-encryption-sdk-java to fix CVE-2023-33201 (#1309) Signed-off-by: Sicheng Song <[email protected]> * Add release note for 2.10.0 release (#1312) * Add release note for 2.10.0 Signed-off-by: Sicheng Song <[email protected]> * Add CVE fix Signed-off-by: Sicheng Song <[email protected]> --------- Signed-off-by: Sicheng Song <[email protected]> * fixing doc link (#1318) * fixing doc link Signed-off-by: Dhrubo Saha <[email protected]> * fixing indentation Signed-off-by: Dhrubo Saha <[email protected]> --------- Signed-off-by: Dhrubo Saha <[email protected]> * Fix unassigned ml system shard replicas (#1315) (#1324) * Fix unassigned ml system shard replicas * Adjust auto replica settings to keep it consistent with AOS default setting * Update plugin/src/main/java/org/opensearch/ml/indices/MLIndicesHandler.java * Modify exception handling * Modify exception messages * Add response check * Add response check and exception handling * Keep error message consistent * Keep error message consistent * Keep error message consistent --------- Signed-off-by: Sicheng Song <[email protected]> Co-authored-by: Yaliang Wu <[email protected]> * Adjust index replicas settings to keep consistent with AOS 2.9 (#1325) Signed-off-by: Sicheng Song <[email protected]> * Make 2.10 release notes up to date (#1345) Signed-off-by: Sicheng Song <[email protected]> * fix spelling (#1363) Signed-off-by: Kalyan <[email protected]> * Add neural search default processor for non OpenAI/Cohere scenario (#1274) * Add neural search default pre/post process function support Signed-off-by: zane-neo <[email protected]> * Fix UT failures Signed-off-by: zane-neo <[email protected]> * Address PR comment to remove nonJson response case Signed-off-by: zane-neo <[email protected]> * Fix low code coverage issue Signed-off-by: zane-neo <[email protected]> * fix format issue Signed-off-by: zane-neo <[email protected]> * Try to fix classNotFound issue in IT Signed-off-by: zane-neo <[email protected]> * revert Try to fix classNotFound issue in IT Signed-off-by: zane-neo <[email protected]> * Change gson dependency to compileOnly Signed-off-by: zane-neo <[email protected]> * Change default pre/post process function name Signed-off-by: zane-neo <[email protected]> * Address code review comments Signed-off-by: zane-neo <[email protected]> * Make preprocess function to default Signed-off-by: zane-neo <[email protected]> * Remove GsonUtil since there already a single instance in StringUtils Signed-off-by: zane-neo <[email protected]> * Fix UT failures Signed-off-by: zane-neo <[email protected]> * Address comments Signed-off-by: zane-neo <[email protected]> * use import instead of fully qualified name Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: Sicheng Song <[email protected]> Signed-off-by: Dhrubo Saha <[email protected]> Signed-off-by: Kalyan <[email protected]> Signed-off-by: zane-neo <[email protected]> Co-authored-by: Sicheng Song <[email protected]> Co-authored-by: Dhrubo Saha <[email protected]> Co-authored-by: Yaliang Wu <[email protected]> Co-authored-by: Kalyan <[email protected]>
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1274-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b48546ae89a51f303ef23ac8d605e3e343b67b34
# Push it to GitHub
git push --set-upstream origin backport/backport-1274-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x Then, create a pull request where the |
5 tasks
1 task
ylwu-amzn
pushed a commit
to ylwu-amzn/ml-commons
that referenced
this pull request
Oct 4, 2023
…pensearch-project#1274) * Fix breaking change caused by opensearch core Signed-off-by: zane-neo <[email protected]> * Add neural search default pre/post process function support Signed-off-by: zane-neo <[email protected]> * Fix UT failures Signed-off-by: zane-neo <[email protected]> * Fix conflicts when backport Signed-off-by: zane-neo <[email protected]> * Fix conflict when backport Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: zane-neo <[email protected]>
5 tasks
ylwu-amzn
pushed a commit
that referenced
this pull request
Oct 4, 2023
…1274) * Fix breaking change caused by opensearch core Signed-off-by: zane-neo <[email protected]> * Add neural search default pre/post process function support Signed-off-by: zane-neo <[email protected]> * Fix UT failures Signed-off-by: zane-neo <[email protected]> * Fix conflicts when backport Signed-off-by: zane-neo <[email protected]> * Fix conflict when backport Signed-off-by: zane-neo <[email protected]> --------- Signed-off-by: zane-neo <[email protected]>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
ml-commons has two default pre/post process function which are for OpenAI and Cohere and written in painless script. There's no default pre/post process function for neural search plugin text embedding case, if user want to use neural search with remote model to text embedding, user has to write complex painless script which is a heavy burden. This PR do two things:
connector.pre_process.neural_search.text_embedding
orconnector.post_process.neural_search.text_embedding
, which is pretty easy for user to use neural search text embedding.Issues Resolved
#1304
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.