[Backport 2.x] Supporting sparse semantic retrieval in neural search #343
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![opensearch-trigger-bot[bot]](https://avatars.githubusercontent.com/u/80134844?s=60&v=4){kind=small}
* sparse mapper field and query builder Signed-off-by: zhichao-aws <[email protected]> * fix typo Signed-off-by: zhichao-aws <[email protected]> * Add map result support in neural search for non text embedding models Signed-off-by: zane-neo <[email protected]> * Fix compilation failure issue Signed-off-by: zane-neo <[email protected]> * Add more UTs Signed-off-by: zane-neo <[email protected]> * add sparse encoding processor Signed-off-by: xinyual <[email protected]> * add sparse encoding processor Signed-off-by: xinyual <[email protected]> * remove guava in gradle Signed-off-by: xinyual <[email protected]> * modify access control Signed-off-by: xinyual <[email protected]> * Add map result support in neural search for non text embedding models Signed-off-by: zane-neo <[email protected]> * Fix compilation failure issue Signed-off-by: zane-neo <[email protected]> * change output logic Signed-off-by: xinyual <[email protected]> * create abstract Signed-off-by: xinyual <[email protected]> * create abstract proccesor Signed-off-by: xinyual <[email protected]> * add abstract class Signed-off-by: xinyual <[email protected]> * remove duplicate code Signed-off-by: xinyual <[email protected]> * remove duplicate code Signed-off-by: xinyual <[email protected]> * remove dl process Signed-off-by: xinyual <[email protected]> * move static to abstract class Signed-off-by: xinyual <[email protected]> * update query rewrite logic Signed-off-by: zhichao-aws <[email protected]> * modify header Signed-off-by: zhichao-aws <[email protected]> * merge conflict Signed-off-by: xinyual <[email protected]> * delete index mapper, change to rank_features Signed-off-by: zhichao-aws <[email protected]> * remove unused import Signed-off-by: zhichao-aws <[email protected]> * list return result Signed-off-by: zhichao-aws <[email protected]> * refactor type and listTypeNestedMapKey, tidy Signed-off-by: zhichao-aws <[email protected]> * forbid nested input. tidy. Signed-off-by: zhichao-aws <[email protected]> * tidy Signed-off-by: zhichao-aws <[email protected]> * enable nested Signed-off-by: zhichao-aws <[email protected]> * fix test Signed-off-by: zhichao-aws <[email protected]> * Add ut it to sparse encoding processor (#6) * fix original UT problem Signed-off-by: xinyual <[email protected]> * add UT IT Signed-off-by: xinyual <[email protected]> * add more UT Signed-off-by: xinyual <[email protected]> * add more ut Signed-off-by: xinyual <[email protected]> * fix typo error Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]> * utils, tidy Signed-off-by: zhichao-aws <[email protected]> * rename to sparse_encoding query Signed-off-by: zhichao-aws <[email protected]> * add validation and ut Signed-off-by: zhichao-aws <[email protected]> * sparse encoding query builder ut Signed-off-by: zhichao-aws <[email protected]> * rename Signed-off-by: zhichao-aws <[email protected]> * UT for utils Signed-off-by: zhichao-aws <[email protected]> * enrich sparse encoding IT mappings Signed-off-by: zhichao-aws <[email protected]> * add it Signed-off-by: zhichao-aws <[email protected]> * add it Signed-off-by: zhichao-aws <[email protected]> * add integ test Signed-off-by: zhichao-aws <[email protected]> * rename resource file Signed-off-by: zhichao-aws <[email protected]> * tidy Signed-off-by: zhichao-aws <[email protected]> * remove BoundedLinearQuery and TokenScoreUpperBound Signed-off-by: zhichao-aws <[email protected]> * tidy Signed-off-by: zhichao-aws <[email protected]> * add delta to loose the equal Signed-off-by: zhichao-aws <[email protected]> * move SparseEncodingQueryBuilder to upper level path Signed-off-by: zhichao-aws <[email protected]> * tidy Signed-off-by: zhichao-aws <[email protected]> * add it Signed-off-by: zhichao-aws <[email protected]> * Update src/main/java/org/opensearch/neuralsearch/ml/MLCommonsClientAccessor.java Co-authored-by: zane-neo <[email protected]> Signed-off-by: zhichao-aws <[email protected]> * Update src/main/java/org/opensearch/neuralsearch/util/TokenWeightUtil.java Co-authored-by: zane-neo <[email protected]> Signed-off-by: zhichao-aws <[email protected]> * restore gradle.propeties Signed-off-by: zhichao-aws <[email protected]> * add release notes Signed-off-by: zhichao-aws <[email protected]> * change field modifier to private for NLPProcessor Signed-off-by: zhichao-aws <[email protected]> * add comments Signed-off-by: zhichao-aws <[email protected]> * use StringUtils to check Signed-off-by: zhichao-aws <[email protected]> * null check Signed-off-by: zhichao-aws <[email protected]> * modify changelog Signed-off-by: zhichao-aws <[email protected]> * nit Signed-off-by: zhichao-aws <[email protected]> * nit Signed-off-by: zhichao-aws <[email protected]> * remove query tokens from user interface Signed-off-by: zhichao-aws <[email protected]> * fix test Signed-off-by: zhichao-aws <[email protected]> * tidy Signed-off-by: zhichao-aws <[email protected]> * update function name Signed-off-by: zhichao-aws <[email protected]> * add javadoc Signed-off-by: zhichao-aws <[email protected]> * remove debug log including inference result Signed-off-by: zhichao-aws <[email protected]> * make query text and model id required Signed-off-by: zhichao-aws <[email protected]> * minor changes based on comments Signed-off-by: zhichao-aws <[email protected]> * add locale to String.format Signed-off-by: zhichao-aws <[email protected]> * update mock model url Signed-off-by: zhichao-aws <[email protected]> --------- Signed-off-by: zhichao-aws <[email protected]> Signed-off-by: zane-neo <[email protected]> Signed-off-by: xinyual <[email protected]> Co-authored-by: zane-neo <[email protected]> Co-authored-by: xinyual <[email protected]> (cherry picked from commit 7bef7a0)
opensearch-trigger-bot
bot
requested review from
heemin32,
navneet1v,
VijayanB,
vamshin,
jmazanec15,
naveentatikonda,
junqiu-lei,
martin-gaievski,
sean-zheng-amazon,
model-collapse,
wujunshen,
zane-neo,
ylwu-amzn and
jngz-es
as code owners
zane-neo
approved these changes
Sep 27, 2023
5 tasks
* fix apache http version Signed-off-by: zhichao-aws <[email protected]> * add import Signed-off-by: zhichao-aws <[email protected]> --------- Signed-off-by: zhichao-aws <[email protected]>
|
waiting opensearch-project/ml-commons#1398 get merged to run sparse encoding integ test |
model-collapse
approved these changes
Sep 27, 2023
| listener.onResponse(vector); | ||
| }, e -> { | ||
| if (RetryUtil.shouldRetry(e, retryTime)) { | ||
| final int retryTimeAdd = retryTime + 1; | ||
| inferenceSentencesWithRetry(targetResponseFilters, modelId, inputText, retryTimeAdd, listener); | ||
| retryableInferenceSentencesWithVectorResult(targetResponseFilters, modelId, inputText, retryTimeAdd, listener); |
There was a problem hiding this comment.
Why you name function like that? An function should be a verb phrase.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 7bef7a0 from #333