Opensearch Serverless filtered query support using knn_score script #15899

GeorgeDittmar · 2024-09-08T01:11:58Z

Description

AWS Opensearch Serverless does not support the painless scripting functions so to perform pre filtering with knn search we have to switch to the non painless knn_score script, https://opensearch.org/docs/latest/search-plugins/knn/knn-score-script/.

This change requires us to handle switching between Opensearch on-prem, AWS Opensearch Managed Service, and AWS Opensearch Serverless. Opensearch on-prem and AWS Managed Opensearch use the same logic luckily.

How service type check is done:

If is_aoss is set during client creation time, then we swap to using the knn_score script as opposed to the painless script. Otherwise, we use the default painless script score functions.

Fixes # (#15560)

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

Yes

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

I tested by running against opensearch serverless as well as docker container version. Also ran current unit tests and they all appeared to pass.

I stared at the code and made sure it makes sense

Suggested Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing unit tests pass locally with my changes
I ran make format; make lint to appease the lint gods

…inless scripting functions according to aws support so must use knn_score instead.

… to be called when aoss is set

…ss is set

…inless scripting functions according to aws support so must use knn_score instead.

… to be called when aoss is set

…ss is set

…nsearch_oss_support

GeorgeDittmar · 2024-09-08T01:15:54Z

Need to pull latest changes to this package still. Just saw there was an update to how deletes are handled.

…ma_index into opensearch_oss_support

GeorgeDittmar · 2024-09-08T17:52:23Z

Need to pull latest changes to this package still. Just saw there was an update to how deletes are handled.

fixed this so should be synced with latest changes. Comments / thoughts welcome on these changes :)

logan-markewich · 2024-09-09T22:56:24Z

...tor_stores/llama-index-vector-stores-opensearch/llama_index/vector_stores/opensearch/base.py

@@ -69,6 +70,7 @@ def __init__(
        text_field: str = "content",
        method: Optional[dict] = None,
        engine: Optional[str] = "nmslib",
+        space_type: Optional[str] = "l2",


What does space_type mean? What does 12 mean? (is this just the naming/value that open search uses? Just confusing for someone out of the loop haha)

space type is basically the distance metric that opensearch uses between vectors. they use the term space type for that. l2 is one of the distance functions. more detail here https://opensearch.org/docs/latest/search-plugins/knn/knn-score-script/

https://opensearch.org/docs/latest/search-plugins/knn/knn-score-script/#spaces better link lol. I could add links in the doc string if that might help for others if they needed to look at it?

ohhhh its L2, not twelve lol

that makes way more sense

logan-markewich · 2024-09-09T22:57:18Z

...tor_stores/llama-index-vector-stores-opensearch/llama_index/vector_stores/opensearch/base.py

        else:
            # https://opensearch.org/docs/latest/search-plugins/knn/painless-functions/
-            search_query = self._default_painless_scripting_query(
+            search_query = self._default_scoring_script_query(
                query_embedding,
                k,
                space_type="l2Squared",


Shouldn't this match self.space_type or nah?

Yeah I wasnt totally sure if that was best here. since the last set of changes used painless scritping and were using space type to hold the l2squared function, which is a variant of the l2 distance metric in the non painless score script, I thought to just keep it defaulted to that. I think it might have been hard coded before but let me double check.

basically l2squared I am not sure depends on the exact space or not, might need to read up on that more.

It seems like it was hardcoded before, so probably fine to leave for now. Just double checking.

Can always improve in a follow-up PR

yeah totally. If others have an idea how best we should proceed on the painless scripting def open to ideas as well.

* v0.11.5 (run-llama#15834) * Box bug fix 202409 (run-llama#15836) * fix document matadata for box file * fix BoxSearchOptions class initialization * bump versions to 0.2.1 * Add oreilly course cookbooks (run-llama#15845) * correct course name (run-llama#15859) * clone llama-deploy docs during docs builds (run-llama#15794) * chore(deps-dev): bump cryptography from 43.0.0 to 43.0.1 in /llama-index-core (run-llama#15840) * Refresh Opensearch index after delete operation (run-llama#15854) * Update ImageReader file loading logic (run-llama#15848) * Minor bug in SQQE (run-llama#15861) * v0.11.6 (run-llama#15863) * remove widgets from recent notebooks (run-llama#15864) * Fix image document deserialization issue (run-llama#15857) * Bugfix: Remove warnings during data insertion for Hybrid Search due to duplicated columns in the schema (run-llama#15846) * Add TablestoreVectorStore. (run-llama#15657) * Removed unused `llama-index-llms-anthropic` dependency from Bedrock Converse (run-llama#15869) Removed unused `llama-index-llms-anthropic` dependency. Incremented to `0.3.0`. * Update bedrock.py (run-llama#15879) Fix minor issue in error message * Add tool calling example in NVIDIA notebook (run-llama#15842) * NVIDIA: Completion fixes (run-llama#15820) * Fix PGVectorStore with latest pydantic, update pydantic imports (run-llama#15886) * add type ignore for streaming agents (run-llama#15887) * [wip] force openai structured output (run-llama#15706) * feature: Make SentenceSplitter's secondary_chunking_regex optional (run-llama#15882) * Bug fix for KuzuPropertyGraphStore: Allow upserting relations even when chunks are absent (run-llama#15889) * Missing tablestore (run-llama#15890) * v0.11.7 (run-llama#15891) * nit: extend workflow tutorial (run-llama#15901) * Modify generate_qa_embedding_pairs to use LLM from Settings (run-llama#15904) * docs: document how to disable global timeout along with more docs (run-llama#15912) document how to disable global timeout along with more docs * fix: model_name property and pydantic v2 in Azure Inference package (run-llama#15907) * fix: model_name property and pydantic v2 * fix: tests * Add InternalServerError to retry decorator (run-llama#15921) * Add InternalServerError to retry decorator * Bump version * docs: fix broken link (run-llama#15924) * Adding vector store for Azure Cosmos DB NoSql (run-llama#14158) * Initial changes * initial changes fixed with example * Initial commit with all the code and test changes * fixing test cases * adding jupyter notebook * cleaning files * resolving comments * fix linting * more linting * add a basic readme --------- Co-authored-by: Aayush Kataria <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]> * update docs for custom embeddings (run-llama#15929) * Add doc.id to Langchain format conversions (run-llama#15928) * Update RankLLM with new rerankers (run-llama#15892) * Feature Request run-llama#15810 :: Add DynamoDBChatStore (run-llama#15917) * Adding support for `MetadataFilters` to WordLift Vector Store (run-llama#15905) * Repair and route colab links to official repo (run-llama#15900) * feat: Add a retry policy to steps (run-llama#15757) * Fix error handling in sharepoint reader (run-llama#15868) * feat: add llama-index llms alibabacloud_aisearch integration (run-llama#15850) * Opensearch Serverless filtered query support using knn_score script (run-llama#15899) * Mistral AI LLM Integration fixes (run-llama#15906) * Fix RagCLI (run-llama#15931) * v0.11.8 (run-llama#15933) * docs: document retry policy (run-llama#15938) * docs: update AI21 docs and readme (run-llama#15937) * docs: update ai21 docs and readme with jamba 1.5 and tool calling * docs: update ai21 docs and readme with jamba 1.5 and tool calling * docs: update ai21 docs and readme with jamba 1.5 and tool calling * Retry retriable errors in neo4j integrations (run-llama#15915) * Add four alibabacloud-aisearch llama-index integrations: rerank, node_parser, readers, embeddings (run-llama#15934) * Differentiate sync and async calls in OpenSearchVectorClient (run-llama#15945) * feat(postgres): add support for engine parameters (run-llama#15951) * feat(postgres): add support for engine parameters - Introduced engine_params to support passing parameters to create_engine. - Updated create_engine and create_async_engine calls to include engine_params. - Initialized engine_params in the constructor. * style(lint): reformat for readability * refactor(postgres): rename engine_params to create_engine_kwargs * refactor(postgres): rename engine_params to create_engine_kwargs * chore: bump version to 0.2.3 * fix(postgres): rename engine_params to create_engine_kwargs * update falkordb client (run-llama#15940) * update falkordb client * bump version * update version * fix * fix: Error when parsing output if tool name contains non-English characters (run-llama#15956) * fix attribute error in PGVectorStore (run-llama#15961) * Catch nest_asyncio errors (run-llama#15975) * NUDGE (run-llama#15954) * Add support for o1 openai models (run-llama#15979) * force temp to 1.0 for o1 (run-llama#15983) * Update concepts.md - fix link for Structured Data Extraction page (run-llama#15982) * Fix the import path example for SimpleMongoReader (run-llama#15988) * Do not pass system prompt from fn calling runner to fn calling worker (run-llama#15986) * Add callback manager to retriever query engine from args (run-llama#15990) * v0.11.9 (run-llama#15992) * Fix: get all documents from Elasticsearch KVStore (run-llama#16006) * Fix Pydantic numeric validation in openai/base.py (run-llama#15993) * feat: add quip reader (run-llama#16000) * [docs/example] Human in loop workflow example (run-llama#16011) * start choose own adventure hitl nb * hitl example * note on alternative implementation * add module guides and run prepare_for_build * fix: removed author info from quip reader (run-llama#16012) fix: removed author info * Fix Pydantic models definition (run-llama#16008) * chore(deps): bump litellm from 1.43.7 to 1.44.8 in /llama-index-integrations/embeddings/llama-index-embeddings-litellm (run-llama#16013) * Hotfix: Fix Citations Text (run-llama#16015) * Jacques/opik integration (run-llama#16007) * update llamacloud index with image nodes (run-llama#15996) * chore: add o1 models pinned versions (run-llama#16025) * add sparse embedding abstraction (run-llama#16018) * Implement `get_nodes` on `PGVectorStore` (run-llama#16026) * Implement `get_nodes` on `PGVectorStore` * Bump version num in pyproject.toml * Update docstring in base.py in MilvusVectorStore adding COSINE as available similarity metric (run-llama#16031) Update docstring in base.py adding COSINE as similarity metric available According to Milvus Documentation, **COSINE** as similarity metric is supported (Both Milvus and Milvus Lite) but in Llama-Index docs was missing. [Link to Milvus official docs](https://milvus.io/docs/metric.md?tab=floating#Similarity-Metrics) I've checked in the [code](https://github.com/run-llama/llama_index/blob/723c2533ed4b7b43b7d814c89af1838f0f1994c2/llama-index-integrations/vector_stores/llama-index-vector-stores-milvus/llama_index/vector_stores/milvus/base.py#L256) , so indeed COSINE is supported, so no more further changes are needed. * Fix: unnecessary warning issue in HuggingFace LLM when tokenizer is provided as argument(run-llama#16035) (run-llama#16037) * Attempt #3 of context/result refactor (run-llama#16036) * temporarily limit lancedb version (run-llama#16045) * fix: new Data Connector adaption for DashVector (run-llama#16028) * v0.11.10 (run-llama#16046) * Fix serde issue for huggingface inference API embedding (run-llama#16053) * wip * wip * fix: fix regression in OctoAI llm provider after 0.11 (run-llama#16002) * fix OctoAI llm provider * adjust code to the latest client * resolve conflicts * fix message format conversion * do not pass max_tokens if None * add chat test * bump version * update LanceDB integration (run-llama#16057) * Adding docstring inside preprocess base code. (run-llama#16060) * Add default to optional args for BedrockEmbedding (run-llama#16067) * wip * wip * Fix optional type for Cohere embedding (run-llama#16068) wip * fix vertex pydantic arguments (run-llama#16069) * Fix result order (run-llama#16078) * Fix result order * Bump version * fix elasticsearch embedding async function (run-llama#16083) * docs: add decorators to the api reference (run-llama#16081) * fix incorrect parameters in VertexAIIndex (run-llama#16080) * feat: update JinaEmbedding for v3 release (run-llama#15971) * Mistralai enable custom endpoint from env (run-llama#16084) * Implement async for multi modal ollama (run-llama#16091) * Remove circular package dep (run-llama#16070) * [chore] fix incorrect `handler.context` for `handler.ctx` in docs (run-llama#16101) fix * fix bug missing import (run-llama#16096) * Add more workflow references to docs (run-llama#16102) * fix cohere tests (run-llama#16104) * Use response synthesizer in context chat engines (run-llama#16017) * Fix mongodb hybrid search, also pass hybrid_top_k in vector retriever (run-llama#16105) * bump openai agent deps (run-llama#16112) * Fixed up test_vectorstore. (run-llama#16113) * Issue 16071: wordpress requires username, password (run-llama#16072) * Issue 16071: wordpress requires username, password * Adding changes suggested in PR template * Use Optional typing keyword * feat: add configurable base_url field in rerank (run-llama#16050) * add base_url * version bump * add default --------- Co-authored-by: Massimiliano Pippi <[email protected]> Co-authored-by: Logan <[email protected]> * Enhance Pandas Query Engine Output Processor (run-llama#16052) * Enhance output processor to temporarily adjust display options. * Make format changes. * Update pyproject.toml --------- Co-authored-by: Logan <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]> * fix workflow docs (run-llama#16117) cr * Update OpenVINO LLM pyproject.toml (run-llama#16130) * Wordpress: Allow control of whether Pages and/or Posts are retrieved (run-llama#16128) * Async achat put operation (run-llama#16127) * [fix] handler.stream_events() doesn't yield `StopEvent` (run-llama#16115) * Improved TLM Rag cookbook (run-llama#16109) * Update chat message class for multi-modal (run-llama#15969) * Add support for Path for SimpleDirectoryReader (run-llama#16108) * Add TopicNodeParser based on MedGraphRAG paper (run-llama#16131) * Sql markdown response (run-llama#16103) * v0.11.11 (run-llama#16134) * Correct Pydantic warning(s) issed for llama-index-llms-ibm (run-llama#16141) Fix Pydantic warnings in llam-index-llms-ibm * feat: add drive link to google drive reader (run-llama#16156) * Introducing new VoyageAI models (run-llama#16150) * Add `required_exts` option to SharePoint reader (run-llama#16152) * User-defined schema in MilvusVectorStore (run-llama#16151) * safe format prompt variables in strings with JSON (run-llama#15734) * account for tools in prompt helper (run-llama#16157) * v0.11.12 (run-llama#16159) --------- Co-authored-by: Logan <[email protected]> Co-authored-by: Ravi Theja <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yannic Spreen-Ledebur <[email protected]> Co-authored-by: Adnan Alkattan <[email protected]> Co-authored-by: Laurie Voss <[email protected]> Co-authored-by: saipjkai <[email protected]> Co-authored-by: ScriptShi <[email protected]> Co-authored-by: Bryce Freshcorn <[email protected]> Co-authored-by: Graham Tibbitts <[email protected]> Co-authored-by: Rashmi Pawar <[email protected]> Co-authored-by: Jerry Liu <[email protected]> Co-authored-by: Caroline Binley <[email protected]> Co-authored-by: Prashanth Rao <[email protected]> Co-authored-by: Asi Greenholts <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]> Co-authored-by: Facundo Santiago <[email protected]> Co-authored-by: gsa9989 <[email protected]> Co-authored-by: Aayush Kataria <[email protected]> Co-authored-by: Ryan Nguyen <[email protected]> Co-authored-by: David Riccitelli <[email protected]> Co-authored-by: GICodeWarrior <[email protected]> Co-authored-by: Javier Torres <[email protected]> Co-authored-by: 才胜 <[email protected]> Co-authored-by: George Dittmar <[email protected]> Co-authored-by: Tarun Jain <[email protected]> Co-authored-by: miri-bar <[email protected]> Co-authored-by: Tomaz Bratanic <[email protected]> Co-authored-by: kobiche <[email protected]> Co-authored-by: Arthur Moura Carvalho <[email protected]> Co-authored-by: Avi Avni <[email protected]> Co-authored-by: Cheese <[email protected]> Co-authored-by: Zac Wellmer <[email protected]> Co-authored-by: Ragul Kachiappan <[email protected]> Co-authored-by: Vinicius <[email protected]> Co-authored-by: Bidimpata-Kerim Aramyan-Tshimanga <[email protected]> Co-authored-by: Laura Ceconi <[email protected]> Co-authored-by: N1eo <[email protected]> Co-authored-by: Chirag Agrawal <[email protected]> Co-authored-by: Andrei Fajardo <[email protected]> Co-authored-by: David Oplatka <[email protected]> Co-authored-by: Jacques Verré <[email protected]> Co-authored-by: Harpinder <[email protected]> Co-authored-by: Richard Liu <[email protected]> Co-authored-by: Jorge Barrachina Gutiérrez <[email protected]> Co-authored-by: Sourabh <[email protected]> Co-authored-by: OceanPresent <[email protected]> Co-authored-by: Simon Suo <[email protected]> Co-authored-by: Ayush Chaurasia <[email protected]> Co-authored-by: preprocess-co <[email protected]> Co-authored-by: guodong <[email protected]> Co-authored-by: polarbear567 <[email protected]> Co-authored-by: Aaron Ji <[email protected]> Co-authored-by: enrico-stauss <[email protected]> Co-authored-by: Selim Çavaş <[email protected]> Co-authored-by: Casey Clements <[email protected]> Co-authored-by: Jonathan Springer <[email protected]> Co-authored-by: Anirudh31415926535 <[email protected]> Co-authored-by: Matin Khajavi <[email protected]> Co-authored-by: Ethan Yang <[email protected]> Co-authored-by: Matthew Turk <[email protected]> Co-authored-by: Andrew Kim <[email protected]> Co-authored-by: José Henrique Luckmann <[email protected]> Co-authored-by: Emanuel Ferreira <[email protected]> Co-authored-by: fzowl <[email protected]> Co-authored-by: João Martins <[email protected]> Co-authored-by: Petros Mitseas <[email protected]>

…un-llama#15899)

GeorgeDittmar added 13 commits September 1, 2024 23:08

checking if is_aoss is set. Opensearch Serverless does not support pa…

6556a88

…inless scripting functions according to aws support so must use knn_score instead.

changing doc string for space_type argument

9153acf

needing to raise an error if painless scritping function is attempted…

3a531da

… to be called when aoss is set

adding error check to not allow for painless scripting function if ao…

483f001

…ss is set

adding some additional comments on certain sections of the logic

ef425d2

checking if is_aoss is set. Opensearch Serverless does not support pa…

a3e8858

…inless scripting functions according to aws support so must use knn_score instead.

changing doc string for space_type argument

b055d85

needing to raise an error if painless scritping function is attempted…

604dbd1

… to be called when aoss is set

adding error check to not allow for painless scripting function if ao…

4ca9af1

…ss is set

adding some additional comments on certain sections of the logic

62b9ec8

Merge remote-tracking branch 'origin/opensearch_oss_support' into ope…

a6e073f

…nsearch_oss_support

more linting

f094ba0

linting change log

a4e58a4

GeorgeDittmar added 3 commits September 7, 2024 21:31

Merge branch 'main' into opensearch_oss_support

5cb4c3d

linting

04a83d9

Merge branch 'opensearch_oss_support' of github.com:GeorgeDittmar/lla…

be44824

…ma_index into opensearch_oss_support

GeorgeDittmar marked this pull request as ready for review September 8, 2024 17:51

dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Sep 8, 2024

GeorgeDittmar added 4 commits September 8, 2024 12:16

adding pyproject.toml version bump

06a3def

Merge branch 'main' into opensearch_oss_support

ac85eec

Merge branch 'main' into opensearch_oss_support

0061715

Merge branch 'main' into opensearch_oss_support

ac0c703

logan-markewich reviewed Sep 9, 2024

View reviewed changes

logan-markewich approved these changes Sep 9, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 9, 2024

logan-markewich merged commit 8be782c into run-llama:main Sep 9, 2024
10 checks passed

raspawar pushed a commit to raspawar/llama_index that referenced this pull request Oct 7, 2024

Opensearch Serverless filtered query support using knn_score script (r…

ca939f4

…un-llama#15899)

raspawar pushed a commit to raspawar/llama_index that referenced this pull request Oct 7, 2024

Opensearch Serverless filtered query support using knn_score script (r…

54c9f75

…un-llama#15899)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Opensearch Serverless filtered query support using knn_score script #15899

Opensearch Serverless filtered query support using knn_score script #15899

GeorgeDittmar commented Sep 8, 2024 •

edited

Loading

GeorgeDittmar commented Sep 8, 2024

GeorgeDittmar commented Sep 8, 2024

logan-markewich Sep 9, 2024

GeorgeDittmar Sep 9, 2024

GeorgeDittmar Sep 9, 2024 •

edited

Loading

logan-markewich Sep 9, 2024 •

edited

Loading

logan-markewich Sep 9, 2024

logan-markewich Sep 9, 2024

GeorgeDittmar Sep 9, 2024

GeorgeDittmar Sep 9, 2024

logan-markewich Sep 9, 2024

GeorgeDittmar Sep 9, 2024

Opensearch Serverless filtered query support using knn_score script #15899

Opensearch Serverless filtered query support using knn_score script #15899

Conversation

GeorgeDittmar commented Sep 8, 2024 • edited Loading

Description

Version Bump?

Type of Change

How Has This Been Tested?

Suggested Checklist:

GeorgeDittmar commented Sep 8, 2024

GeorgeDittmar commented Sep 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GeorgeDittmar Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

logan-markewich Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GeorgeDittmar commented Sep 8, 2024 •

edited

Loading

GeorgeDittmar Sep 9, 2024 •

edited

Loading

logan-markewich Sep 9, 2024 •

edited

Loading