VectorStore for GenAI integrations

[maxjakob]

We want to add a higher-level abstraction for using Elasticsearch in Python. The main motivation is to make integrations into 3rd party GenAI libraries easier and give users a handful of options to choose from without diving into the Query DSL. Ideally the abstraction also works for direct users of the elasticsearch-py client.

Integrations

• LangChain: current and draft branch utilizing this PR
• LlamaIndex: current and draft branch utilizing this PR
• Haystack: current

TODO

• Agree on shape of the interfaces <-- nobody complained ;)
• Move client creation code to 3rd parties, receive client object
• Separate classes for sync and async
• Move embedding service calls to the store, batch the inference
• Move module to helpers.vectorstore
• Deduplicate field names between store and strategies (as in the original implementation)
• Deduplicate application of metadata mapping
• Semantic strategy: test it once there is a Docker image with it or remove it
• Make integration tests work in CI
• Consider limiting values for k and num_candidates for (async) MMR --> CPU-intensive MMR function is now sync only
• Create extra requirements for MMR-related dependencies
• Adhere to repo-specific code styles
• Inquire what needs to be done for the -serverless package --> nothing here, @pquentin will take care of this.
• Add default arg for user agent
• Documentation? <-- We can worry about it later when the integrations are stable.

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[pquentin]

Thanks for this draft! I've reviewed with an eye towards the big things that could be improved to make this feel native in the Python client. I've not looked at the detail of the code, since I don't expect to bring value here.

The big question is going to be the name of this module and its location in the client.

• I don't think store works here: we have to communicate that this is an LLM/GenAI thing. I would thus call it vectorstore or vector_store. (Indeed, according to PEP 8, underscores can be used in the module name if it improves readability.)
• Currently, all the manually written code is in elasticsearch.helpers which felt like the natural location for this new code initially. But I'm less sure now, given existing helpers are simply functions and this is a lot more code with a lot more state. So I'm tempted to think that elasticsearch.vector_store is a better location than elasticsearch.helpers.vector_store.

[maxjakob]

• I don't think store works here: we have to communicate that this is an LLM/GenAI thing. I would thus call it vectorstore or vector_store. (Indeed, according to PEP 8, underscores can be used in the module name if it improves readability.)

It was called vectorstore for the longest time :) I renamed it because of the BM25 capabilities. But you're right, it's more appealing/familiar to people with vector in the name.

[maxjakob]

• Currently, all the manually written code is in elasticsearch.helpers which felt like the natural location for this new code initially. But I'm less sure now, given existing helpers are simply functions and this is a lot more code with a lot more state. So I'm tempted to think that elasticsearch.vector_store is a better location than elasticsearch.helpers.vector_store.

I share this sentiment (but don't have a strong opinion).

[ezimuel]

• I don't think store works here: we have to communicate that this is an LLM/GenAI thing. I would thus call it vectorstore or vector_store. (Indeed, according to PEP 8, underscores can be used in the module name if it improves readability.)

I think vectorstore (or vector_store) is a better name, since it puts evidence on the semantic search feature.

• Currently, all the manually written code is in elasticsearch.helpers which felt like the natural location for this new code initially. But I'm less sure now, given existing helpers are simply functions and this is a lot more code with a lot more state. So I'm tempted to think that elasticsearch.vector_store is a better location than elasticsearch.helpers.vector_store.

Since this code consumes the client as an additional layer, I think putting it into elasticsearch.helpers namespace is the perfect fit. The helpers should collects all the high level features that facilitate the usage of the Elasticsearch endpoints.

[pquentin]

Reviewed two small things, but please let me know when you need a more complete review.

[maxjakob]

@pquentin Can I get your eyes on this again? 👀 Some questions from my side:

1. Do we agree on the general interfaces?
2. Can you give me some pointers on how to best set up vectorstor as an optional dependency? EDIT: ✅
3. Do we need to do anything special for the -serverless package? EDIT: ✅

[maxjakob]

Adding the new classes to the documentation (this may require reformatting the docstrings to follow the sphinx format, as I noted in the code)

👍 reformatting...

Async fixtures and tests

I will do this in a follow-up PR because to do it right, I would like create 2 directories _async and _sync like in the application code. That would involve moving existing tests and I would like to keep that out of this PR.

[pquentin]

Thanks! We looked at this together and found a few really minor formatting things to change. I'll test that quickly and then we can merge! 🎉

[pquentin]

nit: We could consider unit testing all those combinations with ReferenceJson and help from GitHub Copilot to generate the tests. But then up to you as this is also quite simple and the tests are going to look a lot like the original code anyway.

Maybe focus on the raised exceptions, using code like with pytest.raises(match="specify a query_vector"): ....

[maxjakob]

Testing the error states now. Assertions for the ES queries are part of the integration tests.

[pquentin]

Thanks! LGTM.