forked from langchain-ai/langchainjs
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Nc/opensearch merge (langchain-ai#792)
* feat(vectorstore): implement opensearch vector store (langchain-ai#675) * feat(vectorstore): implement opensearch vector store * feat(opensearch): implement filtering by metadata attributes + integration test * Update entryopints, imports, docs --------- Co-authored-by: Igor Shapiro <[email protected]>
- Loading branch information
1 parent
0e52a00
commit 0b8ffb4
Showing
13 changed files
with
512 additions
and
3 deletions.
There are no files selected for viewing
112 changes: 112 additions & 0 deletions
112
docs/docs/modules/indexes/vector_stores/integrations/opensearch.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| --- | ||
| sidebar_class_name: node-only | ||
| --- | ||
|
|
||
| # OpenSearch | ||
|
|
||
| :::tip Compatibility | ||
| Only available on Node.js. | ||
| ::: | ||
|
|
||
| [OpenSearch](https://opensearch.org/) is a fork of [Elasticsearch](https://www.elastic.co/elasticsearch/) that is fully compatible with the Elasticsearch API. Read more about their support for Approximate Nearest Neighbors [here](https://opensearch.org/docs/latest/search-plugins/knn/approximate-knn/). | ||
|
|
||
| Langchain.js accepts [@opensearch-project/opensearch](https://opensearch.org/docs/latest/clients/javascript/index/) as the client for OpenSearch vectorstore. | ||
|
|
||
| ## Setup | ||
|
|
||
| ```bash npm2yarn | ||
| npm install -S @opensearch-project/opensearch | ||
| ``` | ||
|
|
||
| You'll also need to have an OpenSearch instance running. You can use the [official Docker image](https://opensearch.org/docs/latest/opensearch/install/docker/) to get started. You can also find an example docker-compose file [here](https://github.com/hwchase17/langchainjs/blob/main/examples/src/indexes/vector_stores/opensearch/docker-compose.yml). | ||
|
|
||
| ## Index docs | ||
|
|
||
| ```typescript | ||
| import { Client } from "@opensearch-project/opensearch"; | ||
| import { Document } from "langchain/document"; | ||
| import { OpenAIEmbeddings } from "langchain/embeddings"; | ||
| import { OpenSearchVectorStore } from "langchain/vectorstores"; | ||
|
|
||
| const client = new Client({ | ||
| nodes: [process.env.OPENSEARCH_URL ?? "http://127.0.0.1:9200"], | ||
| }); | ||
|
|
||
| const docs = [ | ||
| new Document({ | ||
| metadata: { foo: "bar" }, | ||
| pageContent: "opensearch is also a vector db", | ||
| }), | ||
| new Document({ | ||
| metadata: { foo: "bar" }, | ||
| pageContent: "the quick brown fox jumped over the lazy dog", | ||
| }), | ||
| new Document({ | ||
| metadata: { baz: "qux" }, | ||
| pageContent: "lorem ipsum dolor sit amet", | ||
| }), | ||
| new Document({ | ||
| metadata: { baz: "qux" }, | ||
| pageContent: | ||
| "OpenSearch is a scalable, flexible, and extensible open-source software suite for search, analytics, and observability applications", | ||
| }), | ||
| ]; | ||
|
|
||
| await OpenSearchVectorStore.fromDocuments(docs, new OpenAIEmbeddings(), { | ||
| client, | ||
| indexName: process.env.OPENSEARCH_INDEX, // Will default to `documents` | ||
| }); | ||
| ``` | ||
|
|
||
| ## Query docs | ||
|
|
||
| ```typescript | ||
| import { Client } from "@opensearch-project/opensearch"; | ||
| import { VectorDBQAChain } from "langchain/chains"; | ||
| import { OpenAIEmbeddings } from "langchain/embeddings"; | ||
| import { OpenAI } from "langchain/llms"; | ||
| import { OpenSearchVectorStore } from "langchain/vectorstores"; | ||
|
|
||
| const client = new Client({ | ||
| nodes: [process.env.OPENSEARCH_URL ?? "http://127.0.0.1:9200"], | ||
| }); | ||
|
|
||
| const vectorStore = new OpenSearchVectorStore(new OpenAIEmbeddings(), { | ||
| client, | ||
| }); | ||
|
|
||
| /* Search the vector DB independently with meta filters */ | ||
| const results = await vectorStore.similaritySearch("hello world", 1); | ||
| console.log(JSON.stringify(results, null, 2)); | ||
| /* [ | ||
| { | ||
| "pageContent": "Hello world", | ||
| "metadata": { | ||
| "id": 2 | ||
| } | ||
| } | ||
| ] */ | ||
|
|
||
| /* Use as part of a chain (currently no metadata filters) */ | ||
| const model = new OpenAI(); | ||
| const chain = VectorDBQAChain.fromLLM(model, vectorStore, { | ||
| k: 1, | ||
| returnSourceDocuments: true, | ||
| }); | ||
| const response = await chain.call({ query: "What is opensearch?" }); | ||
|
|
||
| console.log(JSON.stringify(response, null, 2)); | ||
| /* | ||
| { | ||
| "text": " Opensearch is a collection of technologies that allow search engines to publish search results in a standard format, making it easier for users to search across multiple sites.", | ||
| "sourceDocuments": [ | ||
| { | ||
| "pageContent": "What's this?", | ||
| "metadata": { | ||
| "id": 3 | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| */ | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
42 changes: 42 additions & 0 deletions
42
examples/src/indexes/vector_stores/opensearch/docker-compose.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Reference: | ||
| # https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/#sample-docker-composeyml | ||
| version: '3' | ||
| services: | ||
| opensearch: | ||
| image: opensearchproject/opensearch:2.6.0 | ||
| container_name: opensearch | ||
| environment: | ||
| - cluster.name=opensearch | ||
| - node.name=opensearch | ||
| - discovery.type=single-node | ||
| - bootstrap.memory_lock=true | ||
| - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" | ||
| - "DISABLE_INSTALL_DEMO_CONFIG=true" | ||
| - "DISABLE_SECURITY_PLUGIN=true" | ||
| ulimits: | ||
| memlock: | ||
| soft: -1 | ||
| hard: -1 | ||
| volumes: | ||
| - opensearch_data:/usr/share/opensearch/data | ||
| ports: | ||
| - 9200:9200 | ||
| - 9600:9600 | ||
| networks: | ||
| - opensearch | ||
| opensearch-dashboards: | ||
| image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes | ||
| container_name: opensearch-dashboards | ||
| ports: | ||
| - 5601:5601 # Map host port 5601 to container port 5601 | ||
| expose: | ||
| - "5601" # Expose port 5601 for web access to OpenSearch Dashboards | ||
| environment: | ||
| OPENSEARCH_HOSTS: '["http://opensearch:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query | ||
| DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true" # disables security dashboards plugin in OpenSearch Dashboards | ||
| networks: | ||
| - opensearch | ||
| networks: | ||
| opensearch: | ||
| volumes: | ||
| opensearch_data: |
22 changes: 22 additions & 0 deletions
22
examples/src/indexes/vector_stores/opensearch/opensearch.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| import { Client } from "@opensearch-project/opensearch"; | ||
| import { OpenAIEmbeddings } from "langchain/embeddings/openai"; | ||
| import { OpenSearchVectorStore } from "langchain/vectorstores/opensearch"; | ||
|
|
||
| export async function run() { | ||
| const client = new Client({ | ||
| nodes: [process.env.OPENSEARCH_URL ?? "http://127.0.0.1:9200"], | ||
| }); | ||
|
|
||
| const vectorStore = await OpenSearchVectorStore.fromTexts( | ||
| ["Hello world", "Bye bye", "What's this?"], | ||
| [{ id: 2 }, { id: 1 }, { id: 3 }], | ||
| new OpenAIEmbeddings(), | ||
| { | ||
| client, | ||
| indexName: "documents", | ||
| } | ||
| ); | ||
|
|
||
| const resultOne = await vectorStore.similaritySearch("Hello world", 1); | ||
| console.log(resultOne); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.