Elastic Search Vector Store - Unable to Add Documents

[vinodkiran]

Unable to add documents to Elastic Search Vector Store.

  const loader = new DirectoryLoader("./documents", {
    ".txt": (path) => new TextLoader(path),
  });
  const documents = await loader.load()
  const text_splitter = new CharacterTextSplitter({chunkSize:1000, chunkOverlap:0})
  const docs = await text_splitter.splitDocuments(documents)
  console.log("Docs found...."+docs.length);
  const embeddings = new OpenAIEmbeddings();
  const vectorStore = new ElasticVectorSearch(embeddings, clientArgs);
  const ids = await vectorStore.addDocuments(docs);
 
 // the following statement returns IDs but the insertion fails.
  console.log("Docs loaded...."+ids.length);

There is no error logged and the ids array is non-null, but the underlying vector store has no documents.

Full Code

[dosubot]

🤖

Hello,

From the code snippet you've provided, it seems like you're correctly using the addDocuments method from the ElasticVectorSearch class. This method is designed to add documents to the Elasticsearch database by converting the documents to vectors using the embeddings, and then adding the vectors to the database.

However, it's important to note that the addDocuments method is asynchronous and returns a promise. This means that you need to ensure that the promise is fully resolved before you check the vector store for the documents. If you're checking the vector store immediately after calling addDocuments, it's possible that the promise hasn't resolved yet, which would explain why you're not seeing the documents in the vector store.

Here's how you can modify your code to ensure that the promise is fully resolved before checking the vector store:

const ids = await vectorStore.addDocuments(docs);
console.log("Docs loaded...."+ids.length);

// Wait for the promise to resolve
await Promise.all(ids.map(id => vectorStore.getDocument(id)));

// Now check the vector store for the documents

This code uses the Promise.all method to wait for all the promises returned by getDocument to resolve before checking the vector store for the documents. This ensures that all the documents have been fully added to the vector store before you check for them.

If you're still not seeing the documents in the vector store after making this modification, it's possible that there's an issue with the Elasticsearch setup. Please double-check your Elasticsearch configuration and make sure that it's correctly set up to work with the LangChainJS framework.

I hope this helps! Let me know if you have any other questions.

Sources

• examples/src/indexes/vector_stores/elasticsearch/elasticsearch.ts
• langchain/src/vectorstores/elasticsearch.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[vinodkiran]

@dosu-bot

Update:: when the metadata is removed the insertion is successful.

  docs.forEach((doc) => {
    doc.metadata = {};
  });

[dosubot]

🤖

Hello,

Based on the context provided, it seems like the issue you're facing is similar to a previously solved issue in the LangChainJS repository. The issue was titled "Opensearch store does not support object in metadata and fail silently" and it was resolved by either not supporting objects in metadata or by throwing an appropriate error so that the client is aware of the problem.

The problem you're encountering might be due to the Elastic Search Vector Store not being configured to handle metadata. To resolve this, you need to ensure that the Elastic Search Vector Store is properly configured to handle metadata. This can be done by modifying the mapping of the Elastic Search index to include the metadata field.

Here's a general guide on how to modify the mapping of an index in Elastic Search: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html

Unfortunately, without the code for the ElasticVectorSearch class or the addDocuments method, I can't provide a specific code solution.

As for the structure of the metadata object, it's defined as an optional property of the DocumentInput interface and the Document class. It's a generic type that extends a Record of string keys to any value. This means that the metadata object can contain any key-value pairs, where the keys are strings and the values can be of any type.

If the issue persists, it could be due to other factors such as network issues, incorrect Elastic Search configuration, or issues with the documents themselves. In such cases, it would be helpful to check the error message returned by Elastic Search for more information.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• Opensearch store does not support object in metadata and fail silently
• examples/src/indexes/vector_stores/elasticsearch/elasticsearch.ts
• langchain/src/vectorstores/xata.ts
• examples/src/indexes/vector_stores/voy.ts
• langchain/src/document.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jacoblee93]

@ezimuel would you be able to have a look at this?