Obtain nodes from VectorStoreIndex

[KevinMusgrave]

I'm loading an index from disk like this:

index = load_index_from_storage(storage_context, service_context=service_context)

I want to obtain the nodes so that I can create a node_dict to pass into the RecursiveRetriever constructor:

retriever_chunk = RecursiveRetriever(
    "vector",
    retriever_dict={"vector": vector_retriever_chunk},
    node_dict=all_nodes_dict,
    verbose=True,
)

Is there a way to do this?

[dosubot[bot]]

🤖

To obtain the nodes from the loaded index in order to create a node_dict for the RecursiveRetriever constructor in the LlamaIndex framework, you can use the ref_doc_info property of the TreeIndex class. This property retrieves a dictionary mapping of ingested documents and their nodes+metadata.

Here is how you can do it:

# Assume you have an instance of TreeIndex
tree_index = TreeIndex(...)

# Get the nodes
ref_doc_info = tree_index.ref_doc_info

# Now you have a dictionary where the keys are node_ids and the values are RefDocInfo objects
# You can create a node_dict for the RecursiveRetriever constructor like this:
node_dict = {node_id: ref_doc_info.node for node_id, ref_doc_info in ref_doc_info.items()}

In the code above, ref_doc_info is a dictionary where the keys are node_ids and the values are RefDocInfo objects. Each RefDocInfo object has a node attribute which is the actual node object. So, we create a new dictionary node_dict where the keys are node_ids and the values are the actual node objects. This node_dict can be used for the RecursiveRetriever constructor.

Please note that you need to replace TreeIndex(...) with the actual parameters required to create a TreeIndex instance.

You can find more details about this in the llama_index/indices/tree/base.py file.

Regarding the load_index_from_storage function, it is used to load an index from a storage context. It takes in a StorageContext object, an optional index_id and any additional keyword arguments. The function returns a BaseIndex object. If no index is found in the storage context, it raises a ValueError. If more than one index is found but no index_id is specified, it also raises a ValueError.

Here is the function definition:

def load_index_from_storage(
    storage_context: StorageContext,
    index_id: Optional[str] = None,
    **kwargs: Any,
) -> BaseIndex:
    """Load index from storage context.

    Args:
        storage_context (StorageContext): storage context containing
            docstore, index store and vector store.
        index_id (Optional[str]): ID of the index to load.
            Defaults to None, which assumes there's only a single index
            in the index store and load it.
        **kwargs: Additional keyword args to pass to the index constructors.
    """
    index_ids: Optional[Sequence[str]]
    if index_id is None:
        index_ids = None
    else:
        index_ids = [index_id]

    indices = load_indices_from_storage(storage_context, index_ids=index_ids, **kwargs)

    if len(indices) == 0:
        raise ValueError(
            "No index in storage context, check if you specified the right persist_dir."
        )
    elif len(indices) > 1:
        raise ValueError(
            f"Expected to load a single index, but got {len(indices)} instead. "
            "Please specify index_id."
        )

    return indices[0]

You can find more details about this in the llama_index/indices/loading.py file.

Sources

• llama_index/indices/tree/base.py
• llama_index/indices/loading.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[KevinMusgrave]

@dosu-bot I'm not using TreeIndex and the tutorial for RecursiveRetriever doesn't indicate that I need to use TreeIndex: https://docs.llamaindex.ai/en/stable/examples/retrievers/recursive_retriever_nodes.html#chunk-references-smaller-child-chunks-referring-to-bigger-parent-chunk

[dosubot[bot]]

🤖

To create a node_dict for the RecursiveRetriever constructor in the LlamaIndex framework without using the TreeIndex class, you can parse the documents into nodes using the SentenceSplitter class, create an IndexNode for each parsed node, and then convert the list of nodes into a dictionary. Here's how you can do it:

from llama_index.node_parser import SentenceSplitter
from llama_index.schema import IndexNode

# Parse the documents into nodes
node_parser = SentenceSplitter(chunk_size=1024)
base_nodes = node_parser.get_nodes_from_documents(docs)

# Create an IndexNode for each parsed node
for idx, node in enumerate(base_nodes):
    node.id_ = f"node-{idx}"

# Convert the list of nodes into a dictionary
all_nodes_dict = {n.node_id: n for n in base_nodes}

You can then use all_nodes_dict as the node_dict argument when constructing a RecursiveRetriever:

from llama_index.retrievers import RecursiveRetriever
from llama_index import VectorStoreIndex

vector_index = VectorStoreIndex(base_nodes, service_context=service_context)
vector_retriever = vector_index.as_retriever(similarity_top_k=2)

retriever = RecursiveRetriever(
    "vector",
    retriever_dict={"vector": vector_retriever},
    node_dict=all_nodes_dict,
    verbose=True,
)

This will create a RecursiveRetriever that uses the nodes from all_nodes_dict for retrieval. You can find more details in the recursive_retriever_nodes.ipynb example.

Regarding the load_index_from_storage function, it is used to load an index from a given storage context. The function takes in a storage_context of type StorageContext, which contains a document store, index store, and vector store. It also optionally takes an index_id which is the ID of the index to load. If no index_id is provided, the function assumes there's only a single index in the index store and loads it. You can find more details in the loading.py file.

The RecursiveRetriever constructor in the LlamaIndex framework is __init__. It takes several parameters including root_id, retriever_dict, query_engine_dict, node_dict, callback_manager, query_response_tmpl, and verbose. You can find more details in the recursive_retriever.py file.

Sources

• docs/examples/retrievers/recursive_retriever_nodes.ipynb
• llama_index/indices/loading.py
• llama_index/retrievers/recursive_retriever.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[KevinMusgrave]

@dosu-bot That requires access to the Document objects. I want to obtain the nodes directly from a VectorStoreIndex object.

[dosubot[bot]]

🤖

Based on a similar issue found in the LlamaIndex repository, you can retrieve the vectors and the list of indexed documents from the VectorStoreIndex object using the following code:

vector_store = index.storage_context.vector_store
vector_store_dict = vector_store.to_dict()
embedding_dict = vector_store_dict['embedding_dict']
vectors = numpy.array(list(embedding_dict.values()))

index_store = index.storage_context.index_store
index_store_dict = index_store.to_dict()
doc_ids = list(index_store_dict['index_store/data'].values())[0]['__data__']['doc_id_dict']

You can then use these doc_ids to fetch the nodes from the docstore. To associate the vectors with the original documents, you can use:

vector_store._data.text_id_to_doc_id  # this will help map the doc ids to text ids
vector_store._data.embedding_dict  # this is accessed using text ids

However, it seems that the document titles or file names/paths are not stored in the embedding_dict. For this task, you might be better off generating the embeddings yourself outside of LlamaIndex.

To extract nodes directly from a VectorStoreIndex object in the LlamaIndex framework to create a node_dict for use in the RecursiveRetriever constructor, you can use the metadata_dict attribute of the SimpleVectorStoreData class. This attribute is a dictionary that maps node IDs to their corresponding metadata.

Here is a simple way to extract the nodes:

def extract_nodes(vector_store):
    node_dict = {}
    for node_id, metadata in vector_store._data.metadata_dict.items():
        node_dict[node_id] = metadata
    return node_dict

# Usage
vector_store = SimpleVectorStore()  # Initialize your vector store
node_dict = extract_nodes(vector_store)

This function iterates over the metadata_dict of the SimpleVectorStoreData object, and creates a new dictionary (node_dict) that maps node IDs to their corresponding metadata. This node_dict can then be used in the RecursiveRetriever constructor.

Please note that this code assumes that the SimpleVectorStore object has been properly initialized and populated with data. If the SimpleVectorStore object is empty or not properly initialized, the metadata_dict attribute will also be empty, and the node_dict will be empty as well.

For more details, you can refer to the source code of the SimpleVectorStoreData class in the LlamaIndex repository: llama_index/vector_stores/simple.py

I hope this helps! If you have any other questions, feel free to ask.

Sources

• Is it possible to retrieve a list of indexed documents (paths and/or title/metadata), and the vector store itself, from a GPTVectorStoreIndex?
• llama_index/vector_stores/simple.py
• llama_index/indices/vector_store/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[KevinMusgrave]

As answered on discord:

nodes = index.docstore.docs.values()

[dhirajsuvarna]

@KevinMusgrave - even if we get the nodes, how do we generate node_dict from it?

[Falven]

As answered on discord:

nodes = index.docstore.docs.values()

This should be added to the docs!

[badrinathvm]

Hi @KevinMusgrave how can we access the nodes if the nodes have been stored in chroma vector store or open search vector store instead on disk ?

[yzgrfsy]

@badrinathvm ，hi，Have you found a solution? I have a similar problem when i use ES
vector_store = ElasticsearchStore(
index_name=index_name, es_url=es_url
)
index = VectorStoreIndex.from_vector_store(embed_model=Settings.embed_model, vector_store=vector_store,storage_context=storage_context)

[david1542]

I'm joining @badrinathvm's question

[bowenyan]

Did anyone try this? To access the nodes if stored in Chroma DB in advance it will return empty dict.
nodes = index.docstore.docs.values()

These two might be helpful:
#10372
#9893

[hunryandev]

me too, with Qdrant vector store, nodes = index.docstore.docs.values() returned an empty dict

[Avinash-Raj]

@bowenyan @badrinathvm

You could try the below solution to get the nodes (upto a specific count) without asking any question. You could add filters too.

from llama_index.core.vector_stores.types import MetadataFilters, VectorStoreQuery

async def get_nodes(
    index: VectorStoreIndex, filters: Optional[MetadataFilters] = None, limit: int = 100, **kwargs
) -> list[NodeWithScore]:
    """
    Get the nodes by applying filters.
    """
    retriever = index.as_retriever(filters=filters, similarity_top_k=limit, **kwargs)  # how many nodes to return
    query = VectorStoreQuery(
        similarity_top_k=retriever._similarity_top_k,
        filters=retriever._filters,
    )
    query_result = await retriever._vector_store.aquery(query, **retriever._kwargs)
    nodes = retriever._build_node_list_from_query_result(query_result)
    return nodes

[fabiochiusano]

For ChromaDB, this works for me as much as it is ugly:

from llama_index.vector_stores.chroma import ChromaVectorStore

vector_store = ChromaVectorStore(...)
vector_index = VectorStoreIndex.from_vector_store(vector_store)

# set similarity_top_k to a large number to retrieve all the nodes
retriever = vector_index.as_retriever(similarity_top_k=1000000)

# retrieve all nodes
all_nodes = retriever.retrieve("Whatever")
all_nodes = [item.node for item in all_nodes]

It would be nice to have something cleaner though