Storage Strategies

[gorango]

Exploring ways to implement a custom storage solution to persist data to Postgres. Looking at the storageContextFromDefaults implementation, it seems that I should be able to create my own storage classes and provide them to the initializer:

import { PGVectorStore, storageContextFromDefaults } from 'llamaindex'
import { CustomDocumentStore, CustomIndexStore } from '../custom.ts'

const storageContext = await storageContextFromDefaults({
  docStore: new CustomDocumentStore(...args),
  indexStore: new CustomIndexStore(...args),
  vectorStores: new PGVectorStore(...args)
});

Are there any helpful guides or examples of a working implementation for this approach?

I assume I should just implement the methods that are on the KVDocumentStore and the KVIndexStore? It also seems that implementing a pgStore in the style of the current kvStore should allow for easier swapping of storage methods.

The Python version also implements S3 storage and I understand that the plan is to achieve parity between the two libraries. Are there some particular difficulties with implementing flexible storage in this library that we should know before diving in?

[marcusschiesser]

You'll need to implement a doc store and index store for Postgres, similar to

1. https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/storage/docstore/llama-index-storage-docstore-postgres/llama_index/storage/docstore/postgres/base.py and
2. https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/storage/index_store/llama-index-storage-index-store-postgres/llama_index/storage/index_store/postgres/base.py

Would be great if you could get this started, we're happy to help with the PR

[gorango]

Thanks, that seems fairly straightforward. I'll take a stab at it.

[gorango]

Started a PR #1180