memory-map/on-disk index support

[n0thing233]

Is your feature request related to a problem?
With limited memory resource, it is expensive to load the whole index file to memory and do the knn search

What solution would you like?
The Faiss backend natively support loading on-disk index in a memory-mapped way and do the search on the fly:
https://github.com/facebookresearch/faiss/wiki/Indexes-that-do-not-fit-in-RAM
I would like it to be implemented with open search , or is it already implemented?

What alternatives have you considered?
N/A

Do you have any additional context?
N/A

[jmazanec15]

Hi @n0thing233, in general, I think leveraging disk based ANN approaches could be very beneficial.

There are a couple issues with supporting faiss's implementation:

1. We wouldnt be able to support it on all platforms. I dont believe it works on Windows
2. It requires that several smaller indices be merged together to form the on disk based index: https://github.com/facebookresearch/faiss/wiki/Indexes-that-do-not-fit-in-RAM#building-an-on-disk-index. This would be difficult to do for every segment file, unless we decided to create multiple faiss index files per segment.

All that being said, it is definitely worth looking into supporting some form of disk based ANN.

[vamshin]

Please +1 if you are looking for this feature to help prioritize

[jmazanec15]

Lets track in #1134