[META] Binary type support with Faiss engine

[heemin32]

There has been an increasing interest on binary format support from customers recently with many evidence showing good recall with binary value generated from LLM. As KNN with large data set is expensive due to cost of memory, customer want to use binary format instead of float format to reduce the required memory size by 1/32.

Tasks

High Level Tasks:

• Create RFC for the Feature Request. [RFC] Binary vector support #1767
• HNSW support
  • JNI interface
  • POC heemin32@efa0199
  • Faiss patch to support SearchParameter for binary index
  • Index mapping, Ingestion, Query
  • Benchmark
• Script scoring support on binary vector with hamming distance
  • Block non hamming distance scoring on binary vector
  • Implement script scoring with hamming distance on binary vector
• IVF support
  • JNI interface
  • Faiss patch
  • Training, Ingestion, Query
  • Benchmark
• Radial search
  • Radial search with HNSW
  • Radial search with IVF
  • Benchmark

Community Requests

1. [FEATURE] Hamming distance / binary vector support #81