[Feature] k-NN Array support for Vector Field type

[mausch]

In OpenSearch it's expected since forever to be able to pass an array of values to any field.
In fact the documentation says "you can pass an array of values into any field."
However when you try to do that for fields of type knn_vector you get a mapper_parsing_exception

[navneet1v]

@mausch Can you please provide details about your use-case?

[mausch]

A schema where every document has 1..N vectors, plus several other regular OpenSearch fields.
I need to run a kNN search on all vectors, defining the score of the document as the highest similarity of all its vectors against the target vector. Hopefully that makes sense 🙂

At the moment I'm working around things by representing every entity as 1..N documents, but this means I have to duplicate all the other regular fields, and also I have to overspecify the result size.

[naveentatikonda]

@mausch For now, we are not able to add Array support for k-NN Vector Field Type due to the underlying limitation with lucene. By default, knn_vector field supports Binary Doc Values. But, lucene is not supporting Multi BinaryDocValues where the BinaryDocValuesWriter in lucene doesn’t support adding binary values(bytes) of multiple vectors into the same document which throws IllegalArgumentException.

For your use case, we recommend using Nested Field Type as it is supported for ANN Search.

For Example :

PUT train-index
{
    "settings": {
        "index": {
            "knn": true,
            "knn.algo_param.ef_search": 100
        }
    },
    "mappings": {
        "properties": {
            "nested_field": {
                "type": "nested",
                "properties": {
                    "my_vector1": {
                        "type": "knn_vector",
                        "dimension": 3,
                        "method": {
                            "name": "hnsw",
                            "space_type": "l2",
                            "engine": "nmslib",
                            "parameters": {
                                "ef_construction": 128,
                                "m": 24
                            }
                        }
                    }
                }
            }
        }
    }
}

POST train-index/_doc/1
{
    "nested_field" : [
        {
            "my_vector1": [400,50,665],
             "test-field1" : 1
        },
        {
            "my_vector1": [47,589,64],
             "test-field1" : 10
        },
        {
            "my_vector1": [434,52,678],
             "test-field1" : 15
        }
    ] 
}

Please let us know if you have any other questions.

[mausch]

Ah, forgot about nested fields! That's a great workaround, thanks 👍

[heemin32]

For the reference, query will look like

GET train-index/_search
{
  "query": {
    "nested": {
      "path": "nested_field",
      "query": {
        "knn": {
          "nested_field.my_vector1": {
            "vector": [400,50,665],
            "k": 2
          }
        }
      }
    }
  }
}