Add the ability to compute vector similarity scores with the new ValuesSource API

[jpountz]

Description

#12253 introduced the ability to compute vector similarity with the legacy value source API, let's introduce similar functionality to DoubleValuesSource?

[jpountz]

The two values source APIs are very different, so here's a proposal for new method signatures with the new API: DoubleValues distanceFromQueryVector(float[] queryVector, String vectorField) (for float[] vectors) and DoubleValues distanceFromQueryVector(byte[] queryVector, String vectorField) (for byte[] vectors).

[msokolov]

The idea makes sense to me, but I don't like the word "distance" in this context because not all of the similarities are distances in the sense of a metric space. That's why I prefer similarity - maybe we could call it vectorSimilarity or similarityToVector or similarityToQueryVector?

[shubhamvishu]

I would like to take this up!

@jpountz Do you mean we could have something like ByteVectorSimilarityValuesSource and FloatVectorSimilarityValuesSource extending DoubleValuesSource in package org.apache.lucene.queries.function.valuesource or org.apache.lucene.search or maybe something else? Could you please elaborate?

I was thinking if we could have a DVS where #getValues returns the per document vector similarity scores(i.e. DoubleValues) based on the passed VectorSimilarityFunction to DVS ?

[jpountz]

They would be pkg-private in org.apache.lucene.search and exposed via factory methods on DoubleValues.

I was thinking if we could have a DVS where #getValues returns the per document vector similarity scores(i.e. DoubleValues) based on the passed VectorSimilarityFunction to DVS ?

FWIW we don't need to take an input similarity function and could rely on the one that is configured on the FieldInfo of the provided field.

[shubhamvishu]

I see....thanks for clarifying @jpountz

[shubhamvishu]

@jpountz I have raised a PR #12548 that adds the required APIs to DVS for computing vector similarity scores. Thanks!