Add support for Lucene inbuilt Scalar Quantizer

[naveentatikonda]

Description

This PR adds support for inbuilt Scalar Quantizer with Lucene engine which takes fp32 vectors as input and dynamically quantizes them based on the number of bits(default 7) and other parameters like confidence_interval provided in the index mapping. For now, we are not supporting 8 bits due to a recall issue and only supporting 7 bits.

UX changes

To keep it consistent with Faiss, we are adding sq encoder under parameters field. The sq encoder in Lucene supports 3 optional parameters:

• bits - Which decides the number of buckets for the input vector data to be quantized into; Defaults to 7.
• confidence_interval - Which is used to determine the minQuantile and maxQuantile params used in the Quantization process. Acceptable values are 0, >=0.9 && <=1.0. If the confidence_interval is not provided we will set default value as null which is computed later in Lucene from the dimension of the vector as 1 - 1/(1 + dimension).

Sample Index mapping using sq encoder is shown below:

    "mappings": {
        "properties": {
            "my_vector1": {
                "type": "knn_vector",
                "dimension": 2,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "lucene",
                    "parameters": {
                        "encoder": {
                            "name": "sq",
                            "parameters": {
                                "bits": 7,
                                "confidence_interval": 1.0
                            }
                        }
                        "ef_construction": 128,
                        "m": 24,
                    }
                }
            }
        }
    }

Related Issues

#1277

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[naveentatikonda]

Note - I'm still running some benchmarking tests for 4 bits, will complete by weekend and include 4 bits if there are no performance issues. That will be a very small change.

[navneet1v]

• bits - Which decides the number of buckets for the input vector data to be quantized into; Defaults to 7.
• confidence_interval - Which is used to determine the minQuantile and maxQuantile params used in the Quantization process. Acceptable values are 0, >=0.9 && <=1.0. If the confidence_interval is not provided we will set default value as null which is computed later in Lucene from the dimension of the vector as 1 - 1/(1 + dimension).
• compress - Default value is false, which is useful only with 4 bits to compress and provide more memory reduction.

Seeing these different parameters and how they work. Lets ensure documentation is very clear otherwise there will be a lot of confusions.

[jmazanec15]

Whats the validation on the bits parameter?

[jmazanec15]

Is 7-bit quantization implemented significantly different than 8-bit?

[naveentatikonda]

Whats the validation on the bits parameter?

In Lucene.java as part of parameter validation, created a list of integers which has the bits that we are supporting and validating against it.

[naveentatikonda]

Is 7-bit quantization implemented significantly different than 8-bit?

no, the difference is the number of buckets we are quantizing the given vectors into, which is 2^(bits) - 1 buckets

[heemin32]

Nit. This style is error prune.

Suggested change

	if (value.equals(0)) value = 0.0;
	if (value.equals(0)) {
	value = 0.0;
	}

[heemin32]

No conversion here?

if (value.equals(0)) {
  value = 0.0;
  }

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1848-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 71fff475cd76f3fe7fd195a15f54138c9127ad6a
# Push it to GitHub
git push --set-upstream origin backport/backport-1848-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1848-to-2.x.

[opensearch-trigger-bot]

The backport to 2.16 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.16 2.16
# Navigate to the new working tree
cd .worktrees/backport-2.16
# Create a new branch
git switch --create backport/backport-1848-to-2.16
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 71fff475cd76f3fe7fd195a15f54138c9127ad6a
# Push it to GitHub
git push --set-upstream origin backport/backport-1848-to-2.16
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.16

Then, create a pull request where the base branch is 2.16 and the compare/head branch is backport/backport-1848-to-2.16.

[heemin32]

For now, we are not supporting 8 bits due to a recall issue and only supporting 7 bits.

Does that mean, user can only set 7 bits but nothing else?

[naveentatikonda]

For now, we are not supporting 8 bits due to a recall issue and only supporting 7 bits.

Does that mean, user can only set 7 bits but nothing else?

yes, for now only 7 bits. We will support 8 bits and 4 bits after the underlying quantization issues are resolved in lucene.