Add quantization state cache

[ryanbogan]

Description

Adds a cache for storing quantization states. This PR contains the cache and tests without integration. In a later PR, this cache will be integrated with a quantization state reader and the query flow.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Vikasht34]

Quantization Cache Class Looks good , but we need to improve on unit tests for following test cases

1. testSingleThreadedAddAndRetrieve
   2.testMultiThreadedAddAndRetrieve
   3.testMultiThreadedEvict
   4.testConcurrentAddAndEvict
   5.testCacheClearFromMultipleThreads

Let's have all these tests added.

[Vikasht34]

Looks Good to me .

[ryanbogan]

The most recent commits should address @navneet1v 's comments 1-5. Will add to stats in follow up PR. I also still need to add more unit testing for the new functionality.

[navneet1v]

what is reasoning behind these limits?

[ryanbogan]

I added initial ones based on conversation with @Vikasht34 . We'll probably need to think more on this with benchmarking but for now I needed a value to put

[Vikasht34]

Here is the reasoning behind this

Number of segments :- 2500 ( this must be exterme)
Dimesion 16000
Binary Quantization 4 bit
Threshould will looks like this

thresholds[0] {0.1f, 0.2f, 0.3f  upto 16k }
     *   thresholds[1] {0.4f, 0.5f, 0.6f upto 16k}
     *   thresholds[2] {0.7f, 0.8f, 0.9f upto 16k}
     *   thresholds[3] {1.0f, 1.1f, 1.2f upto 16k}

For 2500 segments, the total size of the thresholds array would be:
2500 segments x 16,000 columns x 4 bytes/element = 1,600,000,000 bytes

1,600,000,000 bytes = 1.6 GB (gigabytes)

So, the total size of your thresholds array for 2500 segments would be approximately 1.6 GB.
If you have a 32 GB heap, this array would occupy:
(1.6 GB / 32 GB) x 100% ≈ 5%

this is extreme case assuming all vectors are 16000 size and 2500 segments

Assuming P90 case where avregae Dimesion is 5000 with 4bit :- 0.5 GB which 1.5625% of 32gb heap , so we choose 5% as safe limit which can handle exterme case as well.

CacheBuilder.newBuilder().maximumWeight(2GB), the cache doesn't immediately allocate 2GB of heap space. Instead, the cache will grow in memory usage as entries are added, up to the specified weight limit (in this case, 2gb).

[jmazanec15]

2500 segments x 16,000 columns x 4 bytes/element = 1,600,000,000 bytes

This is 160,000,000 bytes - 150 MB. 5k is 50 MB. That being said, I dont think we need these limits.

[jmazanec15]

Also, if we do need the limits, can we not make them settings? Settings are hard to remove. It seems in the future, we may want to rely on IndexInput and mmap to handle caching. Id prefer not to expose these as settings

[navneet1v]

shouldn't this just be a counter of type long?

[ryanbogan]

Not sure I understand, why would it be a counter instead of just a long holding the max size (which is calculated from the percent setting)

[navneet1v]

Shouldn't this be more of a logic where if entry is not present in the cache then we load the value in the cache and then return that value?

Any reason of not making this as Read-Through cache?

@Vikasht34 thoghts?

[Vikasht34]

Well here is my thoughts on this , I am really not in favour of clubbing cache with Quantization Reader, since we don't know what would be future for Cache interms of unifying intefaces across multiple caches and implementation , I would keep this totally light weight,

Neither I want NativeQuantizationState Reader to know about cache .

QuantizationStateCache: Manages in-memory caching of quantization states.
QuantizationStateReader: Handles reading quantization states from persistent storage.
QuantizationStateWriter: Manages writing quantization states to persistent storage.

From Query flow
I will have my Interface

public interface QuantizationStateLoader {
    QuantizationState getQuantizationState(String fieldName, String typeofQuantization);
}

and this inteface will handle both cache and NativeQuantizationStateReader , to read it from cache , if not read from file , deseralize based on type of Quantization and save it Cache.

So Cache is clean , neither it has to know about type of Quantization State to deseralize and neither about how it was originally stored.

[navneet1v]

So Cache is clean , neither it has to know about type of Quantization State to deseralize and neither about how it was originally stored.

if are looking for making cache clean then the current implementation is not clean. It already has knowledge of QuantizationState object and other QS details.

But whether we want to make cache clean or not that is separate question. But it doesn't ans the question why this cache is not a Read-Through cache? You can just provide a loader functional interface here which is just loads the entry in cache if its not present. Then no-one needs to know about QS.

[navneet1v]

The problem with this kind of design is, we will have to write a wrapper on top of it which will first check if the entry in cache is present or not, and then load data in the cache. Which can be just avoided here.

[navneet1v]

any reason of making this function public?

[ryanbogan]

Eventually, it will be used when indices/the state reader are closed, which separates it from the rebuild() method.

[ryanbogan]

There's one flaky test I'm still trying to fix. (edit: should be fixed now)

[jmazanec15]

We need to serialize to get the weight? That seems too expensive

[ryanbogan]

As of right now, the only guaranteed methods to be in Quantization State are getting the params and toByteArray. This was the only way I could find to estimate the size with those methods

[jmazanec15]

@Vikasht34 we should enhance quantization state to give this estimate

[jmazanec15]

We can take in another review on this - but need to track.

[Vikasht34]

Let me take a look on this. Will address this in upcoming pr.