Iterative Vector Insertion #1840

MrFlap · 2024-07-16T23:54:42Z

Description

This change is an memory-based optimization of KNN index construction. Instead of loading all of the vectors into memory, then all into the index, we only load one (much smaller) batch at a time in order to avoid having copies of vector memory.

This also implements memory allocation on index initialization on HNSW to avoid vector resizing.

Support for creating index from template has not been added yet and will be added once this change is made to the main branch: #1784

Currently no planned support for nmslib.

Issues Resolved

#1600

Benchmarking

Here are some current benchmarks and metrics:

1 MB Vector Streaming Limit on SIFT dataset:

Memory usage:
Recall @K10: 0.983
Ingestion time: 908s

10 MB Vector Streaming Limit Memory Usage

Memory usage:
Recall @K10: 0.983
Ingestion time: 908s

100 MB Vector Streaming Limit Memory Usage

Memory usage:
Recall @K10: 0.983
Ingestion time: 909s

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

navneet1v · 2024-07-16T23:56:42Z

@MrFlap can you attach the benchmarking which you have done on the issue description

MrFlap · 2024-07-17T00:21:01Z

@MrFlap can you attach the benchmarking which you have done on the issue description

Will add more after OSB.

heemin32 · 2024-07-17T15:35:58Z

Anyone knows why tests haven't been executed for this PR?

junqiu-lei · 2024-07-17T15:38:16Z

Anyone knows why tests haven't been executed for this PR?

Might need to resolve conflicts firstly.

navneet1v · 2024-07-17T15:43:00Z

@heemin32 and @junqiu-lei the PR is more of an early look into the code. Since the tests can change based on the src files, the idea here is first we review the src and then same PR can be updated with tests. GH actions may keep on failing because there has been a lot of changes on going in the similar files.

navneet1v

Reviewed the JNI layer and some java files. KNN80DocValuesConsumer class is bit hacky. Lets try to modularize the code bit more to ensure that it is maintainable.

jni/cmake/init-faiss.cmake-e

jni/src/.idea/modules.xml

jni/include/faiss_index_service.h

jni/src/faiss_index_service.cpp

src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80DocValuesConsumer.java

src/main/java/org/opensearch/knn/index/codec/transfer/VectorTransferByte.java

src/main/java/org/opensearch/knn/index/codec/util/KNNCodecUtil.java

src/main/java/org/opensearch/knn/jni/JNIService.java

jni/src/.idea/modules.xml

src/main/java/org/opensearch/knn/index/codec/transfer/VectorTransferByte.java

src/main/java/org/opensearch/knn/index/codec/util/KNNCodecUtil.java

src/main/java/org/opensearch/knn/jni/FaissService.java

jmazanec15 · 2024-07-25T20:30:49Z

src/main/java/org/opensearch/knn/jni/FaissService.java

     */
-    public static native void createBinaryIndex(int[] ids, long vectorsAddress, int dim, String indexPath, Map<String, Object> parameters);
+    public static native void writeBinaryIndex(long indexAddress, String indexPath, int threadCount);


Do we need thread count for writing?

I don't think that read_index or write_index in faiss use OpenMP, but I could be wrong. If they don't, then threadCount isn't necessary.

@MrFlap so did we check if we need thread count for writing the index?

src/main/java/org/opensearch/knn/jni/FaissService.java

src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80DocValuesConsumer.java

jmazanec15 · 2024-07-25T20:35:08Z

src/main/java/org/opensearch/knn/index/codec/KNN80Codec/KNN80DocValuesConsumer.java

        Map<String, Object> parameters = new HashMap<>();
+        ;
+        if (fromScratch) {


Can we just have branch between fromScratch and not during index initialization? Then, everything else should proceed the same.

I don't know if I'm following. You mean have two different functions, one for fromScratch, one for fromTemplate?

jni/include/faiss_index_service.h

jni/include/faiss_wrapper.h

jni/src/faiss_index_service.cpp

src/test/java/org/opensearch/knn/jni/JNIServiceTests.java

src/test/java/org/opensearch/knn/index/memory/NativeMemoryLoadStrategyTests.java

src/main/java/org/opensearch/knn/jni/JNIService.java

heemin32 · 2024-07-26T04:28:08Z

Could you explain how to interpret the graph in benchmark? Like what does time mean and why there is a spike at the end? Also, it would be nice to have a memory usage graph of previous implementation(current state).

In addition to memory usage, I think indexing time comparison between new and old approach is also important. Can we have that data as well?

MrFlap · 2024-07-26T17:08:53Z

Could you explain how to interpret the graph in benchmark? Like what does time mean and why there is a spike at the end? Also, it would be nice to have a memory usage graph of previous implementation(current state).

Will add to the issue, I will also redo the benchmarks once the PR is completed

In addition to memory usage, I think indexing time comparison between new and old approach is also important. Can we have that data as well?

Yes. Will add. I am almost certain that indexing time is unaffected even for small batch sizes.

jni/src/faiss_index_service.cpp

jni/include/faiss_index_service.h

jni/include/org_opensearch_knn_jni_FaissService.h

shatejas

Seems to be significant change, does this need to be behind a feature flag?

.idea/copyright/SPDX_ALv2.xml

jni/include/faiss_index_service.h

jni/src/faiss_index_service.cpp

navneet1v · 2024-08-01T22:30:26Z

@MrFlap so did we check if we need thread count for writing the index?

I just checked, and we currently don't. I can remove it.

remove it then, if its not needed.

Signed-off-by: Andrew Klepchick <[email protected]>

* Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]>

* Iterative Vector Insertion (#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]>

* Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]>

* Iterative Vector Insertion (opensearch-project#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (opensearch-project#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]>

* Integrate KNNVectorValues with vector ANN Search flow (#1952) Signed-off-by: Navneet Verma <[email protected]> * BackPort Java Doc Fix with Code Improvements (#1959) * Quantization Framework Code Structure Improvement (#1967) * BackPort Java Doc Fix with Code Improvements Signed-off-by: VIKASH TIWARI <[email protected]> * Quantization Framework Code Structure Improvement Signed-off-by: VIKASH TIWARI <[email protected]> --------- Signed-off-by: VIKASH TIWARI <[email protected]> * Iterative index integration (#1956) * Iterative Vector Insertion (#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Navneet Verma <[email protected]> Signed-off-by: VIKASH TIWARI <[email protected]> Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Navneet Verma <[email protected]> Co-authored-by: Vikasht34 <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]>

* Iterative Vector Insertion (opensearch-project#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (opensearch-project#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]>

* Integrate KNNVectorValues with vector ANN Search flow (#1952) Signed-off-by: Navneet Verma <[email protected]> * Iterative index integration (#1956) * Iterative Vector Insertion (#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Navneet Verma <[email protected]> Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Navneet Verma <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]>

* Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]>

…at (#1950) * Iterative Vector Insertion (#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]>

…at (opensearch-project#1950) * Iterative Vector Insertion (opensearch-project#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (opensearch-project#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> (cherry picked from commit fd59b9a)

…at (#1950) (#1992) * Iterative Vector Insertion (#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> (cherry picked from commit fd59b9a) Signed-off-by: Tejas Shah <[email protected]>

…at (opensearch-project#1950) * Iterative Vector Insertion (opensearch-project#1840) * Rebased with new version of k-NN Signed-off-by: Andrew Klepchick <[email protected]> * Optimized faiss insertion Signed-off-by: Andrew Klepchick <[email protected]> * Optimized threadCount logic Signed-off-by: Andrew Klepchick <[email protected]> * Removed IDEA files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary cmake file Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to new functions Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex and fixed test cases that use it Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused code Signed-off-by: Andrew Klepchick <[email protected]> * Explained zero initialization for vector transfer Signed-off-by: Andrew Klepchick <[email protected]> * Added locale Signed-off-by: Andrew Klepchick <[email protected]> * Spotless Apply Signed-off-by: Andrew Klepchick <[email protected]> * Account for zero documents in finished batch Signed-off-by: Andrew Klepchick <[email protected]> * Changed where we check for zero docs Signed-off-by: Andrew Klepchick <[email protected]> * Changed tip for return Signed-off-by: Andrew Klepchick <[email protected]> * Use unique pointers to make sure resources are released on exception Signed-off-by: Andrew Klepchick <[email protected]> * Moved createIndex to testUtils Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management so that the underlying index is not deleted after initialized Signed-off-by: Andrew Klepchick <[email protected]> * Created new KNNIndexBuilder graph to make index building more modular Signed-off-by: Andrew Klepchick <[email protected]> * Streamlined logic in KNNIndexBuilder. Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up unnecessary code in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Fixed memory management process Signed-off-by: Andrew Klepchick <[email protected]> * Added note about index initialization in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for case where the exception happens after the indexWriter is released. Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/modules.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/vcs.xml Signed-off-by: Andrew Klepchick <[email protected]> * Delete jni/src/.idea/workspace.xml Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply and free iterative index on exception Signed-off-by: Andrew Klepchick <[email protected]> * Undid hack for checking first document metrics Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Free Vector Transfer on batch ingestion Signed-off-by: Andrew Klepchick <[email protected]> * Undid free Signed-off-by: Andrew Klepchick <[email protected]> * Fixed check for transfer ready Signed-off-by: Andrew Klepchick <[email protected]> * Don't crash when zero vectors inserted? Signed-off-by: Andrew Klepchick <[email protected]> * Reverted to old insertion process? Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added back createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Removed prior createOutput Signed-off-by: Andrew Klepchick <[email protected]> * Test remaking vectorTransfer Signed-off-by: Andrew Klepchick <[email protected]> * Test restructuring of insertion Signed-off-by: Andrew Klepchick <[email protected]> * Fixed case where vector address is immediately discarded Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Split Index Builder into multiple classes Signed-off-by: Andrew Klepchick <[email protected]> * Fixed descriptions of functions in faiss_index_service Signed-off-by: Andrew Klepchick <[email protected]> * Added back copyright files Signed-off-by: Andrew Klepchick <[email protected]> * Removed unused builder names Signed-off-by: Andrew Klepchick <[email protected]> * Modified tests to work with new insertion methods Signed-off-by: Andrew Klepchick <[email protected]> * Track index insertions Signed-off-by: Andrew Klepchick <[email protected]> * Tracked insertions for binary indices Signed-off-by: Andrew Klepchick <[email protected]> * Added back insertIds Signed-off-by: Andrew Klepchick <[email protected]> * Added check for freeVectorData to see if it works with an already deleted address Signed-off-by: Andrew Klepchick <[email protected]> * Cleaned up logs and comments in KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Restructured the logic for KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed package name of KNNIndexBuilder Signed-off-by: Andrew Klepchick <[email protected]> * Changed all package names and deleted unnecessary headers Signed-off-by: Andrew Klepchick <[email protected]> * Fixed for loop Signed-off-by: Andrew Klepchick <[email protected]> * Removed createIndex methods for faiss index service Signed-off-by: Andrew Klepchick <[email protected]> * Fixed package to fit naming conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed name of index builder Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Added comments to NativeIndexBuilder and restructured Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion for memoryAddress Signed-off-by: Andrew Klepchick <[email protected]> * Spotless apply Signed-off-by: Andrew Klepchick <[email protected]> * Changed naming of classes to Writer and changed package name to fit conventions Signed-off-by: Andrew Klepchick <[email protected]> * Changed NativeIndexInfo and NativeVectorInfo to follow builder pattern Signed-off-by: Andrew Klepchick <[email protected]> * Added feature to changelog Signed-off-by: Andrew Klepchick <[email protected]> * Added class descriptions to each NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Changed name to getBytesPerVector Signed-off-by: Andrew Klepchick <[email protected]> * Added == false instead of ! for readability Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming in docvaluesconsumer Signed-off-by: Andrew Klepchick <[email protected]> * SpotlessApply Signed-off-by: Andrew Klepchick <[email protected]> * Made it so that we don't reuse testValues and removed a foot gun Signed-off-by: Andrew Klepchick <[email protected]> * Removed another foot gun in getIndexInfo Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Added deletion on exception cases Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary delete (NativeIndexWriter will handle deletion of vectors on exception) Signed-off-by: Andrew Klepchick <[email protected]> * Added correct logger and getWriter method to NativeIndexWriter Signed-off-by: Andrew Klepchick <[email protected]> * Ensured memory safety on JNI layer so that Java doesn't have to wrap everything in a try catch loop. Signed-off-by: Andrew Klepchick <[email protected]> * Refactored NativeIndexWriter and added comments to FaissService Signed-off-by: Andrew Klepchick <[email protected]> * Removed free in the JNIExport since index will always be freed in writeIndex. Signed-off-by: Andrew Klepchick <[email protected]> * Changed getVectorTransfer back to accept VectorDataType Signed-off-by: Andrew Klepchick <[email protected]> * Reverted free since not guaranteed to be IDMap. Signed-off-by: Andrew Klepchick <[email protected]> * Added all processes in addKNNBinaryField to NativeIndexWriter.createKNNIndex Signed-off-by: Andrew Klepchick <[email protected]> * Fixed javadoc Signed-off-by: Andrew Klepchick <[email protected]> * Applied spotless Signed-off-by: Andrew Klepchick <[email protected]> * Added back writeFooter Signed-off-by: Andrew Klepchick <[email protected]> * Removed threadCount fron writeIndex Signed-off-by: Andrew Klepchick <[email protected]> * Removed redundancies in KNN80DocValuesConsumer Signed-off-by: Andrew Klepchick <[email protected]> * Removed serializationMode Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed double free test as we don't have to worry about that anymore Signed-off-by: Andrew Klepchick <[email protected]> * Accounted for HNSWSQ in index service Signed-off-by: Andrew Klepchick <[email protected]> * Removed delete in catch Signed-off-by: Andrew Klepchick <[email protected]> * Fixed faiss tests to work with writeIndex Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Index Initialization Alloc Method (opensearch-project#1933) * Added methods for allocating memory before inserting vectors to a faiss index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed logic that gets type of index Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statement Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary iostream Signed-off-by: Andrew Klepchick <[email protected]> * Removed flat index Signed-off-by: Andrew Klepchick <[email protected]> * Fixed flat index case Signed-off-by: Andrew Klepchick <[email protected]> * Fixed naming Signed-off-by: Andrew Klepchick <[email protected]> * Properly allocate HNSWSQ storage Signed-off-by: Andrew Klepchick <[email protected]> * Removed print statements Signed-off-by: Andrew Klepchick <[email protected]> * Fixed changelog Signed-off-by: Andrew Klepchick <[email protected]> * Removed unnecessary lib Signed-off-by: Andrew Klepchick <[email protected]> * Made alloc adaptive to different code sizes Signed-off-by: Andrew Klepchick <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> * Integrates FAISS iterative builds with NativeEngines990KnnVectorsFormat Changes include reusing the same vector buffer in the JNI layer Signed-off-by: Tejas Shah <[email protected]> --------- Signed-off-by: Andrew Klepchick <[email protected]> Signed-off-by: Tejas Shah <[email protected]> Co-authored-by: Andrew Klepchick <[email protected]> Signed-off-by: Akash Shankaran <[email protected]>

MrFlap requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, ryanbogan and luyuncheng as code owners July 16, 2024 23:54

MrFlap force-pushed the new_iter_graph_cons branch from 58f7dc7 to 79e0fe3 Compare July 17, 2024 00:30

MrFlap force-pushed the new_iter_graph_cons branch from b0ae412 to e7ccc9e Compare July 19, 2024 21:55

navneet1v reviewed Jul 20, 2024

View reviewed changes

MrFlap commented Jul 20, 2024

View reviewed changes

jni/src/.idea/modules.xml Outdated Show resolved Hide resolved

MrFlap force-pushed the new_iter_graph_cons branch 2 times, most recently from 961cf25 to 73fc9ab Compare July 25, 2024 07:41

jmazanec15 reviewed Jul 25, 2024

View reviewed changes

Vikasht34 reviewed Jul 25, 2024

View reviewed changes

heemin32 reviewed Jul 26, 2024

View reviewed changes

jni/include/faiss_index_service.h Outdated Show resolved Hide resolved

heemin32 reviewed Jul 26, 2024

View reviewed changes

jni/include/org_opensearch_knn_jni_FaissService.h Show resolved Hide resolved

shatejas reviewed Jul 28, 2024

View reviewed changes

Fixed changelog

37dd8cd

Signed-off-by: Andrew Klepchick <[email protected]>

navneet1v requested review from shatejas and navneet1v August 1, 2024 22:51

MrFlap added 5 commits August 1, 2024 17:20

Fixed changelog

016ac25

Signed-off-by: Andrew Klepchick <[email protected]>

Removed double free test as we don't have to worry about that anymore

29cc951

Signed-off-by: Andrew Klepchick <[email protected]>

Accounted for HNSWSQ in index service

fb43952

Signed-off-by: Andrew Klepchick <[email protected]>

Removed delete in catch

7894e40

Signed-off-by: Andrew Klepchick <[email protected]>

Fixed faiss tests to work with writeIndex

4121df7

Signed-off-by: Andrew Klepchick <[email protected]>

shatejas approved these changes Aug 3, 2024

View reviewed changes

navneet1v approved these changes Aug 3, 2024

View reviewed changes

navneet1v merged commit 36a4c0d into opensearch-project:feature/iterative-index-build Aug 3, 2024
51 checks passed

MrFlap mentioned this pull request Aug 7, 2024

[Enhancement] [Build-Time V1] [FEATURE] Native Index Memory Footprint Reduction during Indexing #1938

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Iterative Vector Insertion #1840

Iterative Vector Insertion #1840

MrFlap commented Jul 16, 2024 •

edited

Loading

navneet1v commented Jul 16, 2024

MrFlap commented Jul 17, 2024

heemin32 commented Jul 17, 2024

junqiu-lei commented Jul 17, 2024

navneet1v commented Jul 17, 2024

navneet1v left a comment

jmazanec15 Jul 25, 2024

MrFlap Jul 25, 2024

navneet1v Aug 1, 2024

jmazanec15 Jul 25, 2024

MrFlap Jul 25, 2024

heemin32 commented Jul 26, 2024

MrFlap commented Jul 26, 2024

shatejas left a comment

navneet1v commented Aug 1, 2024 •

edited

Loading

Iterative Vector Insertion #1840

Iterative Vector Insertion #1840

Conversation

MrFlap commented Jul 16, 2024 • edited Loading

Description

Issues Resolved

Benchmarking

1 MB Vector Streaming Limit on SIFT dataset:

10 MB Vector Streaming Limit Memory Usage

100 MB Vector Streaming Limit Memory Usage

Check List

navneet1v commented Jul 16, 2024

MrFlap commented Jul 17, 2024

heemin32 commented Jul 17, 2024

junqiu-lei commented Jul 17, 2024

navneet1v commented Jul 17, 2024

navneet1v left a comment

Choose a reason for hiding this comment

jmazanec15 Jul 25, 2024

Choose a reason for hiding this comment

MrFlap Jul 25, 2024

Choose a reason for hiding this comment

navneet1v Aug 1, 2024

Choose a reason for hiding this comment

jmazanec15 Jul 25, 2024

Choose a reason for hiding this comment

MrFlap Jul 25, 2024

Choose a reason for hiding this comment

heemin32 commented Jul 26, 2024

MrFlap commented Jul 26, 2024

shatejas left a comment

Choose a reason for hiding this comment

navneet1v commented Aug 1, 2024 • edited Loading

MrFlap commented Jul 16, 2024 •

edited

Loading

navneet1v commented Aug 1, 2024 •

edited

Loading