-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement][Build-TimeReduction V1] Remove conversion of KNN Field to string if stored Fields not required #1598
Comments
The fix for the issue is pretty simple rather than doing toString we just need to pass the VectorField to the underline function. |
Resolving the issue as the PR is merged. |
@navneet1v for the flame graph above, what was the setup? Trying to understand impact on ingestion |
I created a 2.14 cluster and did the ingestion. I removed few more things from indexing path like adding recovery_source and parsing of vector field as an array. But you don't need all these changes. Just pick up a 1536D dataset and run it with and without this change and I used a 1536D dataset with 5M records |
Description
While doing some benchmarks with Opensearch k-NN. I see that there is time going in to convert the kNN field to String even when storedFields is marked as false. On doing deep-dive I found out that, we convert the field to the string and then check if stored field is set as true or false.
Code ref:
https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java#L575
Here
point.toString
will convert the object to string which is expensive operation in high dimensionality use cases.Refer the Flame graph for the same for Vector dimension: 1536 ingestion
The text was updated successfully, but these errors were encountered: