Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework schema initialization for better performance #1078

Merged
merged 1 commit into from
Sep 18, 2024

Conversation

bryanlb
Copy link
Contributor

@bryanlb bryanlb commented Sep 18, 2024

Summary

Reworks the schema initialization for much better performance on load. Internal benchmarks shows this has reduced the cache node schema initialization from ~5s on average for 2k fields down to ~ 15ms for the same amount of fields.

This has the added benefit of preventing field conflicts from occurring on the cache nodes, as you first establish the entire nested schema before attempting to merge it into the mapper service.

This also does related work of moving significant amounts of OpenSearch adapter code into static initialization where possible to further improve performance loading a large amount of chunks. This work was split out of #1052.

@bryanlb bryanlb force-pushed the bburkholder/assorted-enhancements branch 2 times, most recently from 216ae38 to 8bea39c Compare September 18, 2024 17:15
@bryanlb bryanlb force-pushed the bburkholder/assorted-enhancements branch from 8bea39c to 8222368 Compare September 18, 2024 17:16
@bryanlb bryanlb marked this pull request as ready for review September 18, 2024 17:50
@bryanlb bryanlb mentioned this pull request Sep 18, 2024
2 tasks
@bryanlb bryanlb merged commit 38b0a07 into master Sep 18, 2024
2 checks passed
@bryanlb bryanlb deleted the bburkholder/assorted-enhancements branch September 18, 2024 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants