OpenSearch 2.14.0 introduces a host of new and updated features designed to increase performance, improve usability, expand access to data sources, and help users build better search and machine learning (ML) applications.
- Updates to OpenSearch’s hybrid search functionality, which combines neural search with lexical search to provide higher-quality results than when using either technique alone, offer performance increases of up to 4x.
- Multi-range traversal adds performance improvements for date histogram queries without sub-aggregations. Improvements should be significant for time-series data analysis use cases, where date histograms are critical. This optimization also addresses a performance regression noted in the PMC benchmark for these queries.
- Expanded multiple data sources functionality is extended to nine external Dashboards plugins: Index Management, ML Commons, Search Relevance, Anomaly Detection, Maps, Security, Notifications, Trace Analytics, and Query Workbench. Multiple data sources are also enabled for two core plugins: Region Map and the time series visual builder, delivering further progress toward a unified user experience for OpenSearch Dashboards.
- API-native ingest joins the OpenSearch machine learning (ML) toolkit. This allows you to integrate any ML model and use models to enrich data streams through the Ingest API, so you can easily transition between model providers and build applications more quickly. A new ML inference processor allows you to perform inferences on any integrated ML model to enrich your pipeline.
- A new semantic cache for LangChain applications lets you use k-NN indexes to cache large language model (LLM) requests and responses. This semantic cache functionality can reduce the number of calls made to LLMs for similar requests and responses.
- A new lower-level vector query interface expands neural sparse functionality, allowing you to run a neural sparse query by providing a sparse vector, a list of weighted tokens, as inputs. This adds an optional interface for building semantic search applications when using semantic sparse encoders.
- An update to the k-NN query interface lets you retrieve only results within a certain maximum distance or vector score threshold. This is particularly well suited for uses cases in which the goal is to retrieve all the results that are highly or sufficiently similar; for example, >= 0.95.
- A search pipelines enhancement allows you to use a single search pipeline as a default for multiple indexes and aliases. This gives you the option to use search pipelines with the benefits of index aliases for improved reusability.
- A new k-NN Clear Cache API lets you clear k-NN indexes from the cache without the need to delete the index or manually set the k-NN cluster settings knn.cache.item.expiry.enabled and knn.cache.item.expiry.minutes.
- An update to the OpenSearch RPM package expands the scope of the configuration files to include all files within the /etc/opensearch directory, helping ensure that plugin-specific configurations are preserved during upgrades.
- Please note that a new PGP public key is available for artifact verification. OpenSearch’s current PGP public key is scheduled to expire on May 12, 2024. Please visit https://opensearch.org/verify-signatures.html to download the new public key, which is scheduled to expire on May 12, 2025.
OpenSearch 2.14.0 includes the following experimental features. Experimental features are disabled by default. For instructions on how to enable them, refer to the documentation for the feature.
- This release introduces experimental support for tiered caching within the request cache. This enables caching of much larger datasets, helping to avoid cache evictions and misses due to limited available memory on a node. The tiered cache framework consists of multiple levels, each with its own size and performance characteristics. Evicted items from upper, faster tiers, such as the on-heap cache, are moved to lower, slower tiers, like the disk cache, which affords greater storage capacity but higher latency.
- This release includes an experimental update to OpenSearch’s remote storage and document replication capabilities. This functionality will allow you to migrate existing indexes with document replication enabled to remote-backed storage clusters.
- Experimental support for cluster-level dynamic application configuration offers users a secure and flexible set of tools for controlling more cluster settings, such as Content Security Policy (CSP) rules, to help assure a good user experience while maintaining security standards.
OpenSearch and OpenSearch Dashboards 2.14.0 includes the following features, enhancements, bug fixes, infrastructure, documentation, maintenance and refactoring updates.
OpenSearch Release Notes.
OpenSearch Dashboards Release Notes.
- Add skipping index and saved queries for WAF integration (#1750)
- Add multi-checkpoint support to integrations (#1742)
- Enhance Explorer to use describe command (#1736)
- Update vpc flow with flint-s3 based DDL assets and dashboard (#1721)
- Add default refresh interval for all the integrations and correct the version on
main(#1717) - More example queries for S3-based integrations (#1714)
- Implement saved query substitution for S3 integrations (#1711)
- Update cloud trail integration with flint-s3 based DDL assets and das… (#1707)
- [Integrations] Add integration of S3 Access log (#1697)
- Update pattern for multiple mview suffixes (#1693)
- HAProxy Flint Integration (#1692)
- Add CloudFront queries for integrations and integration table bug fix (#1687)
- Add integration of WAF log (#1685)
- Improve on ELB integration assets (#1682)
- Add Flint queries for Apache Access integration (#1681)
- Add observability-search link rendering for integrations (#1642)
- Flint Datasource Cypress testing for tables (#1610)
- Update loading state for tables fields in create acceleration flyout (#1576)
- Add info callout for s3 datasources (#1575)
- Flint datasource 2.13 bug bash fix (#1574)
- Fixed small bugs in explorer (#1559)
- Initiate MLInferencelngestProcessor (#2205)
- Add TTL to un-deploy model automatically (#2365)
- ML Model Interface (#2357)
- Support k-NN radial search parameters in neural search (#697)
- Multi datasource support (#311)
- Tenancy access control (#992)
- Add latest sigma rules (#942)
- Fix filter fields, adding geo point and date_nanos (#285) (#286)
- Change ad plugin jar dependency (#288)
- Remove logic about replace quota for finetuning model (#289) (#291)
- Move search index tool to ml-commons repo (#297)
- Move visualization tool to ml-commons (#296) (#298)
- Add k-NN clear cache api (#740)
- Support radial search in k-NN plugin (#1617)
- Support filter and nested field in faiss engine radial search (#1652)
- Adding tracking_total_hits in search query for findings. (#1487)
- Support MDS on List, Detail, Dashboard, Overview pages (#722)
- Support for fetching changes from Lucene store during remote store migration (#1369)
- Added Multidata Source Support (#197)
- Moved enrichment of server features to server side (#181) (#194)
- Add guardrails to default use case params (#658)
- Allow strings for boolean workflow step parameters (#671)
- Add optional delay parameter to no-op step (#674)
- Add model interface support for remote and local custom models (#701)
- Add MDS support for policies, policy managed indices, rollup jobs and transform jobs (#1021)
- Interface change for MDS support and deprecating dataSourceLabel from the URL (#1031)
- Mount MDS on the empty route (#1039)
- Change httpclient to async (#1958)
- Migrate RAG pipeline to async processing. (#2345)
- Filtering hidden model info from model profiling for users other than superadmin (#2332)
- Check model auto deploy (#2288)
- Restrict stash context only for stop words system index (#2283)
- Add a flag to control auto-deploy behavior (#2276)
- BWC tests for text chunking processor (#661)
- Add support for request_cache flag in hybrid query (#663)
- Allowing execution of hybrid query on index alias with filters (#670)
- Allowing query by raw tokens in neural_sparse query (#693)
- Removed stream.findFirst implementation to use more native iteration implement to improve hybrid query latencies by 35% (#706)
- Removed map of subquery to subquery index in favor of storing index as part of disi wrapper to improve hybrid query latencies by 20% (#711)
- Avoid change max_chunk_limit exceed exception in text chunking processor (#717)
- Tenancy access control (#1821)
- Check for and perform upgrades on security configurations (#4251)
- Replace bouncy castle blake2b (#4284)
- Adds saml auth header to differentiate saml requests and prevents auto login as anonymous user when basic authentication fails (#4228)
- Dynamic sign in options (#4137)
- Add index permissions for query insights exporters (#4231)
- Add new stop words system index (#4181)
- Switch to built-in security transports from core (#4119) (#4174) (#4187)
- System index permission grants reading access to documents in the index (#4291)
- Improve cluster initialization reliability (#4002) (#4256)
- Adds Multiple Datasources Support for Security Dashboards Plugin (#1888)
- Adds flow-framework transport action permissions to the static dropdown list (#1908)
- Update copy for tenancy tab (#1881)
- Clear session storage on auto-logout & remove unused saml function (#1872)
- Create a password strength UI for security dashboards plugin (#1762)
- Make the HitQueue size more appropriate for exact search (#1549)
- Implement the Streaming Feature to stream vectors from Java to JNI layer to enable creation of larger segments for vector indices (#1604)
- Remove unnecessary toString conversion of vector field and added some minor optimization in KNNCodec (1613)
- Serialize all models into cluster metadata (#1499)
- Update live mv table name (#1725)
- Changes loading configuration for Explorer default (#1720)
- Remove auto refresh option in create acceleration flyout (#1716)
- Update mv name to include double "_" (#1712)
- Bugfix: Use workflows option in selection (#1704)
- Fix small issues within explorer search bar and sample query (#1683)
- (query assist) Update styles for callout and combo box (#1675)
- Convert cache to session storage (#1669)
- Correctly Utilize Cache in Tables Flyout (#1662)
- Bug Fix for Undefined Association (#1658)
- Prevent logged out datasources call (#1653)
- Update intercept to check logout request (#1650)
- Fix integration flyout successes (#1647)
- Clear callout in query assist (#1646)
- [BUGFIX] Fix integration data reading double escape (#1644)
- Clear cache on any 401 response errors (#1634)
- Updating catch for guardrails (#1631)
- Updating snapshot to fix build (#1627)
- Bug fixes and UI updates for datasources (#1618)
- Create acceleration flyout bug fixes (#1617)
- Fix DSL router, update UI for query assist (#1612)
- Fixed bugs in explorer redirection (#1609)
- Sanitize create acceleration queries and direct queries (#1605)
- Fix create acceleration bugs (#1599)
- Updating usePolling to cleanup after unmount (#1598)
- Enable integration install flyout for other install buttons (#1596)
- Disable close button when integration is being installed (#1591)
- Fix fieldLimit exception in docLevelMonitor. (#1368)
- Include server.basepath config in the manage notifications channel Url. (#914)
- Removed cross cluster monitor experimental banner, and fixed bugs. (#933)
- Fetching timezone from ui settings for Trigger context formatting (#913)
- Populate selected indices from query params on initial load (#713)
- Build query parameters using data_end_time (#741)
- Fix number of total shards in partial search response (#565)
- Handle response for deletion of non-existent autofollow replication rule (#1371)
- Fix zoom level type error in custom layer (#605)
- Handle error state when dataSource switches to invalid dataSource (#199)
- Fix broken osd functional test repo (#189)(#190)
- Reset workflow state to initial state after successful deprovision (#635)
- Silently ignore content on APIs that don't require it (#639)
- Hide user and credential field from search response (#680)
- Throw the correct error message in status API for WorkflowSteps (#676)
- Delete workflow state when template is deleted and no resources exist (#689)
- Fixing model group parsing and restoring context (#695)
- Set ActiveOption prop to undefined on first load (#1042)
- Readonly DataSourceMenu in create rollup and create transform workflow (#1047)
- Fix Transform job create flow where indices won't reset after change of datasource (#1053)
- Fix stopwords npe (#2311)
- Guardrails npe (#2304)
- Not sending failure message when model index isn't present (#2351)
- Fix guardrails mapping (#2279)
- Fix no model group index issue in connector helper notebook (#2336)
- Fixes #2317 predict api not working with asymmetric models (#2318)
- Fixing isHidden null issue (#2337)
- Fix remote register model / circuit breaker 500(#2264)
- Guardrails bug fixes and IT for creating guardrails (#2269)
- Added missing result filter to inference (#2367)
- Making Boolean type for isHidden (#2341
- Clear planningWorkerNodes when model auto-deploys again after undeploy (#2396)
- Avoid race condition in syncup model state refresh (#2405)
- Add a flag to distinguish remote model auto deploy and transport deploy (#2410)
- Add deploySetting in registering local models (#2415)
- Reset current page after data source change (#320)
- Reset to max page when current page overflow (#323)
- Fix async actions are left in neural_sparse query (#438)
- Fix typo for sparse encoding processor factory(#578)
- Add non-null check for queryBuilder in NeuralQueryEnricherProcessor (#615)
- Add max_token_score field placeholder in NeuralSparseQueryBuilder to fix the rolling-upgrade from 2.x nodes bwc tests. (#696)
- Fix multi node "no such index" error in text chunking processor. (#713)
- Adding max http response string length as a setting, and capping http response string based on that setting (#876)
- Adding Max HTTP Response Size IT (#901) (#909)
- Replacing spi.utils isHostInDenyList with core.utils isHostInDenyList (#904) (#913)
- Upgrade AWS version for SDKs to 1.12.687 (#884) (#887)
- Fix initial load from cache for S3 tree (#300)
- Ensure that challenge response contains body (#4268)
- Add logging for audit log that are unable to saving the request body (#4272)
- Use predictable serialization logic for transport headers (#4288)
- Update Log4JSink Default from sgaudit to audit and add test for default values (#4155)
- Remove Pom task dependencies rewrite (#4178) (#4186)
- Misc changes for tests (#4184)
- Add simple roles mapping integ test to test mapping of backend role to role (#4176)
- Fix integ tests after add latest sigma rules (#950)
- Fix keywords bug and add comments (#964)
- Changes doc level query name field from id to rule name and adds validation (#972)
- Fix check for agg rules in detector trigger condition to create chained findings monitor (#992)
- Wait longer for action button to become enabled (#983)
- Fixes saml login flow to work with anonymous auth (#1839)
- Fixes issue with multi-tenancy and default route to use corresponding default route for the selected tenant (#1820)
- Fix oidc cypress test and remove doc link (#1923)
- Add fix for data source disabled plugin should work (#1916)
- Add stored fields for knn_vector type (#1630)
- Enable script score to work with model based indices (#1649)
- Align vacuum statement semantics with Flint Spark (#2606)
- Handle EMRS exception as 400 (#2612)
- Fix pagination for many columns (#2440) (#2441)
- Fix semicolon parsing for async query (#2631)
- Throw OpensearchSecurityException in case of datasource authorization (#2626)
- Add workflow to build and install binary to catch run time issues (#1467)
- Adjusted maven publish workflow to execute automatically when merging a PR. (#1531)
- Update sample cert and admin keystore (#1163)
- Add workflow to verify binary installation works (#693)
- Switch macOS runner to macos-13 from macos-latest since macos-latest is now arm64 (#686)
- Remove checkstyle (#2312)
- Increase rounding delta from 0.005% to 0.5% on RestMLInferenceIngestProcessorIT (#2372)
- Add agent framework security it tests by (#2266)
- Add IT for interface (#2394)
- Fix local build failure for RestMLInferenceIngestProcessorIT (#2402)
- Adding integration tests for scenario of hybrid query with aggregations (#632)
- Add micro-benchmark module in k-NN plugin for benchmark streaming vectors to JNI layer functionality. (#1583)
- Add arm64 check when SIMD is disabled (#1618)
- Skip rebuild from scratch after cmake is run (#1636)
- Increment version to 2.14.0-SNAPSHOT (#2585)
- Add basic developer docs for integration setup and config (#1613)
- Change query access messaging (#1224)
- Dev guide update to include building/using local os-min distro. (#757)
- Added 2.14 release notes. (#1534)
- Added 2.14 release notes. (#945)
- Added 2.14.0.0 release notes. (#648)
- 2.14 release notes. (#201)
- Add connector blueprint for VertexAI Embedding endpoint (#2268)
- Add 2.14.0 release notes (#915)
- Added 2.14.0 release notes. (#1009)
- Added release notes for 2.14.0 (#997)
- Increment version to 2.14.0.0 (#183)
- Refactor integrations setup for easier separation of different setup options (#1741)
- Reformatting integration queries (#1726)
- Increment version to 2.14.0.0 (#1673)
- Enable query assist by default (#1640)
- Update ag-grid dependency to 31 (#1604)
- Fix datagrid snapshots for 2.x (#1590)
- Increment version to 2.14.0-SNAPSHOT. (#1492)
- Increment version to 2.14.0.0 (#931)
- Increment version to 2.14.0 (#549)
- Increment version to 2.14.0-SNAPSHOT (#625)
- Add workflow to verify installation of the plugin zip and dashboards (#165)
- Add AWSHurneyt as maintainer (#177)
- Increment version to 2.14.0.0 (#182)
- Increment version to 2.14.0.0 (#338)
- Increment version to 2.14.0 (#605)
- Dependabot: bump com.google.googlejavaformat:google-java-format (#608)
- Dependabot: bump org.slf4j:slf4j-api from 2.0.12 to 2.0.13 (#611) (#614)
- Dependabot: bump org.gradle.test-retry from 1.5.8 to 1.5.9 (#618) (#619)
- Dependabot: bump com.netflix.nebula.ospackage from 11.8.1 to 11.9.0 (#617) (#620)
- Fix CVE for org.eclipse.core.runtime (#2378)
- Increment version to 2.14.0.0 (#313)
- Update bwc tests for neural_query_enricher neural_sparse search (#652)
- Increment version to 2.14.0-SNAPSHOT (#882)
- Updates sample cert and admin keystore (#862) (#885)
- Updates sample cert and trust-store (#899) (#912)
- Increment version to 2.14.0-SNAPSHOT (#1813)
- Add getProperty.org.bouncycastle.ec.max_f2m_field_size to plugin-security.policy (#4270)
- Add getProperty.org.bouncycastle.pkcs12.default to plugin-security.policy (#4266)
- Bump apache_cxf_version from 4.0.3 to 4.0.4 (#4287)
- Bump ch.qos.logback:logback-classic from 1.5.3 to 1.5.5 (#4248)
- Bump codecov/codecov-action from v3 to v4 (#4237)
- Bump com.fasterxml.woodstox:woodstox-core from 6.6.1 to 6.6.2 (#4195)
- Bump com.google.googlejavaformat:google-java-format from 1.21.0 to 1.22.0 (#4220)
- Bump commons-io:commons-io from 2.15.1 to 2.16.1 (#4196) (#4246)
- Bump com.nulab-inc:zxcvbn from 1.8.2 to 1.9.0 (#4219)
- Bump io.dropwizard.metrics:metrics-core from 4.2.15 to 4.2.25 (#4193) (#4197)
- Bump net.shibboleth.utilities:java-support from 8.4.1 to 8.4.2 (#4245)
- Bump spring_version from 5.3.33 to 5.3.34 (#4250)
- Bump Wandalen/wretry.action from 1.4.10 to 3.3.0 (#4167) (#4198) (#4221) (#4247)
- Bump open_saml_version from 4.3.0 to 4.3.2 (#4303) (#4239)
- Add husky pre commit and lint files (#1859)
- Remove implicit dependency to admin as password (#1855)
- Bump jose from 4.11.2 to 5.2.4 (#1902)
- Increment byte-buddy version to 1.14.9 (#288)
- Refactoring of SparkQueryDispatcher (#2615)
- Removed log entry regarding destination migration. (#1356)
- Set the cancelAfterTimeInterval parameter on SearchRequest object in all MonitorRunners. (#1366)
- Add validation check for doc level query name during monitor creation. (#1506)
- Added input validation, and fixed bug for cross cluster monitors. (#1510)
- Doc-level monitor fan-out approach (#1521)
- Obfuscate ip addresses in alert error message (#511)
- Change doc level query name validation (#630)
- Added validation for the new clusters field. (#633)
- Wrapped URI syntax exception in IllegalArgument exception. (#645)
- Improve error messages for workflow states other than NOT_STARTED (#642)
- Allow detectors to be stopped if underlying workflow is deleted. Don't allow them to then be started/edited (#810)
- Update vega-lite specs with theme based colors (#978)