From 6397916bc817946a5210e082cc43e9ea068fb4ee Mon Sep 17 00:00:00 2001 From: Sagar <99425694+sgup432@users.noreply.github.com> Date: Wed, 8 May 2024 12:41:43 -0700 Subject: [PATCH] Add cache plugin and tiered cache documentation (#7052) * Add cache plugin and tiered cache documentation Signed-off-by: Sagar Upadhyaya * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update index.md Signed-off-by: Sagar Upadhyaya * Update concepts.md (#7049) Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update alert-finding-api.md (#7036) Signed-off-by: Heather Halter Signed-off-by: Sagar Upadhyaya * [DOC] Add urldecode processor documentation (#5994) * Add urldecode processor documentation Signed-off-by: Melissa Vagi * Add text and examples Signed-off-by: Melissa Vagi * Update urldecode.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/urldecode.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update urldecode.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --------- Signed-off-by: Melissa Vagi Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * clarify "contexts" (#7063) Signed-off-by: Stavros Macrakis <134456002+smacrakis@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Add documentation for indices.replication.max_bytes_per_sec (#7048) * Add documentation for indices.replication.max_bytes_per_sec Signed-off-by: Marc Handalian * Update _install-and-configure/configuring-opensearch/index-settings.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Marc Handalian Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update concurrent search docs with composite aggs updates (#7043) Signed-off-by: Jay Deng Signed-off-by: Sagar Upadhyaya * add query_by_tokens option in Neural Sparse Search (#7040) * query tokens Signed-off-by: zhichao-aws * fix typo Signed-off-by: zhichao-aws * typo Signed-off-by: zhichao-aws * fix Signed-off-by: zhichao-aws * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Update _query-dsl/specialized/neural-sparse.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Update _search-plugins/neural-sparse-search.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Update _search-plugins/neural-sparse-search.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: zhichao-aws * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: zhichao-aws Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Update bundled JDK version (April 2024 Patch releases) (#7031) Signed-off-by: Andriy Redko Signed-off-by: Sagar Upadhyaya * Updating documentation for ignore_unavailable search request parameter (#6987) * Updating documentation for ignore_unavailable search request parameter Signed-off-by: Ankit Jain * Update _api-reference/search.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Ankit Jain * Update _api-reference/document-apis/delete-by-query.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Ankit Jain * Update _api-reference/document-apis/update-by-query.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Ankit Jain --------- Signed-off-by: Ankit Jain Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Add documentation for hardware-accelerated compression codecs. (#6841) * Add documentation for hardware-accelerated compression codecs. Signed-off-by: Mulugeta Mammo * Fix based on the review from reviewdog. Signed-off-by: Mulugeta Mammo * Use the registered symbol for Intel and Xeon. Signed-off-by: mulugetam * Minor modifications to the text. Signed-off-by: mulugetam * Rephrasing. Signed-off-by: mulugetam * Paraphrasing. Signed-off-by: mulugetam * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _im-plugin/index-codecs.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update _install-and-configure/configuring-opensearch/index-settings.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Mulugeta Mammo Signed-off-by: mulugetam Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Correct the release version for the upgrade API feature (#6955) Signed-off-by: Peter Nied Signed-off-by: Sagar Upadhyaya * [DOC] Add user agent processor documentation (#5995) * Add user agent processor documentation Signed-off-by: Melissa Vagi * Add user agent processor documentation Signed-off-by: Melissa Vagi * Add examples Signed-off-by: Melissa Vagi * Add user agent processor documentation Signed-off-by: Melissa Vagi * Add user agent processor documentation Signed-off-by: Melissa Vagi * Add examples Signed-off-by: Melissa Vagi * Add examples and text Signed-off-by: Melissa Vagi * Update user-agent.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update user-agent.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/user-agent.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/user-agent.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/user-agent.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/user-agent.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/user-agent.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ingest-pipelines/processors/user-agent.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update user-agent.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi --------- Signed-off-by: Melissa Vagi Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Add understanding results page (#6984) * Add refresh API. Signed-off-by: Archer * Add Understanding Results section Signed-off-by: Archer * Additional feedback. Signed-off-by: Archer * Rework important metrics section. Signed-off-by: Archer * Add summary report breakdown Signed-off-by: Archer * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Delete _api-reference/index-apis/refresh.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update understanding-results.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Heather Halter Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Archer Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Heather Halter Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Add cluster setting for filter rewrite optimization in aggregation (#7033) * Add cluster setting for filter rewrite optimization in aggregation Signed-off-by: bowenlan-amzn * Update search-settings.md Signed-off-by: Melissa Vagi --------- Signed-off-by: bowenlan-amzn Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Signed-off-by: Sagar Upadhyaya * remove-has-childreren (#7072) Signed-off-by: Heather Halter Signed-off-by: Sagar Upadhyaya * Update csp configuration instructions for OSD (#7026) * Update docs about configuring frame-ancestors Signed-off-by: Tianle Huang * update fgac info Signed-off-by: Tianle Huang * add fgac info Signed-off-by: Tianle Huang * fix vale Signed-off-by: Tianle Huang * fix readme Signed-off-by: Tianle Huang * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Melissa Vagi Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Melissa Vagi Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Melissa Vagi Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Melissa Vagi Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Melissa Vagi Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update csp-dynamic-configuration.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> * Update _dashboards/csp/csp-dynamic-configuration.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _dashboards/csp/csp-dynamic-configuration.md Signed-off-by: Melissa Vagi --------- Signed-off-by: Tianle Huang Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * adding do_not_fail_on_forbidden section to docs #4896 (#6958) * adding do_not_fail_on_forbidden section to docs #4896 Signed-off-by: AntonEliatra * Update _security/access-control/permissions.md Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Melissa Vagi * Update permissions.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Signed-off-by: Melissa Vagi * Update permissions.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update permissions.md Signed-off-by: Melissa Vagi Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _security/access-control/permissions.md Signed-off-by: Melissa Vagi --------- Signed-off-by: AntonEliatra Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * feat: fix overlap rate param (#7045) Signed-off-by: Ian Menendez Signed-off-by: Sagar Upadhyaya * adding kibana_server role specification and explanation #4094 (#7066) * adding kibana_user role specification and explanation #4094 Signed-off-by: AntonEliatra * Apply suggestions from code review Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower Signed-off-by: AntonEliatra * Update multi-tenancy-config.md Signed-off-by: AntonEliatra * Apply suggestions from code review Co-authored-by: Melissa Vagi Signed-off-by: AntonEliatra * Update multi-tenancy-config.md Signed-off-by: AntonEliatra --------- Signed-off-by: AntonEliatra Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * expanding on TrustStore and KeyStore #4578 #4060 (#7015) * expanding on TrustStore and KeyStore #4578 #4060 Signed-off-by: AntonEliatra * expanding on TrustStore and KeyStore #4578 #4060 Signed-off-by: AntonEliatra * Update generate-certificates.md Signed-off-by: AntonEliatra * Update opensearch-keystore.md Signed-off-by: AntonEliatra * Apply suggestions from code review Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: AntonEliatra * Update security-admin.md Signed-off-by: AntonEliatra * Apply suggestions from code review Co-authored-by: Melissa Vagi Signed-off-by: AntonEliatra * Apply suggestions from code review Co-authored-by: Melissa Vagi Signed-off-by: AntonEliatra * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: AntonEliatra --------- Signed-off-by: AntonEliatra Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Add documentation for primary rebalancing (#7059) * Add documentation for primary rebalacing Signed-off-by: Arpit Bandejiya * Update _install-and-configure/configuring-opensearch/cluster-settings.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Arpit Bandejiya Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Update stats.md (#7087) Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * navbar yml update. (#7091) * Omitting the named anchor from header Download link. Signed-off-by: Nathan Boot * Updates navbar yml. --------- Signed-off-by: Nathan Boot Co-authored-by: Heather Halter Signed-off-by: Sagar Upadhyaya * Missed a few includes that are referred to by the navbar. (#7093) * Omitting the named anchor from header Download link. Signed-off-by: Nathan Boot * Updates navbar yml. * Navbar sass and html includes required for navbar. --------- Signed-off-by: Nathan Boot Co-authored-by: Heather Halter Signed-off-by: Sagar Upadhyaya * Add missing OPENSEARCH_INITIAL_ADMIN_PASSWORD for both apt/deb and yum/rpm (#7079) * Add missing OPENSEARCH_INITIAL_ADMIN_PASSWORD for apt install on debian.md This page's install command when installing via rep package is missing `OPENSEARCH_INITIAL_ADMIN_PASSWORD` Signed-off-by: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> * More update on deb and rpm installation steps on 2.12 security changes Signed-off-by: Peter Zhu * Move 2.12 top Signed-off-by: Peter Zhu * fix formatting Signed-off-by: Peter Zhu * More tweaks Signed-off-by: Peter Zhu * More tweaks 2 Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Add more things Signed-off-by: Peter Zhu * Add more things Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu --------- Signed-off-by: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> Signed-off-by: Peter Zhu Co-authored-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Sagar Upadhyaya * Update tiered-cache.md Signed-off-by: Sagar Upadhyaya * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/index.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/index.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/index.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/index.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/index.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Update _search-plugins/caching/tiered-cache.md Co-authored-by: Nathan Bower Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: Sagar Upadhyaya * Addressing comments Signed-off-by: Sagar Upadhyaya * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Sagar Upadhyaya Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Heather Halter Signed-off-by: Melissa Vagi Signed-off-by: Stavros Macrakis <134456002+smacrakis@users.noreply.github.com> Signed-off-by: Marc Handalian Signed-off-by: Jay Deng Signed-off-by: zhichao-aws Signed-off-by: Andriy Redko Signed-off-by: Ankit Jain Signed-off-by: Mulugeta Mammo Signed-off-by: mulugetam Signed-off-by: Peter Nied Signed-off-by: Archer Signed-off-by: bowenlan-amzn Signed-off-by: Heather Halter Signed-off-by: Tianle Huang Signed-off-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> Signed-off-by: AntonEliatra Signed-off-by: Ian Menendez Signed-off-by: Arpit Bandejiya Signed-off-by: Nathan Boot Signed-off-by: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> Signed-off-by: Peter Zhu Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Heather Halter Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower Co-authored-by: Stavros Macrakis <134456002+smacrakis@users.noreply.github.com> Co-authored-by: Marc Handalian Co-authored-by: Jay Deng Co-authored-by: zhichao-aws Co-authored-by: Andriy Redko Co-authored-by: Ankit Jain Co-authored-by: mulugetam Co-authored-by: Peter Nied Co-authored-by: bowenlan-amzn Co-authored-by: Tianle Huang <60111637+tianleh@users.noreply.github.com> Co-authored-by: AntonEliatra Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Co-authored-by: Ian Menendez <61611304+IanMenendez@users.noreply.github.com> Co-authored-by: Arpit-Bandejiya Co-authored-by: Nate B <96254688+nateynateynate@users.noreply.github.com> Co-authored-by: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> Co-authored-by: Peter Zhu --- _search-plugins/caching/index.md | 37 ++++++++++ _search-plugins/caching/tiered-cache.md | 92 +++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 _search-plugins/caching/index.md create mode 100644 _search-plugins/caching/tiered-cache.md diff --git a/_search-plugins/caching/index.md b/_search-plugins/caching/index.md new file mode 100644 index 0000000000..d155caf86b --- /dev/null +++ b/_search-plugins/caching/index.md @@ -0,0 +1,37 @@ +--- +layout: default +title: Caching +parent: Improving search performance +has_children: true +nav_order: 100 +--- + +# Caching + +OpenSearch relies on different on-heap cache types to accelerate data retrieval, providing significant improvement in search latency. However, cache size is limited by the amount of memory available on a node. When processing a larger dataset that can potentially be cached, the cache size limit can result in many pieces of data either being removed from the cache or not being cached, causing an incomplete query. This impacts performance because OpenSearch needs to process the query again, causing high resource consumption. + +Understanding how your data uses the cache can help improve your cluster's performance and prevent you from using too much memory, reducing the cost of querying your data. + +## Supported on-heap cache types + +OpenSearch supports the following on-heap cache types: + +- **Request cache**: Caches the local results on each shard. This allows frequently used and potentially resource-heavy search requests to return results almost instantaneously. +- **Query cache**: Caches common data from similar queries at the shard level. The query cache is more granular than the request cache and can cache data to be reused in different queries. +- **Field data cache**: Caches field data and global ordinals, which are both used to support aggregations on certain field types. + +## Additional cache stores + +**Introduced 2.14** +{: .label .label-purple } + +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/10024). +{: .warning} + +In addition to existing custom OpenSearch on-heap cache stores, cache plugins provide the following cache stores: + +- **Disk cache**: Stores the precomputed result of a query on disk. Use a disk cache to cache much larger datasets, provided that the disk's latency is within an acceptable range. +- **Tiered cache**: A multi-level cache in which each tier has its own characteristics and performance levels. For example, a tiered cache can contain both on-heap and disk tiers. By combining different tiers, you can achieve a balance between cache performance and size. To learn more, see [Tiered cache]({{site.url}}{{site.baseurl}}/search-plugins/caching/tiered-cache/). + +In OpenSearch 2.14, the request cache is integrated with cache plugins. You can use a tiered or disk cache as a request-level cache. +{: .note} diff --git a/_search-plugins/caching/tiered-cache.md b/_search-plugins/caching/tiered-cache.md new file mode 100644 index 0000000000..10cdeacab7 --- /dev/null +++ b/_search-plugins/caching/tiered-cache.md @@ -0,0 +1,92 @@ +--- +layout: default +title: Tiered cache +parent: Caching +grand_parent: Improving search performance +nav_order: 10 +--- + +# Tiered cache + +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/10024). +{: .warning} + +A tiered cache is a multi-level cache in which each tier has its own characteristics and performance levels. By combining different tiers, you can achieve a balance between cache performance and size. + +## Types of tiered caches + +OpenSearch provides an implementation of a `_tiered` spillover `cache_`. This implementation spills any items removed from the upper tiers to the lower tiers of cache. The upper tier, such as the on-heap tier, is smaller in size but offers better latency. The lower tier, such as the disk cache, is larger in size but slower in terms of latency. OpenSearch offers both on-heap and disk tiers. + +## Enabling a tiered cache + +To enable a tiered cache, configure the following setting in `opensearch.yml`: + +```yaml +opensearch.experimental.feature.pluggable.caching.enabled: true +``` +{% include copy.html %} + +For more information about ways to enable experimental features, see [Experimental feature flags]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/). + +## Installing required plugins + +To use tiered caching, install a tiered cache plugin. As of OpenSearch 2.13, the only available cache plugin is the `cache-ehcache` plugin. This plugin provides a disk cache implementation that can be used as a disk tier within a tiered cache. For more information about installing non-bundled plugins, see [Additional plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/#additional-plugins). + +A tiered cache will fail to initialize if the `cache-ehcache` plugin is not installed or if disk cache properties are not set. +{: .warning} + +## Tiered cache settings + +In OpenSearch 2.14, a request cache can be used in a tiered cache. To begin, configure the following settings in the `opensearch.yml` file. + +### Cache store name + +To use the OpenSearch-provided tiered spillover cache implementation, set the cache store name to `tiered_spillover`, as shown in the following example: + +```yaml +indices.request.cache.store.name: tiered_spillover: true +``` +{% include copy.html %} + +### Setting on-heap and disk store tiers + +Set the on-heap and disk store tiers to `opensearch_onheap` and `ehcache_disk`, as shown in the following example: + +```yaml +indices.request.cache.tiered_spillover.onheap.store.name: opensearch_onheap +indices.request.cache.tiered_spillover.disk.store.name: ehcache_disk +``` +The `opensearch_onheap` setting uses the built-in on-heap cache available in OpenSearch. + +The `ehcache_disk` setting is the disk cache implementation from [Ehcache](https://www.ehcache.org/) and requires installing the `cache-ehcache` plugin. + +{% include copy.html %} + +### Configuring on-heap and disk stores + +The following table lists the cache store settings for the `opensearch_onheap` store. + +Setting | Data type | Default | Description +:--- | :--- | :--- | :--- +`indices.request.cache.opensearch_onheap.size` | Percentage | 1% of the heap size | The size of the on-heap cache. Optional. +`indices.request.cache.opensearch_onheap.expire` | Time unit | `MAX_VALUE` (disabled) | Specifies a time-to-live (TTL) for the cached results. Optional. + +The following table lists the disk cache store settings for the `ehcache_disk` store. + +Setting | Data type | Default | Description +:--- | :--- | :--- | :--- +`indices.request.cache.ehcache_disk.max_size_in_bytes` | Long | `1073741824` (1 GB) | Defines the size of the disk cache. Optional. +`indices.request.cache.ehcache_disk.storage.path` | String | `""` | Defines the storage path for the disk cache. Required. +`indices.request.cache.ehcache_disk.expire_after_access` | Time unit | `MAX_VALUE` (disabled) | Specifies a TTL for the cached results. Optional. +`indices.request.cache.ehcache_disk.alias` | String | `ehcacheDiskCache#INDICES_REQUEST_CACHE` | Specifies an alias for the disk cache. Optional. +`indices.request.cache.ehcache_disk.segments` | Integer | `16` | Defines the number of segments into which the disk cache is separated. Used for concurrency. Optional. +`indices.request.cache.ehcache_disk.concurrency` | Integer | `1` | Defines the number of distinct write queues created for the disk store, where a group of segments shares a write queue. Optional. + +### Additional settings for the `tiered_spillover` store + +The following table lists additional settings for the `tiered_spillover` store setting. + +Setting | Data type | Default | Description +:--- | :--- | :--- | :--- +`indices.request.cache.tiered_spillover.disk.store.policies.took_time.threshold` | Time unit | `10ms` | A policy used to determine whether to cache a query into a disk cache based on its took time. This is a dynamic setting. Optional. +`indices.request.cache.tiered_spillover.disk.store.enabled` | Boolean | `True` | Enables or disables the disk cache dynamically within a tiered spillover cache. Note: After disabling a disk cache, entries are not removed automatically and requires the cache to be manually cleared. Optional.