diff --git a/breaking-changes.md b/_about/breaking-changes.md
similarity index 98%
rename from breaking-changes.md
rename to _about/breaking-changes.md
index 627547a8f4..46a162e396 100644
--- a/breaking-changes.md
+++ b/_about/breaking-changes.md
@@ -2,7 +2,6 @@
layout: default
title: Breaking changes
nav_order: 5
-parent: OpenSearch documentation
permalink: /breaking-changes/
---
diff --git a/_about/index.md b/_about/index.md
new file mode 100644
index 0000000000..a05387a15a
--- /dev/null
+++ b/_about/index.md
@@ -0,0 +1,82 @@
+---
+layout: default
+title: Getting started
+nav_order: 1
+has_children: false
+has_toc: false
+permalink: /about/
+redirect_from:
+ - /docs/opensearch/
+ - /opensearch/
+ - /opensearch/index/
+---
+
+{%- comment -%}The `/docs/opensearch/` redirect is specifically to support the UI links in OpenSearch Dashboards 1.0.0.{%- endcomment -%}
+
+# OpenSearch and OpenSearch Dashboards
+
+This section contains documentation for OpenSearch and OpenSearch Dashboards.
+
+## Getting started
+
+- [Intro to OpenSearch]({{site.url}}{{site.baseurl}}/intro/)
+- [Quickstart]({{site.url}}{{site.baseurl}}/quickstart/)
+- [Install OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/install-opensearch/index/)
+- [Install OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/install-and-configure/install-dashboards/index/)
+- [See the FAQ](https://opensearch.org/faq)
+
+## Why use OpenSearch?
+
+With OpenSearch, you can perform the following use cases:
+
+
+
+
+
+
+
+
+
+
+
Fast, Scalable Full-text Search
+
Application and Infrastructure Monitoring
+
Security and Event Information Management
+
Operational Health Tracking
+
+
+
Help users find the right information within your application, website, or data lake catalog.
+
Easily store and analyze log data, and set automated alerts for underperformance.
+
Centralize logs to enable real-time security monitoring and forensic analysis.
+
Use observability logs, metrics, and traces to monitor your applications and business in real time.
+
+
+
+
+**Additional features and plugins:**
+
+OpenSearch has several features and plugins to help index, secure, monitor, and analyze your data. Most OpenSearch plugins have corresponding OpenSearch Dashboards plugins that provide a convenient, unified user interface.
+- [Anomaly detection]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/) - Identify atypical data and receive automatic notifications
+- [KNN]({{site.url}}{{site.baseurl}}/search-plugins/knn/) - Find “nearest neighbors” in your vector data
+- [Performance Analyzer]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/) - Monitor and optimize your cluster
+- [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) - Use SQL or a piped processing language to query your data
+- [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/) - Automate index operations
+- [ML Commons plugin]({{site.url}}{{site.baseurl}}/ml-commons-plugin/index/) - Train and execute machine-learning models
+- [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/) - Run search requests in the background
+- [Cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/index/) - Replicate your data across multiple OpenSearch clusters
+
+
+## The secure path forward
+OpenSearch includes a demo configuration so that you can get up and running quickly, but before using OpenSearch in a production environment, you must [configure the Security plugin manually]({{site.url}}{{site.baseurl}}/security/configuration/index/) with your own certificates, authentication method, users, and passwords.
+
+## Looking for the Javadoc?
+
+See [opensearch.org/javadocs/](https://opensearch.org/javadocs/).
+
+## Get involved
+
+[OpenSearch](https://opensearch.org) is supported by Amazon Web Services. All components are available under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) on [GitHub](https://github.com/opensearch-project/).
+The project welcomes GitHub issues, bug fixes, features, plugins, documentation---anything at all. To get involved, see [Contributing](https://opensearch.org/source.html) on the OpenSearch website.
+
+---
+
+OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.
\ No newline at end of file
diff --git a/about.md b/_about/intro.md
similarity index 94%
rename from about.md
rename to _about/intro.md
index 9b81727bb9..9913b859d7 100644
--- a/about.md
+++ b/_about/intro.md
@@ -1,16 +1,10 @@
---
layout: default
-title: About OpenSearch
+title: Intro to OpenSearch
nav_order: 2
-parent: OpenSearch documentation
-redirect_from:
- - /docs/opensearch/
- - /opensearch/
- - /opensearch/index/
+permalink: /intro/
---
-{%- comment -%}The `/docs/opensearch/` redirect is specifically to support the UI links in OpenSearch Dashboards 1.0.0.{%- endcomment -%}
-
# Introduction to OpenSearch
OpenSearch is a distributed search and analytics engine based on [Apache Lucene](https://lucene.apache.org/). After adding your data to OpenSearch, you can perform full-text searches on it with all of the features you might expect: search by field, search multiple indices, boost fields, rank results by score, sort results by field, and aggregate results.
@@ -83,13 +77,13 @@ PUT https://://_doc/
To run a search for the document:
-```
+```json
GET https://://_search?q=wind
```
To delete the document:
-```
+```json
DELETE https://://_doc/
```
diff --git a/quickstart.md b/_about/quickstart.md
similarity index 99%
rename from quickstart.md
rename to _about/quickstart.md
index 05cb8fbfb5..beecb46515 100644
--- a/quickstart.md
+++ b/_about/quickstart.md
@@ -1,8 +1,8 @@
---
layout: default
title: Quickstart
-parent: OpenSearch documentation
nav_order: 3
+permalink: /quickstart/
redirect_from:
- /opensearch/install/quickstart/
---
diff --git a/version-history.md b/_about/version-history.md
similarity index 99%
rename from version-history.md
rename to _about/version-history.md
index d3046f84a2..2303dbdaaa 100644
--- a/version-history.md
+++ b/_about/version-history.md
@@ -2,7 +2,6 @@
layout: default
title: Version history
nav_order: 4
-parent: OpenSearch documentation
permalink: /version-history/
---
diff --git a/_config.yml b/_config.yml
index bb9e7232df..6ad7627d70 100644
--- a/_config.yml
+++ b/_config.yml
@@ -1,4 +1,4 @@
-title: OpenSearch documentation
+title: OpenSearch Documentation
description: >- # this means to ignore newlines until "baseurl:"
Documentation for OpenSearch, the Apache 2.0 search, analytics, and visualization suite with advanced security, alerting, SQL support, automated index management, deep performance analysis, and more.
baseurl: "/docs/latest" # the subpath of your site, e.g. /blog
@@ -88,10 +88,16 @@ collections:
external_links:
permalink: /:collection/:path/
output: true
+ about:
+ permalink: /:collection/:path/
+ output: true
-just_the_docs:
+opensearch_collection:
# Define the collections used in the theme
collections:
+ about:
+ name: About OpenSearch
+ nav_fold: true
install-and-configure:
name: Install and upgrade
nav_fold: true
@@ -135,13 +141,7 @@ just_the_docs:
nav_fold: true
field-types:
name: Mappings and field types
- nav_fold: true
- clients:
- name: Clients
- nav_fold: true
- data-prepper:
- name: Data Prepper
- nav_fold: true
+ nav_fold: true
tools:
name: Tools
nav_fold: true
@@ -152,6 +152,51 @@ just_the_docs:
name: Troubleshooting
nav_fold: true
+clients_collection:
+ collections:
+ clients:
+ name: Clients
+ nav_fold: true
+
+benchmark_collection:
+ collections:
+ benchmark:
+ name: OpenSearch Benchmark
+ nav_fold: true
+
+data_prepper_collection:
+ collections:
+ data-prepper:
+ name: Data Prepper
+ nav_fold: true
+
+# Defaults
+
+defaults:
+ -
+ scope:
+ path: "" # an empty string here means all files in the project
+ values:
+ section: "opensearch"
+ section-name: "OpenSearch and OpenSearch Dashboards"
+ -
+ scope:
+ path: "_data-prepper"
+ values:
+ section: "data-prepper"
+ section-name: "Data Prepper"
+ -
+ scope:
+ path: "_clients"
+ values:
+ section: "clients"
+ section-name: "Clients"
+ -
+ scope:
+ path: "_benchmark"
+ values:
+ section: "benchmark"
+ section-name: "Benchmark"
# Enable or disable the site search
# By default, just-the-docs enables its JSON file-based search. We also have an OpenSearch-driven search functionality.
diff --git a/_data/top_nav.yml b/_data/top_nav.yml
index 73e9db7d2a..70ddbaeb55 100644
--- a/_data/top_nav.yml
+++ b/_data/top_nav.yml
@@ -80,8 +80,26 @@ items:
url: /community_projects
-
label: Documentation
- fragment: docs
- url: /docs/
+ fragments:
+ - docs
+ - opensearch
+ - data-prepper
+ - clients
+ - benchmark
+ url: /docs/latest/
+ children:
+ -
+ label: OpenSearch and Dashboards
+ url: /docs/latest/about/
+ -
+ label: Data Prepper
+ url: /docs/latest/data-prepper/
+ -
+ label: Clients
+ url: /docs/latest/clients/
+ -
+ label: Benchmark
+ url: /docs/latest/benchmark/
-
label: Platform
url: /platform/index.html
diff --git a/_im-plugin/index.md b/_im-plugin/index.md
index f7cd370e9c..30271b0e3b 100644
--- a/_im-plugin/index.md
+++ b/_im-plugin/index.md
@@ -4,8 +4,8 @@ title: Managing indexes
nav_order: 1
has_children: false
redirect_from:
- - /im-plugin/
- /opensearch/index-data/
+ - /im-plugin/
---
# Managing indexes
diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md
index 02a14d8df4..d15cac8948 100644
--- a/_im-plugin/ism/policies.md
+++ b/_im-plugin/ism/policies.md
@@ -164,7 +164,7 @@ Parameter | Description | Type | Required
}
```
-For information about setting replicas, see [Primary and replica shards]({{site.url}}{{site.baseurl}}/opensearch#primary-and-replica-shards).
+For information about setting replicas, see [Primary and replica shards]({{site.url}}{{site.baseurl}}/intro/#primary-and-replica-shards).
### shrink
diff --git a/_includes/banner.html b/_includes/banner.html
index f9ee474054..dadf93f578 100644
--- a/_includes/banner.html
+++ b/_includes/banner.html
@@ -1,5 +1,5 @@
-
+
OpenSearch Documentation
Learn to use OpenSearch, the highly scalable and extensible open-source software suite for search, analytics, observability, and other data-intensive applications.
diff --git a/_includes/cards.html b/_includes/cards.html
new file mode 100644
index 0000000000..4a2533843a
--- /dev/null
+++ b/_includes/cards.html
@@ -0,0 +1,37 @@
+
+
Explore OpenSearch documentation
+
+
+
+
OpenSearch and OpenSearch Dashboards
+
Learn how to power up your search
+
+
+
+
+
+
+
Data Prepper
+
Prepare your data for OpenSearch
+
+
+
+
+
+
Clients
+
OpenSearch in your programming language
+
+
+
+
+
+
+
Benchmark
+
Track OpenSearch performance
+
+
+
+
+
+
+
diff --git a/_ingest-pipelines/processors/grok.md b/_ingest-pipelines/processors/grok.md
deleted file mode 100644
index b2ddca0ac4..0000000000
--- a/_ingest-pipelines/processors/grok.md
+++ /dev/null
@@ -1,212 +0,0 @@
----
-layout: default
-title: Grok
-parent: Ingest processors
-grand_parent: Ingest pipelines
-nav_order: 140
----
-
-# Grok
-
-The `grok` processor is used to parse and structure unstructured data using pattern matching. You can use the `grok` processor to extract fields from log messages, web server access logs, application logs, and other log data that follows a consistent format.
-
-## Grok basics
-
-The `grok` processor uses a set of predefined patterns to match parts of the input text. Each pattern consists of a name and a regular expression. For example, the pattern `%{IP:ip_address}` matches an IP address and assigns it to the field `ip_address`. You can combine multiple patterns to create more complex expressions. For example, the pattern `%{IP:client} %{WORD:method} %{URIPATHPARM:request} %{NUMBER:bytes %NUMBER:duration}` matches a line from a web server access log and extracts the client IP address, the HTTP method, the request URI, the number of bytes sent, and the duration of the request.
-
-The `grok` processor is built on the [Oniguruma regular expression library](https://github.com/kkos/oniguruma/blob/master/doc/RE) and supports all the patterns from that library. You can use the [Grok Debugger](https://grokdebugger.com/) tool to test and debug your grok expressions.
-
-## Grok processor syntax
-
-The following is the basic syntax for the `grok` processor:
-
-```json
-{
- "grok": {
- "field": "your_message",
- "patterns": ["your_patterns"]
- }
-}
-```
-{% include copy-curl.html %}
-
-## Configuration parameters
-
-To configure the `grok` processor, you have various options that allow you to define patterns, match specific keys, and control the processor's behavior. The following table lists the required and optional parameters for the `grok` processor.
-
-Parameter | Required | Description |
-|-----------|-----------|-----------|
-`field` | Required | The name of the field containing the text that should be parsed. |
-`patterns` | Required | A list of grok expressions used to match and extract named captures. The first matching expression in the list is returned. |
-`pattern_definitions` | Optional | A dictionary of pattern names and pattern tuples used to define custom patterns for the current processor. If a pattern matches an existing name, it overrides the pre-existing definition. |
-`trace_match` | Optional | When the parameter is set to `true`, the processor adds a field named `_grok_match_index` to the processed document. This field contains the index of the pattern within the `patterns` array that successfully matched the document. This information can be useful for debugging and understanding which pattern was applied to the document. Default is `false`. |
-`description` | Optional | A brief description of the processor. |
-`if` | Optional | A condition for running this processor. |
-`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. |
-`ignore_missing` | Optional | If set to `true`, the processor does not modify the document if the field does not exist or is `null`. Default is `false`. |
-`on_failure` | Optional | A list of processors to run if the processor fails. |
-`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. |
-
-## Creating a pipeline
-
-The following steps guide you through creating an [ingest pipeline]({{site.url}}{{site.baseurl}}/ingest-pipelines/index/) with the `grok` processor.
-
-**Step 1: Create a pipeline.**
-
-The following query creates a pipeline, named `log_line`. It extracts fields from the `message` field of the document using the specified pattern. In this case, it extracts the `clientip`, `timestamp`, and `response_status` fields:
-
-```json
-PUT _ingest/pipeline/log_line
-{
- "description": "Extract fields from a log line",
- "processors": [
- {
- "grok": {
- "field": "message",
- "patterns": ["%{IPORHOST:clientip} %{HTTPDATE:timestamp} %{NUMBER:response_status:int}"]
- }
- }
- ]
-}
-```
-{% include copy-curl.html %}
-
-**Step 2 (Optional): Test the pipeline.**
-
-{::nomarkdown}{:/} **NOTE** It is recommended that you test your pipeline before you ingest documents.
-{: .note}
-
-To test the pipeline, run the following query:
-
-```json
-POST _ingest/pipeline/log_line/_simulate
-{
- "docs": [
- {
- "_source": {
- "message": "127.0.0.1 198.126.12 10/Oct/2000:13:55:36 -0700 200"
- }
- }
- ]
-}
-```
-{% include copy-curl.html %}
-
-#### Response
-
-The following response confirms that the pipeline is working as expected:
-
-```json
-{
- "docs": [
- {
- "doc": {
- "_index": "_index",
- "_id": "_id",
- "_source": {
- "message": "127.0.0.1 198.126.12 10/Oct/2000:13:55:36 -0700 200",
- "response_status": 200,
- "clientip": "198.126.12",
- "timestamp": "10/Oct/2000:13:55:36 -0700"
- },
- "_ingest": {
- "timestamp": "2023-09-13T21:41:52.064540505Z"
- }
- }
- }
- ]
-}
-```
-
-**Step 3: Ingest a document.**
-
-The following query ingests a document into an index named `testindex1`:
-
-```json
-PUT testindex1/_doc/1?pipeline=log_line
-{
- "message": "127.0.0.1 198.126.12 10/Oct/2000:13:55:36 -0700 200"
-}
-```
-{% include copy-curl.html %}
-
-**Step 4 (Optional): Retrieve the document.**
-
-To retrieve the document, run the following query:
-
-```json
-GET testindex1/_doc/1
-```
-{% include copy-curl.html %}
-
-## Custom patterns
-
-You can use default patterns, or you can add custom patterns to your pipelines using the `patterns_definitions` parameter. Custom grok patterns can be used in a pipeline to extract structured data from log messages that do not match the built-in grok patterns. This can be useful for parsing log messages from custom applications or for parsing log messages that have been modified in some way. Custom patterns adhere to a straightforward structure: each pattern has a unique name and the corresponding regular expression that defines its matching behavior.
-
-The following is an example of how to include a custom pattern in your configuration. In this example, the issue number is between 3 and 4 digits and is parsed into the `issue_number` field and the status is parsed into the `status` field:
-
-```json
-PUT _ingest/pipeline/log_line
-{
- "processors": [
- {
- "grok": {
- "field": "message",
- "patterns": ["The issue number %{NUMBER:issue_number} is %{STATUS:status}"],
- "pattern_definitions" : {
- "NUMBER" : "\\d{3,4}",
- "STATUS" : "open|closed"
- }
- }
- }
- ]
-}
-```
-{% include copy-curl.html %}
-
-## Tracing which patterns matched
-
-To trace which patterns matched and populated the fields, you can use the `trace_match` parameter. The following is an example of how to include this parameter in your configuration:
-
-```json
-PUT _ingest/pipeline/log_line
-{
- "description": "Extract fields from a log line",
- "processors": [
- {
- "grok": {
- "field": "message",
- "patterns": ["%{HTTPDATE:timestamp} %{IPORHOST:clientip}", "%{IPORHOST:clientip} %{HTTPDATE:timestamp} %{NUMBER:response_status:int}"],
- "trace_match": true
- }
- }
- ]
-}
-```
-{% include copy-curl.html %}
-
-When you simulate the pipeline, OpenSearch returns the `_ingest` metadata that includes the `grok_match_index`, as shown in the following output:
-
-```json
-{
- "docs": [
- {
- "doc": {
- "_index": "_index",
- "_id": "_id",
- "_source": {
- "message": "127.0.0.1 198.126.12 10/Oct/2000:13:55:36 -0700 200",
- "response_status": 200,
- "clientip": "198.126.12",
- "timestamp": "10/Oct/2000:13:55:36 -0700"
- },
- "_ingest": {
- "_grok_match_index": "1",
- "timestamp": "2023-11-02T18:48:40.455619084Z"
- }
- }
- }
- ]
-}
-```
-
diff --git a/_layouts/default.html b/_layouts/default.html
index 20596807e6..a209e60393 100755
--- a/_layouts/default.html
+++ b/_layouts/default.html
@@ -56,10 +56,20 @@
Documentation