From fb3dd157efe9dd26806c9a7f1608f691ddbc714b Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 15 Oct 2024 12:23:26 +0100 Subject: [PATCH 1/6] add pattern analyzer docs Signed-off-by: Anton Rubin --- _analyzers/pattern.md | 94 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 _analyzers/pattern.md diff --git a/_analyzers/pattern.md b/_analyzers/pattern.md new file mode 100644 index 0000000000..a711c38b51 --- /dev/null +++ b/_analyzers/pattern.md @@ -0,0 +1,94 @@ +--- +layout: default +title: Pattern analyzer +nav_order: 90 +--- + +# Pattern analyzer + +The `pattern` analyzer allows you to define a custom analyzer that uses a regular expression (regex) to split the input text into tokens. It also provides options for applying regex flags, converting tokens to lowercase, and filtering out `stopwords`. + +## Configuration + +The `pattern` analyzer can be configured using the following parameters: + +- `pattern`: A [Java regular expression](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) used to tokenize the input. Default is `\W+`. (String, _Optional_) +- `flags`: [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. (String, _Optional_) +- `lowercase`: Convert tokens to lower case. Default is `true`. (Boolean, _Optional_) +- `stopwords`: a custom list or predefined list of stop words. Default is `_none_`. (String or list of strings, _Optional_) +- `stopwords_path`: Path (absolute of relative to config directory) to the list of stop words. (String, _Optional_) + + +## Example configuration + +You can use the following command to create index `my_pattern_index` with `pattern` analyzer: + +```json +PUT /my_pattern_index +{ + "settings": { + "analysis": { + "analyzer": { + "my_pattern_analyzer": { + "type": "pattern", + "pattern": "\\W+", + "lowercase": true, + "stopwords": ["and", "is"] + } + } + } + }, + "mappings": { + "properties": { + "my_field": { + "type": "text", + "analyzer": "my_pattern_analyzer" + } + } + } +} +``` +{% include copy-curl.html %} + +## Generated tokens + +Use the following request to examine the tokens generated using the created analyzer: + +```json +POST /my_pattern_index/_analyze +{ + "analyzer": "my_pattern_analyzer", + "text": "OpenSearch is fast and scalable" +} +``` +{% include copy-curl.html %} + +The response contains the generated tokens: + +```json +{ + "tokens": [ + { + "token": "opensearch", + "start_offset": 0, + "end_offset": 10, + "type": "word", + "position": 0 + }, + { + "token": "fast", + "start_offset": 14, + "end_offset": 18, + "type": "word", + "position": 2 + }, + { + "token": "scalable", + "start_offset": 23, + "end_offset": 31, + "type": "word", + "position": 4 + } + ] +} +``` From c40dca389410c4927ca1d1376c164faa23bc02fe Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Tue, 15 Oct 2024 12:32:06 +0100 Subject: [PATCH 2/6] Update pattern.md Signed-off-by: AntonEliatra --- _analyzers/pattern.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_analyzers/pattern.md b/_analyzers/pattern.md index a711c38b51..46e191ce25 100644 --- a/_analyzers/pattern.md +++ b/_analyzers/pattern.md @@ -16,7 +16,7 @@ The `pattern` analyzer can be configured using the following parameters: - `flags`: [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. (String, _Optional_) - `lowercase`: Convert tokens to lower case. Default is `true`. (Boolean, _Optional_) - `stopwords`: a custom list or predefined list of stop words. Default is `_none_`. (String or list of strings, _Optional_) -- `stopwords_path`: Path (absolute of relative to config directory) to the list of stop words. (String, _Optional_) +- `stopwords_path`: Path (absolute or relative to config directory) to the list of stop words. (String, _Optional_) ## Example configuration From 768072e0089541a837d35352c91d5767f78b9648 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Wed, 16 Oct 2024 16:45:15 +0100 Subject: [PATCH 3/6] updating parameter table Signed-off-by: Anton Rubin --- _analyzers/pattern.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/_analyzers/pattern.md b/_analyzers/pattern.md index 46e191ce25..30a66715ac 100644 --- a/_analyzers/pattern.md +++ b/_analyzers/pattern.md @@ -10,13 +10,15 @@ The `pattern` analyzer allows you to define a custom analyzer that uses a regula ## Configuration -The `pattern` analyzer can be configured using the following parameters: +The `pattern` analyzer can be configured using the following parameters. -- `pattern`: A [Java regular expression](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) used to tokenize the input. Default is `\W+`. (String, _Optional_) -- `flags`: [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. (String, _Optional_) -- `lowercase`: Convert tokens to lower case. Default is `true`. (Boolean, _Optional_) -- `stopwords`: a custom list or predefined list of stop words. Default is `_none_`. (String or list of strings, _Optional_) -- `stopwords_path`: Path (absolute or relative to config directory) to the list of stop words. (String, _Optional_) +Parameter | Required/Optional | Data type | Description +:--- | :--- | :--- | :--- +`pattern` | Optional | String | A [Java regular expression](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) used to tokenize the input. Default is `\W+`. +`flags` | Optional | String | [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. +`lowercase` | Optional | Boolean | Convert tokens to lower case. Default is `true`. +`stopwords` | Optional | String or list of strings | Custom list or predefined list of stop words. Default is `_none_`. +`stopwords_path` | Optional | String | Path (absolute or relative to config directory) to the list of stop words. ## Example configuration From fd9313a60a5c9c4f090d3f84fdc4c8e49711591c Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Fri, 6 Dec 2024 13:22:37 -0500 Subject: [PATCH 4/6] Doc review Signed-off-by: Fanit Kolchina --- _analyzers/pattern.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/_analyzers/pattern.md b/_analyzers/pattern.md index 30a66715ac..cbf6fa3a87 100644 --- a/_analyzers/pattern.md +++ b/_analyzers/pattern.md @@ -6,24 +6,24 @@ nav_order: 90 # Pattern analyzer -The `pattern` analyzer allows you to define a custom analyzer that uses a regular expression (regex) to split the input text into tokens. It also provides options for applying regex flags, converting tokens to lowercase, and filtering out `stopwords`. +The `pattern` analyzer allows you to define a custom analyzer that uses a regular expression (regex) to split the input text into tokens. It also provides options for applying regex flags, converting tokens to lowercase, and filtering out stopwords. -## Configuration +## Parameters The `pattern` analyzer can be configured using the following parameters. Parameter | Required/Optional | Data type | Description :--- | :--- | :--- | :--- `pattern` | Optional | String | A [Java regular expression](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) used to tokenize the input. Default is `\W+`. -`flags` | Optional | String | [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. -`lowercase` | Optional | Boolean | Convert tokens to lower case. Default is `true`. -`stopwords` | Optional | String or list of strings | Custom list or predefined list of stop words. Default is `_none_`. -`stopwords_path` | Optional | String | Path (absolute or relative to config directory) to the list of stop words. +`flags` | Optional | String | A string containing pipe-separated [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. +`lowercase` | Optional | Boolean | Whether to convert tokens to lowercase. Default is `true`. +`stopwords` | Optional | String or list of strings | A string specifying a predefined list of stopwords (such as `_english_`) or an array specifying a custom list of stopwords. Default is `_none_`. +`stopwords_path` | Optional | String | The path (absolute or relative to the config directory) to the file containing a list of stop words. -## Example configuration +## Example -You can use the following command to create index `my_pattern_index` with `pattern` analyzer: +Use the following command to create an index named `my_pattern_index` with a `pattern` analyzer: ```json PUT /my_pattern_index @@ -54,7 +54,7 @@ PUT /my_pattern_index ## Generated tokens -Use the following request to examine the tokens generated using the created analyzer: +Use the following request to examine the tokens generated using the analyzer: ```json POST /my_pattern_index/_analyze From 62ad3e138dfbd15f3cc5bb1905f536f21deed4b4 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Fri, 6 Dec 2024 13:28:55 -0500 Subject: [PATCH 5/6] Update _analyzers/pattern.md Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _analyzers/pattern.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_analyzers/pattern.md b/_analyzers/pattern.md index cbf6fa3a87..1e306ad554 100644 --- a/_analyzers/pattern.md +++ b/_analyzers/pattern.md @@ -18,7 +18,7 @@ Parameter | Required/Optional | Data type | Description `flags` | Optional | String | A string containing pipe-separated [Java regex flags](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary) that modify the behavior of the regular expression. `lowercase` | Optional | Boolean | Whether to convert tokens to lowercase. Default is `true`. `stopwords` | Optional | String or list of strings | A string specifying a predefined list of stopwords (such as `_english_`) or an array specifying a custom list of stopwords. Default is `_none_`. -`stopwords_path` | Optional | String | The path (absolute or relative to the config directory) to the file containing a list of stop words. +`stopwords_path` | Optional | String | The path (absolute or relative to the config directory) to the file containing a list of stopwords. ## Example From 4c6a8c5d3bd1e6e478f9b2e1d6eb8954de817034 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:55:16 -0500 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _analyzers/pattern.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_analyzers/pattern.md b/_analyzers/pattern.md index 1e306ad554..0d67999b82 100644 --- a/_analyzers/pattern.md +++ b/_analyzers/pattern.md @@ -6,11 +6,11 @@ nav_order: 90 # Pattern analyzer -The `pattern` analyzer allows you to define a custom analyzer that uses a regular expression (regex) to split the input text into tokens. It also provides options for applying regex flags, converting tokens to lowercase, and filtering out stopwords. +The `pattern` analyzer allows you to define a custom analyzer that uses a regular expression (regex) to split input text into tokens. It also provides options for applying regex flags, converting tokens to lowercase, and filtering out stopwords. ## Parameters -The `pattern` analyzer can be configured using the following parameters. +The `pattern` analyzer can be configured with the following parameters. Parameter | Required/Optional | Data type | Description :--- | :--- | :--- | :---