-
Notifications
You must be signed in to change notification settings - Fork 502
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* add trim token filter docs #8449 Signed-off-by: Anton Rubin <[email protected]> * updating the nav_order Signed-off-by: Anton Rubin <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Anton Rubin <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit deac802) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- Loading branch information
1 parent
d812a72
commit c7977a9
Showing
2 changed files
with
94 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
--- | ||
layout: default | ||
title: Trim | ||
parent: Token filters | ||
nav_order: 430 | ||
--- | ||
|
||
# Trim token filter | ||
|
||
The `trim` token filter removes leading and trailing white space characters from tokens. | ||
|
||
Many popular tokenizers, such as `standard`, `keyword`, and `whitespace` tokenizers, automatically strip leading and trailing white space characters during tokenization. When using these tokenizers, there is no need to configure an additional `trim` token filter. | ||
{: .note} | ||
|
||
|
||
## Example | ||
|
||
The following example request creates a new index named `my_pattern_trim_index` and configures an analyzer with a `trim` filter and a `pattern` tokenizer, which does not remove leading and trailing white space characters: | ||
|
||
```json | ||
PUT /my_pattern_trim_index | ||
{ | ||
"settings": { | ||
"analysis": { | ||
"filter": { | ||
"my_trim_filter": { | ||
"type": "trim" | ||
} | ||
}, | ||
"tokenizer": { | ||
"my_pattern_tokenizer": { | ||
"type": "pattern", | ||
"pattern": "," | ||
} | ||
}, | ||
"analyzer": { | ||
"my_pattern_trim_analyzer": { | ||
"type": "custom", | ||
"tokenizer": "my_pattern_tokenizer", | ||
"filter": [ | ||
"lowercase", | ||
"my_trim_filter" | ||
] | ||
} | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
## Generated tokens | ||
|
||
Use the following request to examine the tokens generated using the analyzer: | ||
|
||
```json | ||
GET /my_pattern_trim_index/_analyze | ||
{ | ||
"analyzer": "my_pattern_trim_analyzer", | ||
"text": " OpenSearch , is , powerful " | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
The response contains the generated tokens: | ||
|
||
```json | ||
{ | ||
"tokens": [ | ||
{ | ||
"token": "opensearch", | ||
"start_offset": 0, | ||
"end_offset": 12, | ||
"type": "word", | ||
"position": 0 | ||
}, | ||
{ | ||
"token": "is", | ||
"start_offset": 13, | ||
"end_offset": 18, | ||
"type": "word", | ||
"position": 1 | ||
}, | ||
{ | ||
"token": "powerful", | ||
"start_offset": 19, | ||
"end_offset": 32, | ||
"type": "word", | ||
"position": 2 | ||
} | ||
] | ||
} | ||
``` |