Vietnamese Analysis plugin integrates Vietnamese language analysis into OpenSearch. It uses C++ tokenizer for Vietnamese library developed by CocCoc team for their Search Engine and Ads systems.
The plugin provides vi_analyzer
analyzer, vi_tokenizer
tokenizer and vi_stop
stop filter. The vi_analyzer
is composed of the vi_tokenizer
tokenizer and stop
filter.
GET /_analyze
{
"analyzer": "vi_analyzer",
"text": "công nghệ thông tin việt nam"
}
The above sentence would produce the following terms:
["công nghệ", "thông tin", "việt nam"]
The vi_analyzer
analyzer accepts the following parameters:
-
dict_path
The path to tokenizer dictionary on system. Defaults to/usr/share/tokenizer/dicts
. -
keep_punctuation
Keep punctuation marks as tokens. Defaults tofalse
. -
split_url
If it's enabled (true
), a domainduydo.me
is split into["duy", "do", "me"]
. If it's disabled (false
)duydo.me
is split into["duydo", "me"]
. Defaults tofalse
. -
stopwords
A pre-defined stop words list like_vi_
or an array containing a list of stop words. Defaults to stopwords.txt file. -
stopwords_path
The path to a file containing stop words.
In this example, we configure the vi_analyzer
analyzer to keep punctuation marks and to use the custom list of stop words:
PUT my-vi-index-00001
{
"settings": {
"analysis": {
"analyzer": {
"my_vi_analyzer": {
"type": "vi_analyzer",
"keep_punctuation": true,
"stopwords": ["rất", "những"]
}
}
}
}
}
GET my-vi-index-00001/_analyze
{
"analyzer": "my_vi_analyzer",
"text": "công nghệ thông tin Việt Nam rất phát triển trong những năm gần đây."
}
The above example produces the following terms:
["công nghệ", "thông tin", "việt nam", "phát triển", "trong", "năm", "gần đây", "."]
git clone https://github.com/coccoc/coccoc-tokenizer.git
cd coccoc-tokenizer && mkdir build && cd build
cmake -DBUILD_JAVA=1 ..
make install
Refer the repo for more information to build the library.
Clone the plugin’s source code:
git clone https://github.com/duydo/opensearch-analysis-vietnamese.git
Optionally, edit the opensearch-analysis-vietnamese/pom.xml
to change the version of OpenSearch (same as plugin version) you want to build the plugin with:
...
<groupId>org.opensearch</groupId>
<artifactId>opensearch-analysis-vietnamese</artifactId>
<version>1.3.2</version>
...
Build the plugin:
cd opensearch-analysis-vietnamese
mvn package
bin/opensearch-plugin install file://target/releases/opensearch-analysis-vietnamese-1.3.2.zip
Vietnamese Analysis Plugin | OpenSearch |
---|---|
master | 1.3.2 |
- JetBrains has provided a free license for IntelliJ IDEA.
- CocCoc team has provided their C++ Vietnamese tokenizer library as open source.
This software is licensed under the Apache 2 license, quoted below.
Copyright by Duy Do
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.