From a28a07866686cfc5ab8b7de254a15db47bd78089 Mon Sep 17 00:00:00 2001 From: Mulugeta Mammo Date: Tue, 2 Apr 2024 12:36:18 -0700 Subject: [PATCH 01/10] Add documentation for hardware-accelerated compression codecs. Signed-off-by: Mulugeta Mammo --- _im-plugin/index-codecs.md | 16 ++++++++++++++++ .../configuring-opensearch/index-settings.md | 15 +++++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index f880e141b3..9a58148709 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -32,6 +32,21 @@ For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compressi When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations). {: .note} +As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel (R) Xeon (R) processors running Linux kernel 3.10 and above. For all other systems and platforms, the codecs use their corresponding software implementations. + +The new hard-accelerated codecs can be used by setting the value of `index.codec`: +* `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4.  +* `qat_deflate` (OpenSearch 2.14 and later) -- uses hardware-accelerated DEFLATE. `qat_deflate` offers a much better compression ratio than `qat_lz4`, but with a modest drop in compression and decompression speed. + +Just like `zstd` and `zstd_no_dict`, the `index.codec.compression_level` can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. The new setting, `index.codec.qatmode`, controls the behavior of the hardware accelerator. It takes two valid values: + +* `auto` -- uses hardware-accelerated compression. Fails over to use software if the hardware accelerator is not available. +* `hardware` -- use hardware-accelerated compression. An exception is thrown if hardware is not available (guarantees a hardware-only compression). + +Refer to the snapshot section of this page for the implications of the above values on snapshot and restore. + +The hardware accelerator that is used is [Intel (R) QAT accelerator](https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html). + ## Choosing a codec The choice of index codec impacts the amount of disk space required to store the index data. Codecs like `best_compression`, `zstd`, and `zstd_no_dict` can achieve higher compression ratios, resulting in smaller index sizes. Conversely, the `default` codec doesn’t prioritize compression ratio, resulting in larger index sizes but faster search operations than `best_compression`. @@ -58,6 +73,7 @@ When creating a [snapshot]({{site.url}}{{site.baseurl}}/tuning-your-cluster/avai When you restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs of the segments in the source snapshot. For example, if the source snapshot contains segments of the `zstd` or `zstd_no_dict` codecs (introduced in OpenSearch 2.9), you won't be able to restore the snapshot to a cluster that runs on an older OpenSearch version because it doesn't support these codecs. +For hardware-accelerated compression codecs, available starting with OpenSearch 2.14, the value of `index.codec.qatmode` affects how snapshots and restores are done. If the value set for `index.codec.qatmode` is `auto` (the default), snapshots and restores can be done with no additional instructions. But if the value was set to `hardware`, it will need to change to `auto` for restore to work on systems that do not have the hardware accelerator. You can specify a new value for`index.codec.qatmode` during a restore process using: `"index_settings": {"index.codec.qatmode": "auto"}`. ### Reindexing When you are performing a [reindex]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/) operation from a source index, the new segments created in the target index will have the properties of the codec settings of the target index. diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index 25cd4b8810..bbf5853690 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -79,10 +79,21 @@ OpenSearch supports the following static index-level index settings: - `best_compression` - `zstd` (OpenSearch 2.9 and later) - `zstd_no_dict`(OpenSearch 2.9 and later) + - `qat_lz4`(OpenSearch 2.14 and later, on supported systems) + - `qat_deflate`(OpenSearch 2.14 and later, on supported systems) - For `zstd` and `zstd_no_dict`, you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. + For `zstd`, `zstd_no_dict`, `qat_lz4`, and `qat_deflate` you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. -- `index.codec.compression_level` (Integer): The compression level setting provides a tradeoff between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. +- `index.codec.compression_level` (Integer): The compression level setting provides a tradeoff between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. The setting is also used by the `qat_lz4` and `qat_deflate` codecs, in OpenSearch 2.14 and later, except that the default compression level for both is 6. + +- `index.codec.qatmode` (String): The hardware acceleration mode used for `qat_lz4` and `qat_deflate` compression codecs: + + Valid values are: + + - `auto` + - `hardware` + + Optional. Default is `auto`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). - `index.routing_partition_size` (Integer): The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than a single shard. To enable routing, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1. From e664ef6c0e5142ee6ff5009d7e5c803d61744960 Mon Sep 17 00:00:00 2001 From: Mulugeta Mammo Date: Tue, 2 Apr 2024 14:36:24 -0700 Subject: [PATCH 02/10] Fix based on the review from reviewdog. Signed-off-by: Mulugeta Mammo --- _im-plugin/index-codecs.md | 6 +++--- .../configuring-opensearch/index-settings.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index 9a58148709..b7cee2de9f 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -32,18 +32,18 @@ For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compressi When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations). {: .note} -As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel (R) Xeon (R) processors running Linux kernel 3.10 and above. For all other systems and platforms, the codecs use their corresponding software implementations. +As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel\(R\) Xeon\(R\) processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use their corresponding software implementations. The new hard-accelerated codecs can be used by setting the value of `index.codec`: * `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4.  * `qat_deflate` (OpenSearch 2.14 and later) -- uses hardware-accelerated DEFLATE. `qat_deflate` offers a much better compression ratio than `qat_lz4`, but with a modest drop in compression and decompression speed. -Just like `zstd` and `zstd_no_dict`, the `index.codec.compression_level` can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. The new setting, `index.codec.qatmode`, controls the behavior of the hardware accelerator. It takes two valid values: +The `index.codec.compression_level` can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. The new setting, `index.codec.qatmode`, controls the behavior of the hardware accelerator. It takes two valid values: * `auto` -- uses hardware-accelerated compression. Fails over to use software if the hardware accelerator is not available. * `hardware` -- use hardware-accelerated compression. An exception is thrown if hardware is not available (guarantees a hardware-only compression). -Refer to the snapshot section of this page for the implications of the above values on snapshot and restore. +Refer to the snapshot section of this page for the implications `index.codec.qatmode` on snapshot and restore. The hardware accelerator that is used is [Intel (R) QAT accelerator](https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html). diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index bbf5853690..9903ec5f38 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -84,7 +84,7 @@ OpenSearch supports the following static index-level index settings: For `zstd`, `zstd_no_dict`, `qat_lz4`, and `qat_deflate` you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. -- `index.codec.compression_level` (Integer): The compression level setting provides a tradeoff between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. The setting is also used by the `qat_lz4` and `qat_deflate` codecs, in OpenSearch 2.14 and later, except that the default compression level for both is 6. +- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a trad-eoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. The setting is also used by the `qat_lz4` and `qat_deflate` codecs, in OpenSearch 2.14 and later, except that the default compression level for both is 6. - `index.codec.qatmode` (String): The hardware acceleration mode used for `qat_lz4` and `qat_deflate` compression codecs: From f79155924a7f3d25d160c7f6be263119bdc1843a Mon Sep 17 00:00:00 2001 From: mulugetam Date: Tue, 2 Apr 2024 14:47:14 -0700 Subject: [PATCH 03/10] Use the registered symbol for Intel and Xeon. Signed-off-by: mulugetam --- _im-plugin/index-codecs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index b7cee2de9f..a994cd996c 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -32,7 +32,7 @@ For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compressi When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations). {: .note} -As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel\(R\) Xeon\(R\) processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use their corresponding software implementations. +As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel®️ Xeon®️ processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use their corresponding software implementations. The new hard-accelerated codecs can be used by setting the value of `index.codec`: * `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4.  From 862f96b19802b09478bd6a213c88ffd63af33fa3 Mon Sep 17 00:00:00 2001 From: mulugetam Date: Tue, 2 Apr 2024 15:18:33 -0700 Subject: [PATCH 04/10] Minor modifications to the text. Signed-off-by: mulugetam --- _im-plugin/index-codecs.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index a994cd996c..c7703d4162 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -36,7 +36,10 @@ As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and L The new hard-accelerated codecs can be used by setting the value of `index.codec`: * `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4.  -* `qat_deflate` (OpenSearch 2.14 and later) -- uses hardware-accelerated DEFLATE. `qat_deflate` offers a much better compression ratio than `qat_lz4`, but with a modest drop in compression and decompression speed. +* `qat_deflate` (OpenSearch 2.14 and later) -- uses hardware-accelerated DEFLATE. + +`qat_deflate` offers a much better compression ratio than `qat_lz4` with a modest drop in compression and decompression speed. +{: .note} The `index.codec.compression_level` can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. The new setting, `index.codec.qatmode`, controls the behavior of the hardware accelerator. It takes two valid values: @@ -73,7 +76,11 @@ When creating a [snapshot]({{site.url}}{{site.baseurl}}/tuning-your-cluster/avai When you restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs of the segments in the source snapshot. For example, if the source snapshot contains segments of the `zstd` or `zstd_no_dict` codecs (introduced in OpenSearch 2.9), you won't be able to restore the snapshot to a cluster that runs on an older OpenSearch version because it doesn't support these codecs. -For hardware-accelerated compression codecs, available starting with OpenSearch 2.14, the value of `index.codec.qatmode` affects how snapshots and restores are done. If the value set for `index.codec.qatmode` is `auto` (the default), snapshots and restores can be done with no additional instructions. But if the value was set to `hardware`, it will need to change to `auto` for restore to work on systems that do not have the hardware accelerator. You can specify a new value for`index.codec.qatmode` during a restore process using: `"index_settings": {"index.codec.qatmode": "auto"}`. +For hardware-accelerated compression codecs, available starting with OpenSearch 2.14, the value of `index.codec.qatmode` affects how snapshots and restores are done. If the value set for `index.codec.qatmode` is `auto` (the default), snapshots and restores can be done with no additional instructions. But if the value is `hardware`, you won't be able to restore the snapshot unless you set it to `auto`. + +You may modify the value of `index.codec.qatmode` during a retore process by setting its value: `"index_settings": {"index.codec.qatmode": "auto"}`. +{: .note} + ### Reindexing When you are performing a [reindex]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/) operation from a source index, the new segments created in the target index will have the properties of the codec settings of the target index. From 4c5b3af68b01e2217f03d192be5a22a106576eb6 Mon Sep 17 00:00:00 2001 From: mulugetam Date: Wed, 3 Apr 2024 23:00:31 -0700 Subject: [PATCH 05/10] Rephrasing. Signed-off-by: mulugetam --- _install-and-configure/configuring-opensearch/index-settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index 9903ec5f38..c4b52a361e 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -84,7 +84,7 @@ OpenSearch supports the following static index-level index settings: For `zstd`, `zstd_no_dict`, `qat_lz4`, and `qat_deflate` you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. -- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a trad-eoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. The setting is also used by the `qat_lz4` and `qat_deflate` codecs, in OpenSearch 2.14 and later, except that the default compression level for both is 6. +- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a trad-eoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if index.codec is set to `zstd` and `zstd_no_dict` in OpenSearch 2.9 and later, and `qat_lz4` and `qat_deflate` in OpenSearch 2.14 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. - `index.codec.qatmode` (String): The hardware acceleration mode used for `qat_lz4` and `qat_deflate` compression codecs: From d2e1c239d967d611bd4ac34307ca341eadc11b73 Mon Sep 17 00:00:00 2001 From: mulugetam Date: Wed, 3 Apr 2024 23:10:36 -0700 Subject: [PATCH 06/10] Paraphrasing. Signed-off-by: mulugetam --- _im-plugin/index-codecs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index c7703d4162..3d3f4aa093 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -76,9 +76,9 @@ When creating a [snapshot]({{site.url}}{{site.baseurl}}/tuning-your-cluster/avai When you restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs of the segments in the source snapshot. For example, if the source snapshot contains segments of the `zstd` or `zstd_no_dict` codecs (introduced in OpenSearch 2.9), you won't be able to restore the snapshot to a cluster that runs on an older OpenSearch version because it doesn't support these codecs. -For hardware-accelerated compression codecs, available starting with OpenSearch 2.14, the value of `index.codec.qatmode` affects how snapshots and restores are done. If the value set for `index.codec.qatmode` is `auto` (the default), snapshots and restores can be done with no additional instructions. But if the value is `hardware`, you won't be able to restore the snapshot unless you set it to `auto`. +For hardware-accelerated compression codecs, available in OpenSearch 2.14 and later, the value of `index.codec.qatmode` affects how snapshots and restores are performed. If the value is `auto` (the default), then snapshots and restores can be performed with no additional instructions. However, if the value is `hardware`, it must be reset to `auto` for the restore process to succeed on systems lacking the hardware accelerator -You may modify the value of `index.codec.qatmode` during a retore process by setting its value: `"index_settings": {"index.codec.qatmode": "auto"}`. +You can modify the value of `index.codec.qatmode` during a restore process by setting its value as follows: `"index_settings": {"index.codec.qatmode": "auto"}`. {: .note} ### Reindexing From f830500952c9e7bf73f66a77d6ce43065f5deea6 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 29 Apr 2024 13:01:36 -0500 Subject: [PATCH 07/10] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _im-plugin/index-codecs.md | 18 ++++++++++-------- .../configuring-opensearch/index-settings.md | 5 +---- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index 3d3f4aa093..70233761b0 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -32,23 +32,25 @@ For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compressi When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations). {: .note} -As of OpenSearch 2.14, hardware-accelerated compression codecs for DEFLATE and LZ4 compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel®️ Xeon®️ processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use their corresponding software implementations. +As of OpenSearch 2.14, hardware-accelerated compression codecs for the `DEFLATE` and `LZ4` compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel®️ Xeon®️ processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use that platform's corresponding software implementations. -The new hard-accelerated codecs can be used by setting the value of `index.codec`: +The new hardware-accelerated codecs can be used by setting one of the following values of the `index.codec` setting: * `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4.  -* `qat_deflate` (OpenSearch 2.14 and later) -- uses hardware-accelerated DEFLATE. +* `qat_deflate` (OpenSearch 2.14 and later): Hardware-accelerated `DEFLATE`. `qat_deflate` offers a much better compression ratio than `qat_lz4` with a modest drop in compression and decompression speed. {: .note} -The `index.codec.compression_level` can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. The new setting, `index.codec.qatmode`, controls the behavior of the hardware accelerator. It takes two valid values: +The `index.codec.compression_level` setting can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. -* `auto` -- uses hardware-accelerated compression. Fails over to use software if the hardware accelerator is not available. -* `hardware` -- use hardware-accelerated compression. An exception is thrown if hardware is not available (guarantees a hardware-only compression). +The `index.codec.qatmode` setting controls the behavior of the hardware accelerator and uses on of the following values: -Refer to the snapshot section of this page for the implications `index.codec.qatmode` on snapshot and restore. +* `auto`: If hardware acceleration fails, the algorithm switches to software acceleration. +* `hardware`: Guarantees hardware-only compression. If hardware is not available, an exception occurs until hardware exists. -The hardware accelerator that is used is [Intel (R) QAT accelerator](https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html). +For information about the `index.codec.qatmode` settings implications on snapshots, see the [Snapshots](#snapshots) section. + +For more information about hardware acceleration on Intel, see the [Intel (R) QAT accelerator overview](https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html). ## Choosing a codec diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index c4b52a361e..ba7c556f1f 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -86,14 +86,11 @@ OpenSearch supports the following static index-level index settings: - `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a trad-eoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if index.codec is set to `zstd` and `zstd_no_dict` in OpenSearch 2.9 and later, and `qat_lz4` and `qat_deflate` in OpenSearch 2.14 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. -- `index.codec.qatmode` (String): The hardware acceleration mode used for `qat_lz4` and `qat_deflate` compression codecs: +- `index.codec.qatmode` (String): The hardware acceleration mode used for the `qat_lz4` and `qat_deflate` compression codecs. Valid values are `auto` and `hardware`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `auto`. Valid values are: - - `auto` - - `hardware` - Optional. Default is `auto`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). - `index.routing_partition_size` (Integer): The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than a single shard. To enable routing, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1. From 05d9cfee01f9695eeed0081173d417f8e11e9eab Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 29 Apr 2024 13:10:08 -0500 Subject: [PATCH 08/10] Update _im-plugin/index-codecs.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _im-plugin/index-codecs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index 70233761b0..c55be8b584 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -35,7 +35,7 @@ When an index segment is created, it uses the current index codec for compressio As of OpenSearch 2.14, hardware-accelerated compression codecs for the `DEFLATE` and `LZ4` compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel®️ Xeon®️ processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use that platform's corresponding software implementations. The new hardware-accelerated codecs can be used by setting one of the following values of the `index.codec` setting: -* `qat_lz4` (OpenSearch 2.14 and later) -- uses hardware-accelerated LZ4.  +* `qat_lz4` (OpenSearch 2.14 and later): Hardware-accelerated `LZ4`.  * `qat_deflate` (OpenSearch 2.14 and later): Hardware-accelerated `DEFLATE`. `qat_deflate` offers a much better compression ratio than `qat_lz4` with a modest drop in compression and decompression speed. From ec15fec817a51b7e579b761eda048535998326cd Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 29 Apr 2024 13:10:17 -0500 Subject: [PATCH 09/10] Update _install-and-configure/configuring-opensearch/index-settings.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _install-and-configure/configuring-opensearch/index-settings.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index ba7c556f1f..3e027ebee6 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -88,7 +88,6 @@ OpenSearch supports the following static index-level index settings: - `index.codec.qatmode` (String): The hardware acceleration mode used for the `qat_lz4` and `qat_deflate` compression codecs. Valid values are `auto` and `hardware`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `auto`. - Valid values are: From 13f9bfca73624ebbce75903b3900963dcdeadf7c Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 30 Apr 2024 10:12:09 -0500 Subject: [PATCH 10/10] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _im-plugin/index-codecs.md | 20 +++++++++---------- .../configuring-opensearch/index-settings.md | 8 ++++---- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/_im-plugin/index-codecs.md b/_im-plugin/index-codecs.md index c55be8b584..5ba1c926bb 100644 --- a/_im-plugin/index-codecs.md +++ b/_im-plugin/index-codecs.md @@ -34,21 +34,21 @@ When an index segment is created, it uses the current index codec for compressio As of OpenSearch 2.14, hardware-accelerated compression codecs for the `DEFLATE` and `LZ4` compression algorithms are available. These hardware-accelerated codecs are available on the latest 4th and 5th Gen Intel®️ Xeon®️ processors running Linux kernel 3.10 and later. For all other systems and platforms, the codecs use that platform's corresponding software implementations. -The new hardware-accelerated codecs can be used by setting one of the following values of the `index.codec` setting: -* `qat_lz4` (OpenSearch 2.14 and later): Hardware-accelerated `LZ4`.  -* `qat_deflate` (OpenSearch 2.14 and later): Hardware-accelerated `DEFLATE`. +The new hardware-accelerated codecs can be used by setting one of the following `index.codec` values: +* `qat_lz4` (OpenSearch 2.14 and later): Hardware-accelerated `LZ4` +* `qat_deflate` (OpenSearch 2.14 and later): Hardware-accelerated `DEFLATE` -`qat_deflate` offers a much better compression ratio than `qat_lz4` with a modest drop in compression and decompression speed. +`qat_deflate` offers a much better compression ratio than `qat_lz4`, with a modest drop in compression and decompression speed. {: .note} The `index.codec.compression_level` setting can be used to specify the compression level for both `qat_lz4` and `qat_deflate`. -The `index.codec.qatmode` setting controls the behavior of the hardware accelerator and uses on of the following values: +The `index.codec.qatmode` setting controls the behavior of the hardware accelerator and uses one of the following values: -* `auto`: If hardware acceleration fails, the algorithm switches to software acceleration. -* `hardware`: Guarantees hardware-only compression. If hardware is not available, an exception occurs until hardware exists. +* `auto`: If hardware acceleration fails, then the algorithm switches to software acceleration. +* `hardware`: Guarantees hardware-only compression. If hardware is not available, then an exception occurs until hardware exists. -For information about the `index.codec.qatmode` settings implications on snapshots, see the [Snapshots](#snapshots) section. +For information about the `index.codec.qatmode` setting's effects on snapshots, see the [Snapshots](#snapshots) section. For more information about hardware acceleration on Intel, see the [Intel (R) QAT accelerator overview](https://www.intel.com/content/www/us/en/architecture-and-technology/intel-quick-assist-technology-overview.html). @@ -78,9 +78,9 @@ When creating a [snapshot]({{site.url}}{{site.baseurl}}/tuning-your-cluster/avai When you restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs of the segments in the source snapshot. For example, if the source snapshot contains segments of the `zstd` or `zstd_no_dict` codecs (introduced in OpenSearch 2.9), you won't be able to restore the snapshot to a cluster that runs on an older OpenSearch version because it doesn't support these codecs. -For hardware-accelerated compression codecs, available in OpenSearch 2.14 and later, the value of `index.codec.qatmode` affects how snapshots and restores are performed. If the value is `auto` (the default), then snapshots and restores can be performed with no additional instructions. However, if the value is `hardware`, it must be reset to `auto` for the restore process to succeed on systems lacking the hardware accelerator +For hardware-accelerated compression codecs, available in OpenSearch 2.14 and later, the value of `index.codec.qatmode` affects how snapshots and restores are performed. If the value is `auto` (the default), then snapshots and restores work without issue. However, if the value is `hardware`, then it must be reset to `auto` in order for the restore process to succeed on systems lacking the hardware accelerator. -You can modify the value of `index.codec.qatmode` during a restore process by setting its value as follows: `"index_settings": {"index.codec.qatmode": "auto"}`. +You can modify the value of `index.codec.qatmode` during the restore process by setting its value as follows: `"index_settings": {"index.codec.qatmode": "auto"}`. {: .note} ### Reindexing diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index 3e027ebee6..20f319d73a 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -79,12 +79,12 @@ OpenSearch supports the following static index-level index settings: - `best_compression` - `zstd` (OpenSearch 2.9 and later) - `zstd_no_dict`(OpenSearch 2.9 and later) - - `qat_lz4`(OpenSearch 2.14 and later, on supported systems) - - `qat_deflate`(OpenSearch 2.14 and later, on supported systems) + - `qat_lz4` (OpenSearch 2.14 and later, on supported systems) + - `qat_deflate` (OpenSearch 2.14 and later, on supported systems) - For `zstd`, `zstd_no_dict`, `qat_lz4`, and `qat_deflate` you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. +For `zstd`, `zstd_no_dict`, `qat_lz4`, and `qat_deflate`, you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`. -- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a trad-eoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if index.codec is set to `zstd` and `zstd_no_dict` in OpenSearch 2.9 and later, and `qat_lz4` and `qat_deflate` in OpenSearch 2.14 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. +- `index.codec.compression_level` (Integer): The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size), but slower compression and decompression speeds lead to higher indexing and search latencies. This setting can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` in OpenSearch 2.9 and later or `qat_lz4` and `qat_deflate` in OpenSearch 2.14 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3. - `index.codec.qatmode` (String): The hardware acceleration mode used for the `qat_lz4` and `qat_deflate` compression codecs. Valid values are `auto` and `hardware`. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `auto`.