Document the index corruption bug that gets fixed via Lucene 8.5.1. (#…

…55232) Using soft deletes on shrunk indices may cause corruption.
elastic · Apr 17, 2020 · 5ac8fe5 · 5ac8fe5
1 parent a80559a
commit 5ac8fe5
Show file tree

Hide file tree

Showing 9 changed files with 74 additions and 0 deletions.
diff --git a/docs/reference/release-notes/7.0.asciidoc b/docs/reference/release-notes/7.0.asciidoc
@@ -9,6 +9,14 @@ Also see <<breaking-changes-7.0,Breaking changes in 7.0>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Indices created in 6.x with <<date,`date`>> and <<date_nanos,`date_nanos`>> fields using formats
 that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results.
 https://github.com/elastic/elasticsearch/pull/52555

diff --git a/docs/reference/release-notes/7.1.asciidoc b/docs/reference/release-notes/7.1.asciidoc
@@ -6,6 +6,14 @@ Also see <<breaking-changes-7.1,Breaking changes in 7.1>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Indices created in 6.x with <<date,`date`>> and <<date_nanos,`date_nanos`>> fields using formats
 that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results.
 https://github.com/elastic/elasticsearch/pull/52555

diff --git a/docs/reference/release-notes/7.2.asciidoc b/docs/reference/release-notes/7.2.asciidoc
@@ -88,6 +88,14 @@ Also see <<breaking-changes-7.2,Breaking changes in 7.2>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Indices created in 6.x with <<date,`date`>> and <<date_nanos,`date_nanos`>> fields using formats
 that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results.
 https://github.com/elastic/elasticsearch/pull/52555

diff --git a/docs/reference/release-notes/7.3.asciidoc b/docs/reference/release-notes/7.3.asciidoc
@@ -143,6 +143,14 @@ Also see <<breaking-changes-7.3,Breaking changes in 7.3>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Indices created in 6.x with <<date,`date`>> and <<date_nanos,`date_nanos`>> fields using formats
 that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results.
 https://github.com/elastic/elasticsearch/pull/52555

diff --git a/docs/reference/release-notes/7.4.asciidoc b/docs/reference/release-notes/7.4.asciidoc
@@ -111,6 +111,14 @@ Also see <<breaking-changes-7.4,Breaking changes in 7.4>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Activating the <<search-slow-log, search slow log>> should be avoided in this version.
 Any attempt to log a slow search can throw an AIOOBE due to a bug that
 performs concurrent modifications on a shared byte array.

diff --git a/docs/reference/release-notes/7.5.asciidoc b/docs/reference/release-notes/7.5.asciidoc
@@ -94,6 +94,14 @@ Also see <<breaking-changes-7.5,Breaking changes in 7.5>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Stop all {transforms} during a rolling upgrade to 7.5.
 If a {transform} is running during upgrade, the {transform} audit index might disappear.
 (issue: {issue}/49730[#49730])

diff --git a/docs/reference/release-notes/7.6.asciidoc b/docs/reference/release-notes/7.6.asciidoc
@@ -155,6 +155,14 @@ Also see <<breaking-changes-7.6,Breaking changes in 7.6>>.
 [float]
 === Known issues
 
+* Applying deletes or updates on an index after it has been shrunk may corrupt
+the index. In order to prevent this issue, it is recommended to stop shrinking
+read-write indices. For read-only indices, it is recommended to force-merge
+indices after shrinking, which significantly reduces the likeliness of this
+corruption in the case that deletes/updates would be applied by mistake. This
+bug is fixed in {es} 7.7 and later versions. More details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
 * Indices created in 6.x with <<date,`date`>> and <<date_nanos,`date_nanos`>> fields using formats
 that are incompatible with java.time patterns will cause parsing errors, incorrect date calculations or wrong search results.
 https://github.com/elastic/elasticsearch/pull/52555

diff --git a/docs/reference/release-notes/7.7.asciidoc b/docs/reference/release-notes/7.7.asciidoc
@@ -447,6 +447,7 @@ Engine::
 * Fix doc_stats and segment_stats of ReadOnlyEngine {pull}53345[#53345] (issues: {issue}51303[#51303], {issue}51331[#51331])
 * Do not wrap soft-deletes reader for segment stats {pull}51331[#51331] (issues: {issue}51192[#51192], {issue}51303[#51303])
 * Account soft-deletes in FrozenEngine {pull}51192[#51192] (issue: {issue}50775[#50775])
+* Fixed an index corruption bug that would occur when applying deletes or updates on an index after it has been shrunk. More details can be found on the https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
 
 Features/CAT APIs::
 * Fix NPE in RestPluginsAction {pull}52620[#52620] (issue: {issue}45321[#45321])

diff --git a/docs/reference/release-notes/highlights-7.7.0.asciidoc b/docs/reference/release-notes/highlights-7.7.0.asciidoc
@@ -7,6 +7,23 @@
 //NOTE: The notable-highlights tagged regions are re-used in the
 //Installation and Upgrade Guide
 
+// tag::notable-highlights[]
+[float]
+=== Fixed index corruption on shrunk indices
+
+Applying deletes or updates on an index after it had been shrunk would likely
+corrupt the index. We advise users of Elasticsearch 6.x who opt in for soft
+deletes on some of their indices and all users of Elasticsearch 7.x to upgrade
+to 7.7 as soon as possible to no longer be subject to this corruption bug. In
+case upgrading in the near future is not an option, we recommend to completely
+stop using `_shrink` on read-write indices and to do a force-merge right after
+shrinking on read-only indices, which significantly reduces the likeliness of
+being affected by this bug in case deletes or updates get applied by mistake.
+This bug is fixed as of {es} 7.7.0. Low-level details can be found on the
+https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue].
+
+// end::notable-highlights[]
+
 // tag::notable-highlights[]
 [float]
 === Significant reduction of heap usage of segments