add RemoveCorruptedShardDataCommand #32281
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/bin/bash | ||
|
|
||
| ES_MAIN_CLASS=org.elasticsearch.index.shard.ShardToolCli \ | ||
| "`dirname "$0"`"/elasticsearch-cli \ | ||
| "$@" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -63,12 +63,6 @@ corruption is detected, it will prevent the shard from being opened. Accepts: | |
| Check for both physical and logical corruption. This is much more | ||
| expensive in terms of CPU and memory usage. | ||
|
|
||
| WARNING: Expert only. Checking shards may take a lot of time on large indices. | ||
| -- | ||
|
|
||
|
|
@@ -279,6 +273,10 @@ Other index settings are available in index modules: | |
|
|
||
| Control over the transaction log and background flush operations. | ||
|
|
||
| <<index-modules-command-line-tools,Command-line tools>>:: | ||
|
|
||
| Command-line tools if shard is corrupted | ||
|
|
||
|
There was a problem hiding this comment. If this is the single elasticsearch-shard tool, I'd refer to it as a single tool. This should be a floated level 3 heading instead of an item in the settings list. I'd go with: [float] You can use the <<index-modules-elasticsearch-shard,elasticsearch-shard>> recovery tool to remove corrupted translog or corrupted Lucene segments if a shard cannot be recovered automatically or restored from backup. |
||
| -- | ||
|
|
||
| include::index-modules/analysis.asciidoc[] | ||
|
|
@@ -297,4 +295,6 @@ include::index-modules/store.asciidoc[] | |
|
|
||
| include::index-modules/translog.asciidoc[] | ||
|
|
||
| include::index-modules/command-line-tools.asciidoc[] | ||
|
|
||
|
There was a problem hiding this comment. @lcawl Instead of including this in the index-modules section, do we want to just go ahead and rename the X-Pack commands section "Command line tools" and include it there? It can still be linked to from the index-modules page, but it would be great if we could move toward having a single command reference. There was a problem hiding this comment. I've created #33005 for that purpose. |
||
| include::index-modules/index-sorting.asciidoc[] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,213 @@ | ||
| [[index-modules-command-line-tools]] | ||
|
|
||
| == Command-line tools | ||
|
|
||
|
There was a problem hiding this comment. This topic should use the tool name in the anchor and heading like we do for the X-Pack tools: [[shard-tool]] The filename should match the anchor: shard-tool.asciidoc. The include (where ever we include the file) will need to be updated accordingly. |
||
| In some cases (a bad drive, user error) the translog or Lucene index on a shard copy | ||
| can become corrupted. When this corruption is detected by Elasticsearch due to mismatching | ||
| checksums, Elasticsearch will fail that shard copy and refuse to use that copy | ||
| of the data. | ||
|
|
||
| *Note*: If there are other copies of the shard available then | ||
| Elasticsearch will automatically recover from one of them using the normal | ||
| shard allocation and recovery mechanism. In particular, if the corrupt shard | ||
| copy was the primary when the corruption was detected then one of its replicas | ||
| will be promoted in its place. | ||
|
|
||
| You can also use <<modules-snapshots,snapshot and restore>> to restore the index. | ||
|
|
||
| Please consider using of this tool like the last resort if there is no copy of the data | ||
| from which Elasticsearch can recover successfully. | ||
|
|
||
| We provide a command-line tool for this - `elasticsearch-shard`. | ||
|
|
||
| The cost of applying this tool is losing the corrupted data. It could be any lost data | ||
| regardless of time: could be a recent or an old data. | ||
|
|
||
|
There was a problem hiding this comment. In general, try to stick to present tense. I don't think it's necessary to go into what causes data corruption or how the normal shard recovery process works. I'd pare this down to: The elasticsearch-shard command enables you to remove a corrupted translog or corrupted Lucene segments if a shard cannot be recovered automatically or restored from backup. WARNING: You will lose the corrupted data when you run When Elasticsearch detects that a shard's translog or Lucene index is corrupted, it fails that shard copy and quits using it. Under normal conditions, the shard is automatically recovered from another copy. If no good copy of the shard is available and you cannot restore from backup, you can use This is somewhat redundant, but it reinforces when it's appropriate to use the tool. |
||
| [WARNING] | ||
| The `elasticsearch-shard` tool should *not* be run while Elasticsearch is | ||
| running. If you attempt to run this tool while Elasticsearch is running, you | ||
| will permanently lose the documents that were contained only in the translog! | ||
|
|
||
|
There was a problem hiding this comment. I'd flip this around: "Stop Elasticsearch before running There was a problem hiding this comment. I think the tool should (does?) protect against running concurrently with an Elasticsearch node by obtaining the node lock for itself, meaning that this warning need not be so emphatically scary. I also think the second sentence is incorrect, you risk losing arbitrarily many documents when running this tool, regardless of whether Elasticsearch is running or not, but you may or may not lose the documents contained only in the translog. (I think this warning was a bit misleading when talking about the existing translog truncation tool, in the sense that truncating the translog means you permanently lose the documents that were contained only in the translog, regardless of whether Elasticsearch is running concurrently or not) |
||
| [WARNING] | ||
| After dropping the corrupted part the allocation id of the shard is changed. | ||
|
There was a problem hiding this comment. We generally try to avoid stacking Warnings and Notes. This second one is really more of a next step than a warning--I'd change it to a paragraph and include it in the truncate translog & remove shards sections. When you use We aren't totally consistent in what we call a command, but we should avoid referring to Elasticsearch APIs as commands, as that just muddies things further. |
||
| `elasticsearch-shard` provides details of command that has to be run after the node | ||
| restart to apply changes: | ||
| You should run follow command to apply allocation id changes: | ||
| [source,txt] | ||
| -------------------------------------------------- | ||
| $ curl -XPOST 'http://localhost:9200/_cluster/reroute' -d ' | ||
| { | ||
| "commands" : [ | ||
| { | ||
| "allocate_stale_primary" : { | ||
| "index" : "twitter", | ||
| "shard" : 0, | ||
| "node" : "pAfJBgAAQACIfI2M_____w", | ||
| "accept_data_loss" : true | ||
| } | ||
| } | ||
| ] | ||
| }' | ||
| -------------------------------------------------- | ||
|
|
||
| === What to do if the translog becomes corrupted? | ||
|
|
||
|
There was a problem hiding this comment. We generally use a verb or gerund (-ing) pattern for tasks. I'd change this to Make sure you explictly specify an anchor for each heading. I think we want to float these headings so it's all on one page? [float] |
||
| In order to drop corrupted translog use `truncate-translog` subcommand: | ||
|
|
||
|
There was a problem hiding this comment. "In order to" can almost always be replaced by just "To...". To remove corrupted translog files, use the
|
||
| * you should specify index name with `--index` and shard id `--shard-id` | ||
| * or specify the full path to corrupted translog with the `-d` option | ||
|
|
||
| [source,txt] | ||
| -------------------------------------------------- | ||
| $ bin/elasticsearch-shard truncate-translog -d /var/lib/elasticsearchdata/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/ | ||
| Checking existing translog files | ||
| !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | ||
| ! WARNING: Elasticsearch MUST be stopped before running this tool ! | ||
| ! ! | ||
| ! WARNING: Documents inside of translog files will be lost ! | ||
| ! ! | ||
| ! WARNING: The following files will be DELETED! ! | ||
| !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | ||
| --> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-41.ckp | ||
| --> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-6.ckp | ||
| --> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-37.ckp | ||
| --> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-24.ckp | ||
| --> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-11.ckp | ||
|
|
||
| Continue and DELETE files? [y/N] y | ||
| Reading translog UUID information from Lucene commit from shard at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/index] | ||
| Translog Generation: 3 | ||
| Translog UUID : AxqC4rocTC6e0fwsljAh-Q | ||
| Removing existing translog files | ||
| Creating new empty checkpoint at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog.ckp] | ||
| Creating new empty translog at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-3.tlog] | ||
|
|
||
| Marking index with the new history uuid : TAUddBstTciV9wAiA6sKFA | ||
| Changing allocation id ceU7CskxT4yRw4M-GLO1mg to nFa2DcCsSlady4LlnJeaEQ | ||
| You should run follow command to apply allocation id changes: | ||
|
|
||
| $ curl -XPOST 'http://localhost:9200/_cluster/reroute' -d ' | ||
| { | ||
| "commands" : [ | ||
| { | ||
| "allocate_stale_primary" : { | ||
| "index" : "twitter", | ||
| "shard" : 0, | ||
| "node" : "pAfJBgAAQACIfI2M_____w", | ||
| "accept_data_loss" : true | ||
| } | ||
| } | ||
| ] | ||
| }' | ||
| Done. | ||
| -------------------------------------------------- | ||
|
|
||
| === What to do if the Lucene index becomes corrupted? | ||
|
|
||
|
There was a problem hiding this comment. As above...I'd change this to: [float] |
||
| In a similar cases the index on a shard copy can become corrupted. | ||
| Like in case with corrupted translog when index corruption is detected by Elasticsearch due | ||
| to mismatching checksums, Elasticsearch will fail that shard copy and refuse to use that copy of the data. | ||
| If there are other copies of the shard available then Elasticsearch will automatically recover from one of | ||
| them using the normal shard allocation and recovery mechanism. | ||
|
|
||
|
There was a problem hiding this comment. I don't think it's necessary to repeat this information. |
||
| In order to remove corrupted segments use `remove-corrupted-segments` subcommand: | ||
| It writes a new segments file that removes reference to problematic (corrupted) Lucene segments if there is | ||
| no copy of the data from which Elasticsearch can recover successfully. | ||
|
|
||
|
There was a problem hiding this comment. In order to > To (as above). I'd go with: To remove corrupted Lucene index segments, use the
|
||
| Highly recommended to make a complete backup of your index before using this to remove corrupted documents | ||
| from your index! | ||
|
|
||
|
There was a problem hiding this comment. I'd change this to a warning (and also add it to the translog section?). WARNING: Back up your data before running |
||
| * you should specify index name with `--index` and shard id `--shard-id` | ||
| * or specify the full path to corrupted translog with the `-d` option | ||
|
|
||
| You can get an overview of the corruption with `--dry-run` option : | ||
|
|
||
|
There was a problem hiding this comment. I'd be inclined to break the dry-run option out into it's own section: To see what segments are corrupted and will be dropped, use the
|
||
| [source,txt] | ||
| -------------------------------------------------- | ||
| $ bin/elasticsearch-shard remove-corrupted-segments --dry-run --index twitter --shard-id 0 | ||
|
|
||
| Opening index @ /var/lib/elasticsearchdata/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/index/ | ||
|
|
||
| WARNING: Corrupted segments found - 568 documents are damaged | ||
|
|
||
| -------------------------------------------------- | ||
|
|
||
| Running `remove-corrupted-segments` without `--dry-run` requires interactive confirmation to drop damaged segments: | ||
|
|
||
|
There was a problem hiding this comment. I'd just include this as part of the remove segments section and say, "You must confirm that you want to remove the corrupted segments:" |
||
| [source,txt] | ||
| -------------------------------------------------- | ||
|
|
||
| $ bin/elasticsearch-shard remove-corrupted-segments --index twitter --shard-id 0 | ||
|
|
||
| Opening index @ /var/lib/elasticsearchdata/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/index/ | ||
|
|
||
| Segments file=segments_8 numSegments=6 version=7.4.0 id=efcaej17mrbjqf9jf5js52e94 userData={history_uuid=3Mu-8x3zTMm8TIZxwTkTZw, local_checkpoint=1896, max_seq_no=1896, max_unsafe_auto_id_timestamp=-1, translog_generation=7, translog_uuid=2n8vuupLQWSh5LDQzRe2fQ} | ||
| 1 of 2: name=_0 maxDoc=1 | ||
| version=7.4.0 | ||
| id=efcaej17mrbjqf9jf5js52e8k | ||
| codec=Lucene70 | ||
| compound=true | ||
| numFiles=3 | ||
| size (MB)=0.004 | ||
| diagnostics = {java.runtime.version=10.0.2+13, java.vendor=Oracle Corporation, java.version=10.0.2, java.vm.version=10.0.2+13, lucene.version=7.4.0, os=Mac OS X, os.arch=x86_64, os.version=10.13.6, source=flush, timestamp=1532081797245} | ||
| no deletions | ||
| test: open reader.........OK [took 0.001 sec] | ||
| test: check integrity.....OK [took 0.000 sec] | ||
| test: check live docs.....OK [took 0.000 sec] | ||
| test: field infos.........OK [9 fields] [took 0.000 sec] | ||
| test: field norms.........OK [2 fields] [took 0.000 sec] | ||
| test: terms, freq, prox...OK [5 terms; 5 terms/docs pairs; 2 tokens] [took 0.000 sec] | ||
| test: stored fields.......OK [2 total field count; avg 2.0 fields per doc] [took 0.000 sec] | ||
| test: term vectors........OK [0 total term vector count; avg 0.0 term/freq vector fields per doc] [took 0.000 sec] | ||
| test: docvalues...........OK [5 docvalues fields; 0 BINARY; 3 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.000 sec] | ||
| test: points..............OK [1 fields, 1 points] [took 0.000 sec] | ||
|
|
||
| 2 of 2: name=_1 maxDoc=568 | ||
| version=7.4.0 | ||
| id=efcaej17mrbjqf9jf5js52e8q | ||
| codec=Lucene70 | ||
| compound=true | ||
| numFiles=3 | ||
| size (MB)=1.148 | ||
| diagnostics = {java.runtime.version=10.0.2+13, java.vendor=Oracle Corporation, java.version=10.0.2, java.vm.version=10.0.2+13, lucene.version=7.4.0, os=Mac OS X, os.arch=x86_64, os.version=10.13.6, source=flush, timestamp=1532081798123} | ||
| no deletions | ||
| test: open reader.........FAILED | ||
| WARNING: exorciseIndex() would remove reference to this segment; | ||
|
|
||
| WARNING: 1 broken segments (containing 568 documents) detected | ||
| Took 0.049 sec total. | ||
| !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | ||
| ! WARNING: 568 documents will be lost. ! | ||
| ! ! | ||
| ! WARNING: YOU WILL LOSE DATA. ! | ||
| !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! | ||
|
|
||
| Continue and remove 568 docs from the index ? [y/N] | ||
|
|
||
| Writing... | ||
| OK | ||
| Wrote new segments file "segments_8" | ||
|
|
||
| Marking index with the new history uuid : TAUddBstTciV9wAiA6sKFA | ||
| Changing allocation id ceU7CskxT4yRw4M-GLO1mg to nFa2DcCsSlady4LlnJeaEQ | ||
| You should run follow command to apply allocation id changes: | ||
|
|
||
| $ curl -XPOST 'http://localhost:9200/_cluster/reroute' -d ' | ||
| { | ||
| "commands" : [ | ||
| { | ||
| "allocate_stale_primary" : { | ||
| "index" : "twitter", | ||
| "shard" : 0, | ||
| "node" : "pAfJBgAAQACIfI2M_____w", | ||
| "accept_data_loss" : true | ||
| } | ||
| } | ||
| ] | ||
| }' | ||
| Deleted corrupt marker corrupted_cJv5hCxeTE2p3AucpgCJyg | ||
|
|
||
| -------------------------------------------------- | ||
|
|
||
| You can also use the `-h` option to get a list of all options and parameters | ||
| that the `elasticsearch-shard` tool supports. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -78,3 +78,12 @@ The parent circuit breaker defines a new setting `indices.breaker.total.use_real | |
| heap memory instead of only considering the reserved memory by child circuit breakers. When this | ||
| setting is `true`, the default parent breaker limit also changes from 70% to 95% of the JVM heap size. | ||
| The previous behavior can be restored by setting `indices.breaker.total.use_real_memory` to `false`. | ||
|
|
||
| ==== `fix` value for `index.shard.check_on_startup` is removed | ||
|
There was a problem hiding this comment. I thought we planned on documenting this breakage in the 6.0 breaking changes, and its removal in the backport of #32279. In any case I think this shouldn't be part of this PR. |
||
|
|
||
| `elasticsearch-shard remove-corrupted-segments` tool has to be used instead of | ||
| `index.shard.check_on_startup: fix` setting. | ||
|
There was a problem hiding this comment. Let's consistently refer to it as the "elasticsearch-shard" tool. I'd just say: Use the |
||
|
|
||
| ==== `elasticsearch-translog` tool merged into `elasticsearch-shard` | ||
|
|
||
| Instead of `elasticsearch-translog` tool you should use `elasticsearch-shard truncate-translog`. | ||
|
There was a problem hiding this comment. Similarly: Use the |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The removal of the
elasticsearch-translogtool is a breaking change, so this cannot happen in 6.5. At the moment this PR is only tagged for 7.0, which is ok, but it cannot be backported as-is.