Snapshot/Restore: Ensure that shard failure reasons are correctly stored in CS #25941
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…red in CS The failure reason for snapshot shard failures might not be propagated properly if the master node changes after the errors were reported by other data nodes. This commits ensures that the snapshot shard failure reason is preserved properly and adds workaround for reading old snapshot files where this information might not have been preserved. Closes elastic#25878
imotov
added
:Distributed Coordination/Snapshot/Restore
ywelsch
approved these changes
Jul 28, 2017
| } else { | ||
| String nodeId = in.readOptionalString(); | ||
| State shardState = State.fromValue(in.readByte()); | ||
| String reason = shardState.failed() ? "" : null; |
There was a problem hiding this comment.
can you add a comment here saying why we set reason to ""?
| // Workaround for https://github.com/elastic/elasticsearch/issues/25878 | ||
| // Some old snapshot might still have null in shard failure reasons | ||
| if (snapshotShardFailure.reason == null) { | ||
| snapshotShardFailure.reason = ""; |
There was a problem hiding this comment.
What I don't quite understand: Why will it happily parse the reason field if it is null? Currently we parse it using text(), shouldn't that fail and should we use textOrNull() instead?
There was a problem hiding this comment.
You are right, it should be textOrNull().
| @@ -1128,7 +1128,7 @@ public ClusterState execute(ClusterState currentState) throws Exception { | |||
| for (ObjectObjectCursor<ShardId, ShardSnapshotStatus> shardEntry : snapshotEntry.shards()) { | |||
| ShardSnapshotStatus status = shardEntry.value; | |||
| if (!status.state().completed()) { | |||
| shardsBuilder.put(shardEntry.key, new ShardSnapshotStatus(status.nodeId(), State.ABORTED)); | |||
| shardsBuilder.put(shardEntry.key, new ShardSnapshotStatus(status.nodeId(), State.ABORTED, "aborted")); | |||
There was a problem hiding this comment.
maybe extend the message to "aborted by snapshot deletion"
| @@ -135,6 +135,13 @@ public static String blockMasterFromFinalizingSnapshot(final String repositoryNa | |||
| return masterName; | |||
| } | |||
|
|
|||
| public static String blockMasterFromCreatingSnapshot(final String repositoryName) { | |||
There was a problem hiding this comment.
The method name made me think that it would prevent the master from creating a snapshot at all. Maybe we can call it something along the lines of "blockMasterFromFinalizingSnapshot"?
|
@ywelsch thanks a lot for review and help! I will let it "cook" in CI on the master branch over the weekend and then I will backport it to 5.6 and update the version in writeTo and readFrom if everything goes well. |
imotov
added a commit
that referenced
this pull request
Aug 3, 2017
…red in CS (#25941) The failure reason for snapshot shard failures might not be propagated properly if the master node changes after the errors were reported by other data nodes. This commits ensures that the snapshot shard failure reason is preserved properly and adds workaround for reading old snapshot files where this information might not have been preserved. Closes #25878
imotov
added a commit
that referenced
this pull request
Aug 3, 2017
…red in CS (#25941) The failure reason for snapshot shard failures might not be propagated properly if the master node changes after the errors were reported by other data nodes. This commits ensures that the snapshot shard failure reason is preserved properly and adds workaround for reading old snapshot files where this information might not have been preserved. Closes #25878
imotov
added a commit
that referenced
this pull request
Aug 14, 2017
…red in CS (#26127) The failure reasons for snapshot shard failures might not be propagated properly if the master node changes after errors were reported by other data nodes, which causes them to be stored as null in snapshot files. This commits adds a workaround for reading such snapshot files where this information might not have been preserved and makes sure that the reason is not null if it gets cluster state from another master. This is a partial backport of #25941 to 5.6. Closes #25878
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The failure reason for snapshot shard failures might not be propagated properly if the master node changes after the errors were reported by other data nodes. This commits ensures that the snapshot shard failure reason is preserved properly and adds workaround for reading old snapshot files where this information might not have been preserved.
Closes #25878