Migrate peer recovery from translog to retention lease #49448
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnhatn
added
>bug
:Distributed Indexing/Recovery
|
Pinging @elastic/es-distributed (:Distributed/Recovery) |
|
Hmm, a new test is failing. I am looking at it. |
ywelsch
reviewed
Nov 22, 2019
qa/full-cluster-restart/src/test/java/org/elasticsearch/upgrades/FullClusterRestartIT.java
Outdated
Show resolved
Hide resolved
|
I have an implementation that fallbacks to translog if an index was created before 7.4, and the recovering replica does not have a PRRL. I think we should disable translog retention after every copy has established its PRRLs. However, this would require coordination. Another option is to make this decision locally. We also need to persist this decision so that we won't re-enable translog retention in a full cluster restart. WDYT? |
|
ReplicationTracker already has this field |
dnhatn
changed the title
|
Please hold off the review as the test failure relates to this change. I will ping after I have resolved it. |
|
run elasticsearch-ci/packaging-sample-matrix |
ywelsch
suggested changes
Dec 3, 2019
server/src/main/java/org/elasticsearch/index/seqno/ReplicationTracker.java
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
|
@ywelsch Thanks for reviewing. |
dnhatn
added a commit
that referenced
this pull request
Dec 15, 2019
We turn off the translog retention policy asynchronously using the generic threadpool; hence, we need to assert busily here Relates #49448
dnhatn
added a commit
that referenced
this pull request
Dec 15, 2019
Since 7.4, we switch from translog to Lucene as the source of history for peer recoveries. However, we reduce the likelihood of operation-based recoveries when performing a full cluster restart from pre-7.4 because existing copies do not have PPRL. To remedy this issue, we fallback using translog in peer recoveries if the recovering replica does not have a peer recovery retention lease, and the replication group hasn't fully migrated to PRRL. Relates #45136
dnhatn
added a commit
to dnhatn/elasticsearch
that referenced
this pull request
Dec 16, 2019
Since 7.4, we switch from translog to Lucene as the source of history for peer recoveries. However, we reduce the likelihood of operation-based recoveries when performing a full cluster restart from pre-7.4 because existing copies do not have PPRL. To remedy this issue, we fallback using translog in peer recoveries if the recovering replica does not have a peer recovery retention lease, and the replication group hasn't fully migrated to PRRL. Relates elastic#45136
dnhatn
added a commit
that referenced
this pull request
Dec 16, 2019
Since 7.4, we switch from translog to Lucene as the source of history for peer recoveries. However, we reduce the likelihood of operation-based recoveries when performing a full cluster restart from pre-7.4 because existing copies do not have PPRL. To remedy this issue, we fallback using translog in peer recoveries if the recovering replica does not have a peer recovery retention lease, and the replication group hasn't fully migrated to PRRL. Relates #45136
dnhatn
added a commit
that referenced
this pull request
Dec 24, 2019
We need to make sure that the global checkpoints and peer recovery retention leases were advanced to the max_seq_no and synced; otherwise, we can risk expiring some peer recovery retention leases because of the file-based recovery threshold. Relates #49448
dnhatn
added a commit
that referenced
this pull request
Dec 24, 2019
We need to make sure that the global checkpoints and peer recovery retention leases were advanced to the max_seq_no and synced; otherwise, we can risk expiring some peer recovery retention leases because of the file-based recovery threshold. Relates #49448
SivagurunathanV
pushed a commit
to SivagurunathanV/elasticsearch
that referenced
this pull request
Jan 23, 2020
Since 7.4, we switch from translog to Lucene as the source of history for peer recoveries. However, we reduce the likelihood of operation-based recoveries when performing a full cluster restart from pre-7.4 because existing copies do not have PPRL. To remedy this issue, we fallback using translog in peer recoveries if the recovering replica does not have a peer recovery retention lease, and the replication group hasn't fully migrated to PRRL. Relates elastic#45136
SivagurunathanV
pushed a commit
to SivagurunathanV/elasticsearch
that referenced
this pull request
Jan 23, 2020
We turn off the translog retention policy asynchronously using the generic threadpool; hence, we need to assert busily here Relates elastic#49448
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since 7.4, we switch from translog to Lucene as the source of history for peer recoveries. However, we reduce the likelihood of operation-based recoveries when performing a full cluster restart from pre-7.4 because existing copies do not have PPRL.
To remedy this issue, we fallback using translog in peer recoveries if the recovering replica does not have a peer recovery retention lease, and the replication group hasn't fully migrated to PRRL.
Relates #45136