Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail earlier Put Follow requests for closed leader indices #47582

Merged

Conversation

tlrx
Copy link
Member

@tlrx tlrx commented Oct 4, 2019

Today when following a new leader index, we fetch the remote cluster state, check the remote cluster license, check the user privileges, retrieve the index shard stats before initiating a CCR restore session.

But if the leader index to follow is closed, we're executing a bunch of operations that would inevitability fail at some point (on retrieving the index shard stats, because this type of request forbid closed indices when resolving indices). We could fail a Put Follow request at the first step by checking the leader index state directly from the remote cluster state.

This also helps the Resume Follow API to fail a bit earlier.

@tlrx tlrx added >non-issue :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features v7.5.0 v7.4.1 labels Oct 4, 2019
@tlrx tlrx requested review from martijnvg and dnhatn October 4, 2019 15:05
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/CCR)

Copy link
Member

@dnhatn dnhatn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @tlrx.

@tlrx tlrx merged commit 2ea8844 into elastic:master Oct 7, 2019
@tlrx tlrx added the v8.0.0 label Oct 7, 2019
tlrx added a commit to tlrx/elasticsearch that referenced this pull request Oct 7, 2019
…7582)

Today when following a new leader index, we fetch the remote cluster state, 
check the remote cluster license, check the user privileges, retrieve the 
index shard stats before initiating a CCR restore session.

But if the leader index to follow is closed, we're executing a bunch of 
operations that would inevitability fail at some point (on retrieving the
 index shard stats, because this type of request forbid closed indices 
when resolving indices). We could fail a Put Follow request at the first 
step by checking the leader index state directly from the remote cluster 
state.

This also helps the Resume Follow API to fail a bit earlier.
tlrx added a commit that referenced this pull request Oct 7, 2019
Backport of (#47582)

Today when following a new leader index, we fetch the remote cluster state, 
check the remote cluster license, check the user privileges, retrieve the 
index shard stats before initiating a CCR restore session.

But if the leader index to follow is closed, we're executing a bunch of 
operations that would inevitability fail at some point (on retrieving the
 index shard stats, because this type of request forbid closed indices 
when resolving indices). We could fail a Put Follow request at the first 
step by checking the leader index state directly from the remote cluster 
state.

This also helps the Resume Follow API to fail a bit earlier.
tlrx added a commit that referenced this pull request Oct 7, 2019
Backport of #47582

Today when following a new leader index, we fetch the remote cluster state, 
check the remote cluster license, check the user privileges, retrieve the 
index shard stats before initiating a CCR restore session.

But if the leader index to follow is closed, we're executing a bunch of 
operations that would inevitability fail at some point (on retrieving the
 index shard stats, because this type of request forbid closed indices 
when resolving indices). We could fail a Put Follow request at the first 
step by checking the leader index state directly from the remote cluster 
state.

This also helps the Resume Follow API to fail a bit earlier.
tlrx added a commit that referenced this pull request Oct 9, 2019
Similarly to #47582, Auto-follow patterns creates following 
indices as long as the remote index matches the pattern and 
the remote primary shards are all started. But since 7.2 closed 
indices are also replicated, and it does not play well with CCR 
auto-follow patterns as they create following indices for closed 
leader indices too.

This comit changes the getLeaderIndicesToFollow() so that closed 
indices are excluded from auto-follow patterns.
tlrx added a commit to tlrx/elasticsearch that referenced this pull request Oct 9, 2019
Backport of (elastic#47721) for 7.x.

Similarly to elastic#47582, Auto-follow patterns creates following
indices as long as the remote index matches the pattern and
the remote primary shards are all started. But since 7.2 closed
indices are also replicated, and it does not play well with CCR
auto-follow patterns as they create following indices for closed
leader indices too.

This commit changes the getLeaderIndicesToFollow() so that closed
indices are excluded from auto-follow patterns.
tlrx added a commit that referenced this pull request Oct 9, 2019
Backport of (#47721) for 7.x.

Similarly to #47582, Auto-follow patterns creates following
indices as long as the remote index matches the pattern and
the remote primary shards are all started. But since 7.2 closed
indices are also replicated, and it does not play well with CCR
auto-follow patterns as they create following indices for closed
leader indices too.

This commit changes the getLeaderIndicesToFollow() so that closed
indices are excluded from auto-follow patterns.
tlrx added a commit that referenced this pull request Oct 9, 2019
Backport of (#47721) for 7.x.

Similarly to #47582, Auto-follow patterns creates following
indices as long as the remote index matches the pattern and
the remote primary shards are all started. But since 7.2 closed
indices are also replicated, and it does not play well with CCR
auto-follow patterns as they create following indices for closed
leader indices too.

This commit changes the getLeaderIndicesToFollow() so that closed
indices are excluded from auto-follow patterns.
@tlrx tlrx deleted the fail-earlier-when-following-closed-indices branch October 18, 2019 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >non-issue v7.4.1 v7.5.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants