Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always include the matching node when resolving point in time #61658

Merged
merged 6 commits into from
Sep 1, 2020

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Aug 27, 2020

If shards are relocated to new nodes, then searches with a point in time will fail, although a pit keeps search contexts open. This commit solves this problem by reducing info used by SearchShardIterator and always including the matching nodes when resolving a point in time.

Closes #61627

@dnhatn dnhatn added >test Issues or PRs that are addressing/adding tests :Search/Search Search-related issues that do not fall into other categories v8.0.0 v7.10.0 labels Aug 27, 2020
@dnhatn dnhatn requested a review from jimczi August 27, 2020 20:43
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Search)

@elasticmachine elasticmachine added the Team:Search Meta label for search team label Aug 27, 2020
Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why we need to disable anything since we keep a reference of the index shard in the local shard reader context ? Are you sure that the problem is the allocation rebalance and not a timeout of the PIT in the test ? If it's not a timeout, we should understand why the reference is not enough when a rebalance occurs. In such case, the PIT should still be valid and the removal of the shard should only happen when the PIT is closed or timed out.

@dnhatn dnhatn changed the title Disable allocation rebalance in point in time test Always include the matching node when resolving point in time Sep 1, 2020
@dnhatn dnhatn added >non-issue and removed >test Issues or PRs that are addressing/adding tests labels Sep 1, 2020
@dnhatn
Copy link
Member Author

dnhatn commented Sep 1, 2020

@jimczi I've updated the PR as we discussed. Can you take another look? Thank you!

@dnhatn dnhatn requested a review from jimczi September 1, 2020 03:51
Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left one question but the change looks good to me. Thanks for adding the explicit tests for rebalance. That made me think that we should maybe have the same test for deleted indices ?

@dnhatn
Copy link
Member Author

dnhatn commented Sep 1, 2020

@elasticmachine test this please

@dnhatn
Copy link
Member Author

dnhatn commented Sep 1, 2020

That made me think that we should maybe have the same test for deleted indices?

@jimczi Sure, will do. If one index of a PIT is deleted, should we allow search to return a partial result when allowPartialSearchResults is true?

@jimczi
Copy link
Contributor

jimczi commented Sep 1, 2020

Sure, will do. If one index of a PIT is deleted, should we allow search to return a partial result when allowPartialSearchResults is true?

By deleted, I meant deleted by the user after the creation of the PIT. In such case the PIT should still be valid since we still have the reference of the index shard in the context ? Regarding partial results, yes I think we should allow to return partial results if some contexts are not found. Isn't it what's implemented today ?

@dnhatn
Copy link
Member Author

dnhatn commented Sep 1, 2020

In such case the PIT should still be valid since we still have the reference of the index shard in the context ?

I think we remove all reader contexts when an index is deleted.

Regarding partial results, yes I think we should allow to return partial results if some contexts are not found. Isn't it what's implemented today ?

We don't. I will work on it in a follow-up.

@dnhatn dnhatn merged commit 294e9cc into elastic:master Sep 1, 2020
@dnhatn dnhatn deleted the disable-alloc-balance-pit branch September 1, 2020 23:12
dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Sep 10, 2020
…c#61658)

If shards are relocated to new nodes, then searches with a point in time
will fail, although a pit keeps search contexts open. This commit solves
this problem by reducing info used by SearchShardIterator and always
including the matching nodes when resolving a point in time.

Closes elastic#61627
dnhatn added a commit that referenced this pull request Sep 10, 2020
If shards are relocated to new nodes, then searches with a point in time
will fail, although a pit keeps search contexts open. This commit solves
this problem by reducing info used by SearchShardIterator and always
including the matching nodes when resolving a point in time.

Closes #61627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>non-issue :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team v7.10.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

PointInTimeIT#testMultipleIndices failed (Unexpected ShardFailures)
4 participants