Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bug in sort optimization #1903

Merged
merged 2 commits into from
Sep 23, 2020
Merged

Fix bug in sort optimization #1903

merged 2 commits into from
Sep 23, 2020

Conversation

mayya-sharipova
Copy link
Contributor

Fix bug how iterator with skipping functionality
advances and produces docs

Relates to #1725

@mayya-sharipova
Fix bug in sort optimization
4773625 
Fix bug how iterator with skipping functionality
advances and produces docs

Relates to apache#1725
@mayya-sharipova
Copy link
Contributor Author

@jimczi I could not reproduce an issue of duplicate documents in Lucene, and I am still not sure how it actually happens.

Even as we were incorrectly advancing only scorerIterator, as filteredIterator is an intersection of collectorIterator and scorerIterator, a next document from filteredIterator must be a document > the current document of scorerIterator? No?

There was also a question why we did not detect duplicate documents in the tests. First, they don't happen in the tests, and before I was not using AssertingIndexSearcher. I've changed the tests on _doc sort to use AssertingIndexSearcher

@mayya-sharipova
Copy link
Contributor Author

mayya-sharipova commented Sep 22, 2020 •
edited
Loading

@jimczi I was able to reproduce a bug, and in aaab92a added a test that reproducibly fails for me without this patch. It is a very particular combination of docs and queries to have a BitSetIterator as a scorerIterator that also ensures to go through DefaultBulkSorter::scoreRange section for scoring.

jimczi
jimczi approved these changes Sep 23, 2020
View reviewed changes
Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good to me. Thanks for adding the test.

lucene/core/src/test/org/apache/lucene/search/TestFieldSortOptimizationSkipping.java Show resolved Hide resolved
@mayya-sharipova mayya-sharipova merged commit 7d90b85 into apache:master Sep 23, 2020
@mayya-sharipova mayya-sharipova deleted the fix-comparator-bug branch September 23, 2020 13:09
@mayya-sharipova mayya-sharipova mentioned this pull request Sep 23, 2020
Merged
mayya-sharipova added a commit that referenced this pull request Sep 23, 2020
@mayya-sharipova
Fix bug in sort optimization (#1903) (#1915)
da03351 
Fix bug how iterator with skipping functionality
advances and produces docs

Relates to #1725
Backport for #1903
@asfimport asfimport mentioned this pull request Nov 15, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jimczi jimczi jimczi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mayya-sharipova @jimczi